Sedgemoor - Working code provided

[canalla450]

Name of Council

Sedgemoor

Example Postcode

TA6 4AA

Additional Information

Sedgemoor is one of I believe several councils that use the itouchvision service.
Access to the data is normally hidden behind a two step process where you have to enter your postcode on the first page, then select your property from a drop down list (setting the UPRN in the background). Once you've made your selection, you can then view a page with the collection details.

I've managed to get the below code to work; it will need tidying and optimising as Python is not my programming language of choice.

An example postcode and UPRN are hard coded at the top.

import requests
import json 
from datetime import datetime
from bs4 import BeautifulSoup

P153_POST_CODE = 'TA6 4AA'
P153_UPRN = '10090857775'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36'
    }

with requests.Session() as s:

    #Set Headers
    s.headers = headers
    
    #Get the first page - This is the Search for property by Post Code page
    resource = s.get('https://iweb.itouchvision.com/portal/f?p=customer:BIN_DAYS:::NO:RP:UID:625C791B4D9301137723E9095361401AE8C03934')
    #Create a BeautifulSoup object from the page's HTML
    soup = BeautifulSoup(resource.text,'html.parser')
    
    #The page contains a number of values that must be passed into subsequent requests - extract them here
    payload = {i['name']:i.get('value','') for i in soup.select('input[name]')}
    payload2 = {i['data-for']:i.get('value','') for i in soup.select('input[data-for]')}
    payloadsalt = soup.select_one('input[id="pSalt"]').get('value')
    payloadprotected = soup.select_one('input[id="pPageItemsProtected"]').get('value')

    #Add the PostCode and 'SEARCH' to the payload
    payload['p_request'] = 'SEARCH'
    payload['P153_POST_CODE'] = P153_POST_CODE

    # Manipulate the lists and build the JSON that must be submitted in further requests - some data is nested 
    mergedlist = {**payload, **payload2}
    newlist = []
    theotherlist = {}
    for key in mergedlist.keys():
        templist = {}
        val = mergedlist[key]
        if key in ['P153_UPRN','P153_TEMP', 'P153_SYSDATE','P0_LANGUAGE','P153_POST_CODE']:
            templist = {"n": key, "v":val}
            newlist.append(templist)
        elif key in ["p_flow_id","p_flow_step_id", "p_instance","p_page_submission_id","p_request","p_reload_on_submit"]:
             theotherlist[key] = val
        else:
            templist = {"n": key, "v":"",  "ck":val}
            newlist.append(templist)

    buildthejson = {'pageItems': {'itemsToSubmit': newlist, 'protected': payloadprotected, 'rowVersion': '', 'formRegionChecksums': [] } , 'salt': payloadsalt}
    json_object = json.dumps(buildthejson,separators=(',', ':'))
    theotherlist["p_json"] = json_object

    # Set Referrer header
    s.headers.update({'referer': 'https://iweb.itouchvision.com/portal/f?p=customer:BIN_DAYS:::NO:RP:UID:625C791B4D9301137723E9095361401AE8C03934'})

    # Generate POST including all the JSON we just built
    posted = s.post('https://iweb.itouchvision.com/portal/wwv_flow.accept',data=theotherlist)

    # The second page on the portal would normally allow you to select your property from a drop down list of those that are at the Post Code entered in the previous page
    
    # The required cookies are stored within the session so re-use the session to keep them
    resource = s.get('https://iweb.itouchvision.com/portal/itouchvision/r/customer/bin_days')

    #Create a BeautifulSoup object from the page's HTML
    moresoup = BeautifulSoup(resource.text,'html.parser')
    
    #The page contains a number of values that must be passed into subsequent requests - extract them here   
    payload = {i['name']:i.get('value','') for i in moresoup.select('input[name]')}
    payload2 = {i['data-for']:i.get('value','') for i in moresoup.select('input[data-for]')}
    payloadsalt = moresoup.select_one('input[id="pSalt"]').get('value')
    payloadprotected = moresoup.select_one('input[id="pPageItemsProtected"]').get('value')

    #Add the UPRN and 'SUBMIT' to the payload
    payload['p_request'] = 'SUBMIT'
    payload['P153_UPRN'] = P153_UPRN

    # Manipulate the lists and build the JSON that must be submitted in further requests - some data is nested 
    mergedlist = {**payload, **payload2}
    newlist = []
    theotherlist = {}
    for key in mergedlist.keys():
        templist = {}
        val = mergedlist[key]
        if key in ['P153_UPRN','P153_TEMP', 'P153_SYSDATE','P0_LANGUAGE']:
            templist = {"n": key, "v":val}
            newlist.append(templist)
        elif key in ['P153_ZABY']:
            templist = {"n": key, "v": "1", "ck":val}
            newlist.append(templist)
        elif key in ['P153_POST_CODE']:
            templist = {"n": key, "v": P153_POST_CODE, "ck":val}
            newlist.append(templist)
        elif key in ["p_flow_id","p_flow_step_id", "p_instance","p_page_submission_id","p_request","p_reload_on_submit"]:
             theotherlist[key] = val
        else:
            templist = {"n": key, "v":"",  "ck":val}
            newlist.append(templist)

    buildthejson = {'pageItems': {'itemsToSubmit': newlist, 'protected': payloadprotected, 'rowVersion': '', 'formRegionChecksums': [] } , 'salt': payloadsalt}

    json_object = json.dumps(buildthejson,separators=(',', ':'))
    theotherlist["p_json"] = json_object     

    # Generate POST including all the JSON we just built
    posted = s.post('https://iweb.itouchvision.com/portal/wwv_flow.accept',data=theotherlist)   

    # The third and final page on the portal shows the detail of the waste collection services
        
    # The required cookies are stored within the session so re-use the session to keep them
    resource = s.get('https://iweb.itouchvision.com/portal/itouchvision/r/customer/bin_days')

    #Create a BeautifulSoup object from the page's HTML
    delicioussoup = BeautifulSoup(resource.text,'html.parser')

    thoselovelyvalues = {}

    # Loop through the items on the page and build a JSON object for ingestion
    for i in delicioussoup.select('.t-MediaList-item'):
        for j in i.select('.t-MediaList-body'):
            k = j.select('span')[1].get_text(strip=True)
            l = j.select('.t-MediaList-desc')[0].get_text(strip=True)
            thoselovelyvalues[k] = datetime.strptime(l, "%A, %d %B, %Y").strftime('%Y-%m-%d')

    print (json.dumps(thoselovelyvalues))

Example output of the above script:

{"GARDEN": "2023-01-26", "RECYCLING": "2023-01-20", "REFUSE": "2023-01-20"}

[dp247]

I'll add the project files and make a PR.

[OliverCullimore]

As @canalla450 mentioned this script should work for the following councils also once implemented:

Mendip District Council
Somerset West and Taunton Council
South Somerset District Council

Which appear to be merging into a single "SomersetCouncil" soon which could possibly be a good name for this script when it's added @dp247

[dp247]

As @canalla450 mentioned this script should work for the following councils also once implemented:

Mendip District Council Somerset West and Taunton Council South Somerset District Council

Which appear to be merging into a single "SomersetCouncil" soon which could possibly be a good name for this script when it's added @dp247

May have forgot I assigned this, I'll work on it over the weekend 😀