UnauthorizedException is back

[peternijssen]

The UnauthorizedException is back like in the previous issue. Unclear is yet why, but apparently not everyone is suffering from this case, as noted in https://community.home-assistant.io/t/lovelace-postnl/112433/240

I tried to debug, but having a hard time to track my requests on an android device.

The earlier mentioned user agents here don't solve the problem. So it's unclear if it's related to the header.

cc @IcyPalm

[IcyPalm]

Yeah... PostNL made it a LOT harder. I appears that they are using: Akamai Bot Manager

Based on the X-acf-sensor-data header

I'm not sure if/how we are able to bypass this.

[peternijssen]

Damn... That sounds like they are actively blocking us....

[peternijssen]

USPS had the same case: home-assistant/core#13688

[IcyPalm]

Damn... That sounds like they are actively blocking us....

It's a possibility, if any PostNL devs are reading this: Let's have a beer and talk this over 🥂

[peternijssen]

I wonder if we set the update time to once a day, if we are being "unblocked" after like a week or so. Since it took some time for some people to get blocked.

[peternijssen]

I got around it.

Make the following changes in the code:

Change:

AUTHENTICATE_URL = BASE_URL + "/mobile/token"

To:

AUTHENTICATE_URL = BASE_URL + "/web/token"

Change:

"client_id": "pwAndroidApp",

To:

"client_id": "pwWebApp",

You are now using their web login, which still gives you an access key which also works with the mobile end points.

Potentially all end points could be rebuild to the web interface even....

[IcyPalm]

We have to check whether that is in the design specs of Homeassistant though, since it's borderline webscraping.

But I like the idea, I was just fiddling with the web api myself

[peternijssen]

Well, it's still an API. So not really scraping an HTML page. On the other hand, PostNL didn't build a "bot protection" without reasons I suppose.

[D2R9]

Well I still have trouble to login. I changed the lines as you said, but still received an 403 http error :-(

[IcyPalm]

I am working on a fix, currently trying out new ways to interface with the multiple endpoints

[iMicknl]

Did anyone have contact with PostNL regarding the API implementation? Possibly we could start the discussion and also discuss items like refresh time with them.

[peternijssen]

Tried through their webcare team, but failed. Was thinking about reaching out to them through Linkedin, so let me see what I can do.

[megapearl]

I got around it.

Make the following changes in the code:

Change:

AUTHENTICATE_URL = BASE_URL + "/mobile/token"

To:

AUTHENTICATE_URL = BASE_URL + "/web/token"

Change:

"client_id": "pwAndroidApp",

To:

"client_id": "pwWebApp",

You are now using their web login, which still gives you an access key which also works with the mobile end points.

Potentially all end points could be rebuild to the web interface even....

Works for me!

[D2R9]

I got around it.
Make the following changes in the code:
Change:

AUTHENTICATE_URL = BASE_URL + "/mobile/token"

To:

AUTHENTICATE_URL = BASE_URL + "/web/token"

Change:

"client_id": "pwAndroidApp",

To:

"client_id": "pwWebApp",

You are now using their web login, which still gives you an access key which also works with the mobile end points.
Potentially all end points could be rebuild to the web interface even....

Works for me!

Not for me, still got 403 error.

[Muyz]

I got around it.
Make the following changes in the code:
Change:

AUTHENTICATE_URL = BASE_URL + "/mobile/token"

To:

AUTHENTICATE_URL = BASE_URL + "/web/token"

Change:

"client_id": "pwAndroidApp",

To:

"client_id": "pwWebApp",

You are now using their web login, which still gives you an access key which also works with the mobile end points.
Potentially all end points could be rebuild to the web interface even....

Works for me!

Does not work for me as well.

[IcyPalm]

Life got in the way, I had less time this week than expected.
I probably won't have any time until sunday/monday to work on this with the attention it needs

[0x00-0xFF]

This works:

curl -i -s -k -X $'POST' \
    -H $'Host: jouw.postnl.nl' -H $'Accept: */*' -H $'Accept-Encoding: gzip, deflate' -H $'Accept-Language: nl' -H $'Content-Length: 100' -H $'Content-Type: application/x-www-form-urlencoded' \
    --data-binary $'\x0d\x0a\x0d\x0aclient_id=pwWebApp&grant_type=password&password=yoursecretpassword&username=some%40mailaccount.com' \
    $'https://jouw.postnl.nl/web/token'

So some headers are not necessary...
Fix:

postnl_api.AUTHENTICATE_URL = "https://jouw.postnl.nl/web/token"
postnl_api.DEFAULT_HEADER = {"Api-Version":"4.18", "User-Agent": "PostNL/iOS/4.32.0"}

[sanderjonkers]

I got around it.

Make the following changes in the code:

Change:

AUTHENTICATE_URL = BASE_URL + "/mobile/token"

To:

AUTHENTICATE_URL = BASE_URL + "/web/token"

Change:

"client_id": "pwAndroidApp",

To:

"client_id": "pwWebApp",

You are now using their web login, which still gives you an access key which also works with the mobile end points.

Potentially all end points could be rebuild to the web interface even....

Where (which file / code) do I need to change this?

[icepick3000]

I tried the suggestions from above but with no luck.. it's still not working for me. I keep on getting;

_LOGGER.debug(f"Access denied. Failed to refresh, attempt {count} of {max}.")

I have opened a webcare ticket with postnl about 1.5 years ago with a request for a decent API or some form of IFTTT integration.. but they never got back to me. Everytime I ask them the ticket is still pending.

[peternijssen]

I've also set the scan_interval to max once a day for now. Not sure if that solves the issue, together with the above mentioned work-around.

[0x00-0xFF]

Still working like a charm here...
Setting the scan_interval most likely helpes keeping the relationship with postnl in good order.

But once a day, guessing we might miss out some alerts. Do we know for sure at what time postnl updates the data?

[D2R9]

Okay it is working. But how can I get the latest python-postNL-api to work with LoveLace from https://community.home-assistant.io/u/ptnijssen ? Is there a recently updated howto?

[iMicknl]

@0x00-0xFF, @peternijssen someone who would like to do a PR with the discussed changes? I could publish it asap and let's get it to HA core.

[0x00-0xFF]

Done

[iMicknl]

Thanks @0x00-0xFF, it has been published on PyPi as version 1.2.3.

@peternijssen, @IcyPalm, would you like to try it out? Let's have it thoroughly tested and get it back in HA. For me it works with those changes.

[reharmsen]

Good work guys!
Anyone been in touch with PostNL yet?
Any expectations when this could be fixed in HA-core or in a HA-Custom- Component?

I really miss my PostNL info in HA

[0x00-0xFF]

You could load it as a custom component, the fix is working for me. Clone this repo and add it to your your custom components and let us know if it's working for you as well.

[metbril]

@0x00-0xFF Do you mean to load this new version of the api as a custom component? How should we do that? Or copy the postnl component and adjust it to use the new api?

I followed these steps:

1. Download latest home assistant code as zip
2. Copy postnl component folder to my custom_components folder
3. Modified the manifest to use api version 1.2.3
4. Restarted HA

This is the result:

.
.
.
INFO:homeassistant.util.package:Attempting install of postnl_api==1.2.3
.
.
.
2019-12-15 16:04:57 INFO (SyncWorker_1) [homeassistant.loader] Loaded postnl from custom_components.postnl
.
.
.
2019-12-15 16:04:59 WARNING (MainThread) [homeassistant.loader] You are using a custom integration for postnl which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant.
.
.
.
2019-12-15 16:05:00 ERROR (SyncWorker_7) [custom_components.postnl.sensor] Can't connect to the PostNL webservice
Traceback (most recent call last):
  File "/Users/robert/.homeassistant/deps/lib/python/site-packages/postnl_api/postnl_api.py", line 179, in _request_login
    data = response.json()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/robert/.homeassistant/custom_components/postnl/sensor.py", line 46, in setup_platform
    api = PostNL_API(username, password)
  File "/Users/robert/.homeassistant/deps/lib/python/site-packages/postnl_api/postnl_api.py", line 57, in __init__
    self._request_login()
  File "/Users/robert/.homeassistant/deps/lib/python/site-packages/postnl_api/postnl_api.py", line 182, in _request_login
    raise (UnauthorizedException())
postnl_api.postnl_api.UnauthorizedException

[metbril]

As you can see @iMicknl it isn't working for me. 😢

[metbril]

I have also tried to install the lib through pypi and that isn't working either. Same error message.

[0x00-0xFF]

It definitely is working for me though, a few days ago I had to fix the code again -> #20

[metbril]

I tried again:

$ pip3 install postnl-api
$ python3 -m postnl_api.test_postnl_api someone@example.org PaSsW0rdwithlettersandnumbersonly

Error again. Tried with my own internet, a VPN server in NL and one abroad.
If it is working for some, I can only assume blocking takes place at the user level.
I created a completely new account and tried again. Same error.

[0x00-0xFF]

My latest fix in PR20 is not merged yet, and also not on pypi.
#22

[metbril]

#17 (comment):

Clone this repo and add it to your your custom components and let us know if it's working for you as well.

I'm willing to give it a try, but how should I clone the API itself (not the postnl HA component) as a custom component?