Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🤟 Fix user-agent banning by ESIOS server #35

Merged
merged 8 commits into from Dec 1, 2021
Merged

Conversation

azogue
Copy link
Owner

@azogue azogue commented Dec 1, 2021

closes #34

related to home-assistant/core#60556

Quick-Fix motivated by the last change in the ESIOS server (on 2021-11-30 😱), which is now apparently banning HomeAssistant requests, filtering us out because of the 'User-Agent' headers data 😤, as the server is returning a 403 status code error for a PUBLIC url 🤷.

So, until future moves to authenticated endpoints, or alternative data sources, let's continue playing 🐈 and 🐁 with the ESIOS webadmins 🤣🤣

[Serious note for those webadmins]: I'm just a volunteer, and I'm an email away for contact, so please, if you are reading this, give me a call before filtering us all out 🤝🥺

Changes:

  • Substantially decrease the number of API requests to ESIOS, avoiding unnecessary calls to refresh data for the same electricity prices.
    Before, when used from the pvpc_hourly_pricing HA Core integration, the ESIOS API was called 2 times/hour from 0h to 20h, and 4 times/hour in the evening, from 20h to 0h, retrieving today + tomorrow prices.
    This makes a total of ~56 requests/day, which is not a lot 😅, but it seems the aggregated total for the HA user base (🔥 >30k requests/day just counting users pushing HA analytics) is being some kind of a problem for ESIOS,
    as it looks like they're trying to bane us 🥺😭
    Now, the API handler avoids calls to retrieve already available prices, cutting down the number of requests to just 1-2 requests/day 🤩

  • 🐛 Set standard User-Agent header info, to try to avoid server-side banning 🙈, and rotate it if banning is detected, using common User-Agent browser identifiers.

  • ♻️ Minor code refactor to prepare for future library changes, in order to move to authenticated API endpoints in future versions.

@azogue azogue merged commit 68c7637 into master Dec 1, 2021
@azogue
Copy link
Owner Author

azogue commented Dec 1, 2021

🚀 For those who cannot wait until the fix is integrated and published in HA Core, just unpack the following ZIP file in your config/custom_components folder, so your HomeAssistant instance with the PVPC price sensor could work again after a simple restart (using python3.9 and HA Core >= 2021.11.0!)

🚀 pvpc_hourly_pricing.zip

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration 🧪, try this one instead 😜: esios.zip

@deltazerorsan
Copy link

🚀 For those who cannot wait until the fix is integrated and published in HA Core, just unpack the following ZIP file in your config/custom_components folder, so your HomeAssistant instance with the PVPC price sensor could work again after a simple restart (using python3.9 and HA Core >= 2021.11.0!)

🚀 pvpc_hourly_pricing.zip

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration 🧪, try this one instead 😜: esios.zip

The custom pvpc is running, thanks.

I'm also trying esios, and I got a token today, but I can't see the Esios in the integration list... (yes the folder is inside custom_components)

HA over raspi, core-2021.11.5.

Any directions appreciated!

@martinezpenya
Copy link

rocket For those who cannot wait until the fix is integrated and published in HA Core, just unpack the following ZIP file in your config/custom_components folder, so your HomeAssistant instance with the PVPC price sensor could work again after a simple restart (using python3.9 and HA Core >= 2021.11.0!)

rocket pvpc_hourly_pricing.zip

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration test_tube, try this one instead stuck_out_tongue_winking_eye: esios.zip

Yes! it works again with custom pvpc.

Tomorrow I'll probably get the esios token and test the other one.
Thank you!.

@jjprz
Copy link

jjprz commented Dec 1, 2021

🚀 For those who cannot wait until the fix is integrated and published in HA Core, just unpack the following ZIP file in your config/custom_components folder, so your HomeAssistant instance with the PVPC price sensor could work again after a simple restart (using python3.9 and HA Core >= 2021.11.0!)

🚀 pvpc_hourly_pricing.zip

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration 🧪, try this one instead 😜: esios.zip

Testing...

Logger: custom_components.esios.esios_data.api
Source: custom_components/esios/esios_data/api.py:242
Integration: ESIOS API Data (documentation, issues)
First occurred: 00:00:44 (1 occurrences)
Last logged: 00:00:44

2021-12-01 23:00:00+00:00 not found in {} -> list index out of range

@bvis
Copy link

bvis commented Dec 2, 2021

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration 🧪, try this one instead 😜: esios.zip

I have now an ESIOS token, do you have instructions about its configuration with this esios.zip version?

@amibumping
Copy link

amibumping commented Dec 2, 2021

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration test_tube, try this one instead stuck_out_tongue_winking_eye: esios.zip

I have now an ESIOS token, do you have instructions about its configuration with this esios.zip version?

Go to integrations, add new one, look up for esios api, and then enter your token and save.
I didn't test it myself yet.

EDIT: working

@jeancsamin
Copy link

Applying the quick fix using token ends up in REE forbidding the API call again. Not sure how the code works, but if it triggers many updates while the price curves don't change, i.e. we are pulling the same data over and over again, they end up banning the access via token as well.

@fernandodpr
Copy link

I'm now using the new ESIOS integration. It works!! 😃🥳. Congratulations for your work and will inform you in case of any issues.
Could you provide the GitHub link of the new integration?

@deltazerorsan
Copy link

I'm now using the new ESIOS integration. It works!! 😃🥳. Congratulations for your work and will inform you in case of any issues. Could you provide the GitHub link of the new integration?

Hi @fernandodpr,

I'm trying to install ESIOS as a custom component but the integration do not appear on the list, Is there any special procedure I'm missing?

Thanks!

@fernandodpr
Copy link

fernandodpr commented Dec 2, 2021

Hi @deltazerorsan,
I followed the instructions provided by @azogue :

  1. Downloaded the zip file " esios.zip" and extracted in the folder config/custom_components. This should create a folder called esios.
  2. Reboot your HA instance.
  3. Go to Configuration -> Integrations-> Press the '+ ADD INTEGRATION' button -> Search for "ESIOS API Data"
  4. Add the integration, by filling the required inputs. API TOKEN, location and a few more checkboxes.
  5. All done! Search for your sensor, mine It's called sensor.esios_pvpc

@jumavi
Copy link

jumavi commented Dec 2, 2021

Hola,
La integración con el token personal funciona perfecto. He desintalado la versión oficial y obviamente reiniciado pero me siguen apareciendo dos errores. ¿que puede ser?

Logger: aiopvpc.pvpc_data
Source: /usr/local/lib/python3.9/site-packages/aiopvpc/pvpc_data.py:231
First occurred: 1 de diciembre de 2021 21:57:38 (96 occurrences)
Last logged: 16:27:19

Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-03-28' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '859', 'X-Iinfo': '7-36927098-0 2NNN RT(1638458838950 41) q(0 -1 -1 1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'visid_incap_1885724=KidT3bTWTI2TGT+AnDKTHtblqGEAAAAAQUIPAAAAAABu0gwW9yWPEzn0GXj7gbnN; expires=Thu, 01 Dec 2022 17:05:24 GMT; HttpOnly; path=/; Domain=.esios.ree.es; Secure; SameSite=None', 'Set-Cookie': 'incap_ses_510_1885724=EKq/CL/EZERT51dhzOITB9blqGEAAAAAouTHo/Y/3dMijGy+Jb4gqg==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-12-01' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '860', 'X-Iinfo': '6-22198842-0 2NNN RT(1638458839404 33) q(0 -1 -1 -1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'visid_incap_1885724=m3ogOnXBSIijujykXcxQLdflqGEAAAAAQUIPAAAAAADF1cT7IWYCtULlP3xTCLj5; expires=Thu, 01 Dec 2022 17:05:24 GMT; HttpOnly; path=/; Domain=.esios.ree.es; Secure; SameSite=None', 'Set-Cookie': 'incap_ses_510_1885724=7SSdJDKJmWgr6FdhzOITB9flqGEAAAAA0ZxM5jMuqdXVRK1Rw9yxvw==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '860', 'X-Iinfo': '4-73046775-0 2NNN RT(1638458839401 42) q(0 -1 -1 -1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'visid_incap_1885724=20eopxSuSEeSU7o8Ou/Y0dflqGEAAAAAQUIPAAAAAACScp5Jrd363bO15qK7STBU; expires=Thu, 01 Dec 2022 17:04:13 GMT; HttpOnly; path=/; Domain=.esios.ree.es; Secure; SameSite=None', 'Set-Cookie': 'incap_ses_510_1885724=kjx2QI2gXikx6FdhzOITB9flqGEAAAAAMxelCM76E5mJNSfGwzFBZw==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-30' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '860', 'X-Iinfo': '4-73046778-0 2NNN RT(1638458839404 48) q(0 -1 -1 -1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'visid_incap_1885724=lurvS6o6QGq2hXiMd8GQH9flqGEAAAAAQUIPAAAAAACqK3BpygLpPOEmLOchKK6N; expires=Thu, 01 Dec 2022 17:04:13 GMT; HttpOnly; path=/; Domain=.esios.ree.es; Secure; SameSite=None', 'Set-Cookie': 'incap_ses_510_1885724=sVE9LNpD1Ws16FdhzOITB9flqGEAAAAAXbDb0j6H+CmpxtEaJiNVsw==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-12-02' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '861', 'X-Iinfo': '10-86545175-0 2cNN RT(1638458839406 66) q(0 -1 -1 -1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'visid_incap_1885724=EXkD5c+USy+k138g+QWOB9flqGEAAAAAQUIPAAAAAADMaBYOj1gZR3PctkN7uz5E; expires=Thu, 01 Dec 2022 17:04:13 GMT; HttpOnly; path=/; Domain=.esios.ree.es; Secure; SameSite=None', 'Set-Cookie': 'incap_ses_510_1885724=+gt6Q9ixgxpB6FdhzOITB9flqGEAAAAAycP95M+uQqkHT8QQaMnomg==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>

Logger: aiopvpc.pvpc_data
Source: /usr/local/lib/python3.9/site-packages/aiopvpc/pvpc_data.py:425
First occurred: 1 de diciembre de 2021 21:57:38 (38 occurrences)
Last logged: 16:27:19

BAD Download of PVPC prices from 2021-11-29 09:00:00 to 2021-12-02 13:27:15.397859 in 0.13 sec
BAD Download of PVPC prices from 2021-11-29 09:00:00 to 2021-12-02 14:27:15.400131 in 0.13 sec
BAD Download of PVPC prices from 2021-11-29 09:00:00 to 2021-12-02 15:27:15.401872 in 0.26 sec
BAD Download of PVPC prices from 2021-03-28 01:00:00 to 2021-03-28 03:00:00 in 0.17 sec
BAD Download of PVPC prices from 2021-11-29 09:00:00 to 2021-12-02 16:27:15.403588 in 0.14 sec

@martinezpenya
Copy link

Hi @deltazerorsan, I followed the instructions provided by @azogue :

1. Downloaded the zip file " esios.zip" and extracted in the folder config/custom_components. This should create a folder called **esios**.

2. Reboot your HA instance.

3. Go to Configuration -> Integrations-> Press the '+ ADD INTEGRATION' button -> Search for "_ESIOS API Data_"

4. Add the integration, by filling the required inputs. [API TOKEN](https://api.esios.ree.es/), location and a few more checkboxes.

5. All done! Search for your sensor, mine It's called **sensor.esios_pvpc**

I've followed this steps with my personal token and is working!.
Thank you!

@deltazerorsan
Copy link

deltazerorsan commented Dec 2, 2021

SOLVED, IT WAS A PROBLEM IN HA WITH ANY MODULE

ESIOS WORKING PROPERLY:

Hi @deltazerorsan, I followed the instructions provided by @azogue :

  1. Downloaded the zip file " esios.zip" and extracted in the folder config/custom_components. This should create a folder called esios.
  2. Reboot your HA instance.
  3. Go to Configuration -> Integrations-> Press the '+ ADD INTEGRATION' button -> Search for "ESIOS API Data"
  4. Add the integration, by filling the required inputs. API TOKEN, location and a few more checkboxes.
  5. All done! Search for your sensor, mine It's called sensor.esios_pvpc

Yeah! My problem is between step 2 and 3... no ESIOS API Data... Are you using a development version of HA?

Thanks

@fernandodpr
Copy link

.... ESIOS API Data... Are you using a development version of HA?

No, I'm using standard version of HA. Could you look into /config/info. To check if HA It's detecting the integration?

@deltazerorsan
Copy link

deltazerorsan commented Dec 2, 2021

.... ESIOS API Data... Are you using a development version of HA?

No, I'm using standard version of HA. Could you look into /config/info. To check if HA It's detecting the integration?

Nope, it's not there... I have other three custom components, including pvpc, working correctly... mmm, so estrange...

I got the zip from here: https://github.com/azogue/aiopvpc/files/7531844/esios.zip

@etatus
Copy link

etatus commented Dec 3, 2021

Is there any official repository for the API version of pvpc_hourly_pricing integration (ESIOS API connector)

The one referenced in the integration (esios.zip) doesn't seem to work:
https://github.com/azogue/ha-esios

@josecarre
Copy link

🚀 For those who cannot wait until the fix is integrated and published in HA Core, just unpack the following ZIP file in your config/custom_components folder, so your HomeAssistant instance with the PVPC price sensor could work again after a simple restart (using python3.9 and HA Core >= 2021.11.0!)

🚀 pvpc_hourly_pricing.zip

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration 🧪, try this one instead 😜: esios.zip

gracias :), llevaba varios días usando velas por no saber el precio! xD.
Ya vuelve a funcionar con el aiopvpc 2.3.0 gracias :), estoy a la espera de recibir el token de REE, cuando lo tenga me paso a la nueva/esios. Se seguirían usando los mismos sensores en HA? por mantener el histórico de precios, o son sensores nuevos? thanks!

@fernandodpr
Copy link

@josecarre el sensor nuevo por defecto tiene otra id. Lo que puedes hacer es modificar la id una vez instalado y colocar la misma que tenías anteriormente (recuerda eliminar la otra integración). De esa forma en los gráficos tendrás histórico, en grafana también en caso de que lo uses y tampoco tendrás que modificar tus automatizaciones.

@mrtmsgh
Copy link

mrtmsgh commented Dec 11, 2021

rocket For those who cannot wait until the fix is integrated and published in HA Core, just unpack the following ZIP file in your config/custom_components folder, so your HomeAssistant instance with the PVPC price sensor could work again after a simple restart (using python3.9 and HA Core >= 2021.11.0!)

rocket pvpc_hourly_pricing.zip

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration test_tube, try this one instead stuck_out_tongue_winking_eye: esios.zip

I've tried the two options and both are working OK for me. Thanks!

@maarlo
Copy link

maarlo commented Dec 20, 2021

I have tried esios.zip and works properly. Thanks!

@samolo75
Copy link

I tried also esios.zip with the personal token and it works. But it only shows the price on the day, not for the next day. Am I alone?

@deltazerorsan
Copy link

deltazerorsan commented Dec 21, 2021 via email

@samolo75
Copy link

You need to wait until 20:00 ... 20:30 and then you have next day. Untill 00:00. It works fine for me. El mar., 21 dic. 2021 12:31, samolo75 @.> escribió:

I tried also esios.zip with the personal token and it works. But it only shows the price on the day, not for the next day. Am I alone? — Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM33FISVUM6M7DDNMHDGW2DUSBQQ5ANCNFSM5JFLYPSQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.
>

I am pretty sure I did check late on the day, for a few days and I never got the next day prices. I will keep checking. Thanks

@deltazerorsan
Copy link

Yeah it's a little trickier cause the attributes for next day are present only between the time REE offers them and 00:00.

Right now 22:50 CET I can see the "next day" attributes, like:

min_price (next day): 0.42475 min_price_at (next day): 4

but later they disappear and you get an Unknown wherever you are using it... But tomorrow they reappear whit the values of the next day, and so on.

@samolo75
Copy link

Yeah it's a little trickier cause the attributes for next day are present only between the time REE offers them and 00:00.

Right now 22:50 CET I can see the "next day" attributes, like:

min_price (next day): 0.42475 min_price_at (next day): 4

but later they disappear and you get an Unknown wherever you are using it... But tomorrow they reappear whit the values of the next day, and so on.

Ok. Sorry. It actually works fine when I checked yesterday. I guess I didn't do it right before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

sensor.pvpc: repeated bad data update, mark component as unavailable source