Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

close(): fix exception and enable fast-close #156

Merged
merged 4 commits into from
Mar 5, 2024
Merged

close(): fix exception and enable fast-close #156

merged 4 commits into from
Mar 5, 2024

Conversation

bablokb
Copy link
Contributor

@bablokb bablokb commented Feb 28, 2024

The current code throws an exception during close() when it tries to convert an empty buffer (chunk_header).
This PR also adds an optional parameter fast=False. Passing True will skip reading.

This addresses #130 (also mentioned in #138)

@tannewt
Copy link
Member

tannewt commented Feb 29, 2024

Let's not add it to close() because that changes it from the Requests API: https://requests.readthedocs.io/en/latest/api/#requests.Response.close

Instead, please add it as a kwarg on Session(). The Session() API already varies from CPython.

Copy link
Member

@tannewt tannewt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move kwarg to fast_close kwarg in Session().

@bablokb
Copy link
Contributor Author

bablokb commented Mar 1, 2024

Maybe I am missing something, but this would imply that I

  • add a kwarg to Session()
  • the constructor saves this as a public property
  • Response.__init__() will query the property from the Session-object passed to it and save it as an instance attribute
  • Response.close() uses this instance attribute

This seems rather involved. In my solution, I added an optional argument to Response.close(), so as long as you don't use it, the code is compatible. And the whole Response class is not compatible after all, there are many differences already.

But probably the whole drain-the-buffer-code should not be here after all, but somewhere where the socket is actually closed. Until this is implemented, I see both of my changes as a workaround. Instead of changing the code in many places, it might be wiser to just remove the additional option and wait for the real solution.

@tannewt
Copy link
Member

tannewt commented Mar 1, 2024

And the whole Response class is not compatible after all, there are many differences already.

What are these differences? The intent is that the same code will work with CPython's request.

This seems rather involved.

It is a bit of plumbing but it isn't complex and it preserves the API. I'd have session pass it into the response object as a separate kwarg.

In my solution, I added an optional argument to Response.close(), so as long as you don't use it, the code is compatible.

"so as long as you don't use it" is a big caveat. We shouldn't make it possible to write incompatible code so you don't need to worry about it or need to redo it later.

@bablokb
Copy link
Contributor Author

bablokb commented Mar 2, 2024

What are these differences? The intent is that the same code will work with CPython's request.

e.g. the constructor of Response in CPython does not take any arguments.

I'd have session pass it into the response object as a separate kwarg.

What is the fundamental difference between adding an additional kwarg to the Response constructor compared to adding a kwarg to Response.close()?

Anyhow, I will do what you suggested to get the fix for the chunk_size exception merged, that is the most important problem.

@tannewt
Copy link
Member

tannewt commented Mar 4, 2024

What are these differences? The intent is that the same code will work with CPython's request.

e.g. the constructor of Response in CPython does not take any arguments.

I'd have session pass it into the response object as a separate kwarg.

What is the fundamental difference between adding an additional kwarg to the Response constructor compared to adding a kwarg to Response.close()?

Response objects aren't usually created by user code. Instead, they are constructed by the library internally and returned. To me, that makes the constructor more private than close.

Anyhow, I will do what you suggested to get the fix for the chunk_size exception merged, that is the most important problem.

Thanks!

@dhalbert dhalbert requested a review from tannewt March 5, 2024 14:56
Copy link
Member

@tannewt tannewt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@tannewt tannewt merged commit 815b326 into adafruit:main Mar 5, 2024
1 check passed
@justmobilize
Copy link
Collaborator

@bablokb what is your thought around fast_close? I understand the chunk bug fix, but don't fully understand this one.

@dhalbert
Copy link
Contributor

Is there a CPython requests equivalent to the fast_close functionality, or is there something different about our requests semantics?

@justmobilize
Copy link
Collaborator

So I know we have some open issues - the one this fixed #130 and #153. I am working through those and testing across boards since the former mentioned issues with ESP32SPI (which I don't know if this one fixes or breaks)

@bablokb
Copy link
Contributor Author

bablokb commented Mar 11, 2024

@bablokb what is your thought around fast_close? I understand the chunk bug fix, but don't fully understand this one.

There are streams that never end (e.g. internet radio mp3-streams). In this case reading the socket until it is empty will never succeed.

@justmobilize
Copy link
Collaborator

Was there a reason for not just fixing close to not keep reading? Since it really shouldn't

@bablokb
Copy link
Contributor Author

bablokb commented Mar 11, 2024

These were two distinct and unrelated errors:

  • empty chunks triggered a syntax exception. This will show up for streams with finite content
  • infinite streams don't have empty chunks, but would never close

So my error was that I fixed both in a single PR. It was so simple but judging by all the discussions this already created it probably was a bad idea.

@dhalbert
Copy link
Contributor

I can revert the PR if it makes sense, or if you can just rebase and fix, @justmobilize, that's fine too.

@justmobilize
Copy link
Collaborator

@dhalbert I should have my PR ready by Wednesday. I can do either:

  1. include the empty_chucks and fix and not having close keep reading read
  2. remove the fast_close and fix and not having close keep reading read
    Up to you on what's best (revert now or wait)

@dhalbert
Copy link
Contributor

@justmobilize Do 2., so that we have at least one fix for now.

@justmobilize
Copy link
Collaborator

Sounds good

@justmobilize
Copy link
Collaborator

@bablokb #159 removes all fetching anything on close. So should be good. Please let me know if you run into any issues

@bablokb
Copy link
Contributor Author

bablokb commented Mar 13, 2024

Tested it with 9.0.0-rc0 and a Pico-W and it works fine.

adafruit-adabot added a commit to adafruit/Adafruit_CircuitPython_Bundle that referenced this pull request Mar 22, 2024
Updating https://github.com/adafruit/Adafruit_CircuitPython_AGS02MA to 1.0.8 from 1.0.7:
  > Merge pull request adafruit/Adafruit_CircuitPython_AGS02MA#4 from FoamyGuy/fix_readme

Updating https://github.com/adafruit/Adafruit_CircuitPython_DisplayIO_SSD1306 to 2.0.0 from 1.8.0:
  > Merge pull request adafruit/Adafruit_CircuitPython_DisplayIO_SSD1306#39 from makermelissa/master

Updating https://github.com/adafruit/Adafruit_CircuitPython_ESP32SPI to 7.1.0 from 7.0.1:
  > Merge pull request adafruit/Adafruit_CircuitPython_ESP32SPI#196 from anecdata/exit
  > Merge pull request adafruit/Adafruit_CircuitPython_ESP32SPI#192 from mMerlin/new-sample
  > Merge pull request adafruit/Adafruit_CircuitPython_ESP32SPI#193 from justmobilize/fix-readme-requirements

Updating https://github.com/adafruit/Adafruit_CircuitPython_FT5336 to 1.0.2 from 1.0.1:
  > Merge pull request adafruit/Adafruit_CircuitPython_FT5336#3 from adafruit/buttons_example

Updating https://github.com/adafruit/Adafruit_CircuitPython_GPS to 3.10.13 from 3.10.12:
  > Merge pull request adafruit/Adafruit_CircuitPython_GPS#102 from jkittner/minutes_fix

Updating https://github.com/adafruit/Adafruit_CircuitPython_HT16K33 to 4.6.9 from 4.6.7:
  > Merge pull request adafruit/Adafruit_CircuitPython_HT16K33#118 from adafruit/matrix_fill_color_swap
  > Merge pull request adafruit/Adafruit_CircuitPython_HT16K33#117 from DJDevon3/main

Updating https://github.com/adafruit/Adafruit_CircuitPython_LTR329_LTR303 to 3.0.6 from 3.0.5:
  > Merge pull request adafruit/Adafruit_CircuitPython_LTR329_LTR303#8 from FoamyGuy/fix_readme

Updating https://github.com/adafruit/Adafruit_CircuitPython_MAX31856 to 0.12.0 from 0.11.8:
  > Merge pull request adafruit/Adafruit_CircuitPython_MAX31856#28 from diegosolano/main

Updating https://github.com/adafruit/Adafruit_CircuitPython_MMC56x3 to 1.0.8 from 1.0.7:
  > Merge pull request adafruit/Adafruit_CircuitPython_MMC56x3#4 from FoamyGuy/fix_readme

Updating https://github.com/adafruit/Adafruit_CircuitPython_PCA9685 to 3.4.15 from 3.4.14:
  > Merge pull request adafruit/Adafruit_CircuitPython_PCA9685#55 from DJDevon3/WorkingBranch

Updating https://github.com/adafruit/Adafruit_CircuitPython_SI5351 to 1.4.4 from 1.4.3:
  > Merge pull request adafruit/Adafruit_CircuitPython_SI5351#31 from kamocat/main

Updating https://github.com/adafruit/Adafruit_CircuitPython_SSD1306 to 2.12.17 from 2.12.16:
  > Merge pull request adafruit/Adafruit_CircuitPython_SSD1306#85 from waynedyck/fix-pillow-10-errors

Updating https://github.com/adafruit/Adafruit_CircuitPython_TCA8418 to 1.0.11 from 1.0.10:
  > Merge pull request adafruit/Adafruit_CircuitPython_TCA8418#13 from FoamyGuy/fix_readme

Updating https://github.com/adafruit/Adafruit_CircuitPython_TSC2007 to 1.1.0 from 1.0.9:
  > Merge pull request adafruit/Adafruit_CircuitPython_TSC2007#6 from makermelissa/main

Updating https://github.com/adafruit/Adafruit_CircuitPython_TT21100 to 1.0.2 from 1.0.1:
  > Merge pull request adafruit/Adafruit_CircuitPython_TT21100#5 from FoamyGuy/fix_readme

Updating https://github.com/adafruit/Adafruit_CircuitPython_VL53L4CD to 1.2.2 from 1.2.0:
  > Merge pull request adafruit/Adafruit_CircuitPython_VL53L4CD#15 from FoamyGuy/fix_readme

Updating https://github.com/adafruit/Adafruit_CircuitPython_Wiznet5k to 5.0.9 from 5.0.7:
  > Merge pull request adafruit/Adafruit_CircuitPython_Wiznet5k#148 from pinkavaj/pi-fix-if
  > Merge pull request adafruit/Adafruit_CircuitPython_Wiznet5k#143 from us3r64/fix/socket-recv-timeout

Updating https://github.com/adafruit/Adafruit_CircuitPython_BitbangIO to 1.3.15 from 1.3.14:
  > Merge pull request adafruit/Adafruit_CircuitPython_BitbangIO#32 from kbsriram/i2c-clock-stretching
  > Merge pull request adafruit/Adafruit_CircuitPython_BitbangIO#31 from kbsriram/fix-phase

Updating https://github.com/adafruit/Adafruit_CircuitPython_HID to 6.1.1 from 6.1.0:
  > Merge pull request adafruit/Adafruit_CircuitPython_HID#125 from dhalbert/fix-class-docs

Updating https://github.com/adafruit/Adafruit_CircuitPython_NTP to 3.0.13 from 3.0.12:
  > Merge pull request adafruit/Adafruit_CircuitPython_NTP#31 from anecdata/cm_multi_radio

Updating https://github.com/adafruit/Adafruit_CircuitPython_PyCamera to 1.3.0 from 1.2.0:
  > Merge pull request adafruit/Adafruit_CircuitPython_PyCamera#33 from FoamyGuy/overlay_scale

Updating https://github.com/adafruit/Adafruit_CircuitPython_Requests to 3.2.2 from 3.0.1:
  > Merge pull request adafruit/Adafruit_CircuitPython_Requests#170 from DJDevon3/DJDevon3-TwitchAPI
  > Merge pull request adafruit/Adafruit_CircuitPython_Requests#169 from DJDevon3/DJDevon3-SteamAPI
  > Merge pull request adafruit/Adafruit_CircuitPython_Requests#166 from DJDevon3/GithubAPI
  > Merge pull request adafruit/Adafruit_CircuitPython_Requests#165 from DJDevon3/DJDevon3-PremiereLeagueAPI
  > Merge pull request adafruit/Adafruit_CircuitPython_Requests#160 from DJDevon3/main
  > Merge pull request adafruit/Adafruit_CircuitPython_Requests#164 from DJDevon3/DJDevon3-FitbitAPI
  > Merge pull request adafruit/Adafruit_CircuitPython_Requests#163 from DJDevon3/DJDevon3-DiscordAPI
  > Merge pull request adafruit/Adafruit_CircuitPython_Requests#162 from DJDevon3/DJDevon3-DiscordActive
  > Merge pull request adafruit/Adafruit_CircuitPython_Requests#161 from DJDevon3/MastodonBranch
  > Merge pull request adafruit/Adafruit_CircuitPython_Requests#159 from justmobilize/remove-response-close-read-all
  > Merge pull request adafruit/Adafruit_CircuitPython_Requests#156 from bablokb/main

Updating https://github.com/adafruit/Adafruit_CircuitPython_RGBLED to 1.2.0 from 1.1.19:
  > Merge pull request adafruit/Adafruit_CircuitPython_RGBLED#25 from BiffoBear/Error_check_colors

Updating https://github.com/adafruit/Adafruit_CircuitPython_RSA to 1.2.20 from 1.2.19:
  > Merge pull request adafruit/Adafruit_CircuitPython_RSA#40 from FoamyGuy/add_hashlib_req

Updating https://github.com/adafruit/Adafruit_CircuitPython_Bundle/circuitpython_library_list.md to NA from NA:
  > Updated download stats for the libraries
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants