Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: Request.from_curl() with $-prefixed string literals #5901

Merged
merged 7 commits into from
Apr 19, 2023

Conversation

jxlil
Copy link
Contributor

@jxlil jxlil commented Apr 19, 2023

Hi @wRAR

I added an action for the --data argument that removes the string prefix and decodes the string. Although Request.from_curl() now works correctly with requests like the one you shared in the issue, I don't know if this is the best way to approach this.

curl_command = "curl 'https://www.example.org/' --data-raw $'{\"$filters\":\"Filter\u0021\"}'"
curl_to_request_kwargs(curl_command)

# output: {'method': 'POST', 'url': 'https://www.example.org/', 'body': '{"$filters":"Filter!"}'}

Closes #5899

@codecov
Copy link

codecov bot commented Apr 19, 2023

Codecov Report

Merging #5901 (2b5a4a3) into master (02f3e8d) will decrease coverage by 0.47%.
The diff coverage is 100.00%.

❗ Current head 2b5a4a3 differs from pull request most recent head f69ba43. Consider uploading reports for the commit f69ba43 to get more accurate results

@@            Coverage Diff             @@
##           master    #5901      +/-   ##
==========================================
- Coverage   88.77%   88.30%   -0.47%     
==========================================
  Files         162      162              
  Lines       11159    11058     -101     
  Branches     1813     1638     -175     
==========================================
- Hits         9906     9765     -141     
- Misses        966      997      +31     
- Partials      287      296       +9     
Impacted Files Coverage Δ
scrapy/utils/curl.py 100.00% <100.00%> (ø)

... and 70 files with indirect coverage changes

scrapy/utils/curl.py Outdated Show resolved Hide resolved
jxlil and others added 2 commits April 19, 2023 01:04
Co-authored-by: Adrián Chaves <adrian@chaves.io>
@jxlil jxlil marked this pull request as ready for review April 19, 2023 07:45
@wRAR
Copy link
Member

wRAR commented Apr 19, 2023

Hmm this only handles the escapes that work in Python strings (\uHHHH like in the example, likely some others), but it's nice that it does this without using any additional libraries, so I'm fine with merging it.

@wRAR wRAR merged commit abc9c1a into scrapy:master Apr 19, 2023
@jxlil jxlil deleted the fix/5899 branch April 21, 2023 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Request.from_curl() with $-prefixed string literals
3 participants