Change `requests` to `httpx` #208

camilamaia · 2020-07-13T14:48:49Z

Change requests to httpx https://github.com/encode/httpx

Features

HTTPX builds on the well-established usability of requests, and gives you:

A broadly requests-compatible API.
Standard synchronous interface, but with async support if you need it.
HTTP/1.1 and HTTP/2 support.
Ability to make requests directly to WSGI applications or ASGI applications.
Strict timeouts everywhere.
Fully type annotated.
99% test coverage.

Plus all the standard features of requests...

International Domains and URLs
Keep-Alive & Connection Pooling
Sessions with Cookie Persistence
Browser-style SSL Verification
Basic/Digest Authentication
Elegant Key/Value Cookies
Automatic Decompression
Automatic Content Decoding
Unicode Response Bodies
Multipart File Uploads
HTTP(S) Proxy Support
Connection Timeouts
Streaming Downloads
.netrc Support
Chunked Requests

The text was updated successfully, but these errors were encountered:

tpiekarski · 2020-07-16T17:30:59Z

Migrated recently HTTP requests from requests to aiohttp and looking forward to migrate some more requests to httpx this time.
Would like to take a look and migrate these in the next week to prepare myself for coming sprint at EuroPython :)
Did you start to change to httpx?

camilamaia · 2020-07-16T20:44:48Z

Migrated recently HTTP requests from requests to aiohttp and looking forward to migrate some more requests to httpx this time.
Would like to take a look and migrate these in the next week to prepare myself for coming sprint at EuroPython :)
Did you start to change to httpx?

This would be awesome! I didn't have the chance to start tackling this issue, so you can go ahead 🚀

If you have any troubles running it locally, please let me know. We are moving to poetry and soon it will be way easier to configure your development env.

tpiekarski · 2020-07-20T05:33:50Z

Migrated recently HTTP requests from requests to aiohttp and looking forward to migrate some more requests to httpx this time.
Would like to take a look and migrate these in the next week to prepare myself for coming sprint at EuroPython :)
Did you start to change to httpx?

This would be awesome! I didn't have the chance to start tackling this issue, so you can go ahead

That's great :)

If you have any troubles running it locally, please let me know. We are moving to poetry and soon it will be way easier to configure your development env.

Thanks, I keep that in mind.

Pradhvan · 2020-07-29T20:12:15Z

@tpiekarski are you working on this ? If not, I would be happy to take this up. 😄

tpiekarski · 2020-07-30T08:32:48Z

Please go ahead, I am out of order for some time. Thanks 😊

Pradhvan · 2020-08-05T18:26:39Z

@camilamaia on a quick overview of the issue I found requests to be used only in two files.

scanapi/scanapi/tree/request_node.py : here I can try to find an quivalent of requests.requests for httpx

        response = requests.request(
            method,
            url,
            headers=self.headers,
            params=self.params,
            json=self.body,
            allow_redirects=False,
        )

scanapi/scanapi/tree/testing_node.py:

import requests

requests is being imported but it isn't being used. But the linter is not showing it as an unused import? Am I missing something here?

Besides those, a couple of test files and of what I saw for tests requests is only used for GET. So for tests, it would be an easy replacement I am assuming.

scanapi/tests/unit/test_hide_utils.py
scanapi/tests/unit/test_scan.py
scanapi/tests/unit/test_utils.py
scanapi/tests/unit/evaluators/test_code_evaluator.py

marcuxyz · 2020-08-06T19:40:33Z

@camilamaia on a quick overview of the issue I found requests to be used only in two files.

scanapi/scanapi/tree/request_node.py : here I can try to find an quivalent of requests.requests for httpx
        response = requests.request(
            method,
            url,
            headers=self.headers,
            params=self.params,
            json=self.body,
            allow_redirects=False,
        )
scanapi/scanapi/tree/testing_node.py:
import requests
requests is being imported but it isn't being used. But the linter is not showing it as an unused import? Am I missing something here?

Besides those, a couple of test files and of what I saw for tests requests is only used for GET. So for tests, it would be an easy replacement I am assuming.

scanapi/tests/unit/test_hide_utils.py

scanapi/tests/unit/test_scan.py

scanapi/tests/unit/test_utils.py

scanapi/tests/unit/evaluators/test_code_evaluator.py

to remove imports and unused variables. I recommend:

https://pypi.org/project/autoflake/#:~:text=To%20remove%20all%20unused%20imports,remove-unused-variables%20option.

camilamaia · 2020-08-13T17:16:46Z

I tested changing requests to httpx, but we have one problem. Curlify does not work with httpx:

Loading file ../examples/demo-api/.scanapi.yaml
Loading file ../examples/demo-api/scanapi.yaml
Writing documentation
Making request GET http://demo.scanapi.dev/api/health/
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/camilamaia/workspace/scanapi-org/scanapi/scanapi/__main__.py", line 62, in run
    scan()
  File "/Users/camilamaia/workspace/scanapi-org/scanapi/scanapi/scan.py", line 50, in scan
    write_report(results)
  File "/Users/camilamaia/workspace/scanapi-org/scanapi/scanapi/scan.py", line 63, in write_report
    reporter.write(results)
  File "/Users/camilamaia/workspace/scanapi-org/scanapi/scanapi/reporter.py", line 27, in write
    content = render(template_path, context, has_external_template)
  File "/Users/camilamaia/workspace/scanapi-org/scanapi/scanapi/template_render.py", line 13, in render
    return chosen_template.render(**context)
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/Users/camilamaia/workspace/scanapi-org/scanapi/scanapi/templates/html.jinja", line 408, in top-level template code
    <p>{{request|curlify}}</p>
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/curlify.py", line 25, in to_curl
    if request.body:
AttributeError: 'Request' object has no attribute 'body'

https://github.com/ofw/curlify/blob/master/curlify.py#L27

I only changed this to this:

import httpx
....
response = httpx.request(
            method,
            url,
            headers=self.headers,
            params=self.params,
            json=self.body,
            allow_redirects=False,
        )

If I remove this line, everything works properly.

About the differences between requests and httpx: https://www.python-httpx.org/compatibility/

marcorichetta · 2021-06-23T00:32:55Z

File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/curlify.py", line 25, in to_curl
if request.body:
AttributeError: 'Request' object has no attribute 'body'
https://github.com/ofw/curlify/blob/master/curlify.py#L27

I only changed [this](https://github.com/scanapi/scanapi/blob/97c4451c6b4a52d13f32c617053bbfe8cecdc75f/scanapi/tree/request_node.py#L99) to this:

```python
import httpx
....
response = httpx.request(
            method,
            url,
            headers=self.headers,
            params=self.params,
            json=self.body,
            allow_redirects=False,
        )
If I remove this line, everything works properly.

It seems curlify doesn't support httpx at the moment, specifically:

httpx does not have a .body but contents can be accessed via .read() instead.

I think the best option is to use this enhanced (and updated) version of curlify https://github.com/marcuxyz/curlify2, otherwise it'll be needed to maybe fork (?) curlify or modify it's use inside this project.

scanapi/scanapi/reporter.py

Line 23 in eba2ecd

env.filters["curlify"] = curlify.to_curl

camilamaia · 2021-06-30T18:16:51Z

@marcorichetta great catch. And in fact we are already using curlify2:

https://github.com/scanapi/scanapi/blob/master/pyproject.toml#L23

Actually, @marcuxyz created this curlify2 exactly for this reason!

marcorichetta · 2021-07-03T21:19:07Z

@marcorichetta great catch. And in fact we are already using curlify2:

https://github.com/scanapi/scanapi/blob/master/pyproject.toml#L23

Actually, @marcuxyz created this curlify2 exactly for this reason!

Oh, I read #208 (comment) and thought it was unresolved.

So this issue is already solved by PR #330, right?

camilamaia · 2021-07-05T14:42:35Z

@marcorichetta yes, exactly, the curl issue was solved. But we are not using httpx yet. We are still using requests.

taconi · 2022-11-27T12:17:53Z

@Pradhvan are you working on this ? If not, I would be happy to take this up

Pradhvan · 2022-11-28T17:22:18Z

@Pradhvan are you working on this ? If not, I would be happy to take this up

no, feel free to take it. 😄

camilamaia added the Feature New feature or request label Jul 13, 2020

djalmaaraujo mentioned this issue Jul 22, 2020

Proxy Support #215

Open

camilamaia added the Breaking Change It will break the previous versions of ScanAPI label Jul 25, 2020

marcuxyz self-assigned this Oct 1, 2020

Pradhvan assigned taconi and unassigned marcuxyz Nov 28, 2022

taconi mentioned this issue Nov 28, 2022

feat: change requests to httpx #557

Merged

8 tasks

camilamaia closed this as completed in #557 May 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change `requests` to `httpx` #208

Change `requests` to `httpx` #208

camilamaia commented Jul 13, 2020 •

edited

tpiekarski commented Jul 16, 2020

camilamaia commented Jul 16, 2020

tpiekarski commented Jul 20, 2020

Pradhvan commented Jul 29, 2020 •

edited

tpiekarski commented Jul 30, 2020

Pradhvan commented Aug 5, 2020

marcuxyz commented Aug 6, 2020

camilamaia commented Aug 13, 2020 •

edited

marcorichetta commented Jun 23, 2021

camilamaia commented Jun 30, 2021

marcorichetta commented Jul 3, 2021

camilamaia commented Jul 5, 2021

taconi commented Nov 27, 2022

Pradhvan commented Nov 28, 2022

Change requests to httpx #208

Change requests to httpx #208

Comments

camilamaia commented Jul 13, 2020 • edited

Features

tpiekarski commented Jul 16, 2020

camilamaia commented Jul 16, 2020

tpiekarski commented Jul 20, 2020

Pradhvan commented Jul 29, 2020 • edited

tpiekarski commented Jul 30, 2020

Pradhvan commented Aug 5, 2020

marcuxyz commented Aug 6, 2020

camilamaia commented Aug 13, 2020 • edited

marcorichetta commented Jun 23, 2021

camilamaia commented Jun 30, 2021

marcorichetta commented Jul 3, 2021

camilamaia commented Jul 5, 2021

taconi commented Nov 27, 2022

Pradhvan commented Nov 28, 2022

Change `requests` to `httpx` #208

Change `requests` to `httpx` #208

camilamaia commented Jul 13, 2020 •

edited

Pradhvan commented Jul 29, 2020 •

edited

camilamaia commented Aug 13, 2020 •

edited