Skip to content
This repository has been archived by the owner on Sep 7, 2023. It is now read-only.

Httpx networks #2727

Closed
wants to merge 16 commits into from
Closed

Httpx networks #2727

wants to merge 16 commits into from

Conversation

dalf
Copy link
Contributor

@dalf dalf commented Apr 5, 2021

What does this PR do?

settings.yml:

  • outgoing.networks:

    • can contains network definition
    • propertiers: enable_http, verify, http2, max_connections, max_keepalive_connections,
      keepalive_expiry, local_addresses, support_ipv4, support_ipv6, proxies, max_redirects, retries
    • retries: 0 by default, number of times searx retries to send the HTTP request (using different IP & proxy each time)
    • local_addresses can be "192.168.0.1/24" (it supports IPv6)
    • support_ipv4 & support_ipv6: both True by default
      see Source ip select #1034
  • each engine can define a "network" section:

    • either a full network description
    • either reference an existing network
  • all HTTP requests of engine use the same HTTP configuration (it was not the case before, see proxy configuration in master)

Why is this change important?

See https://github.com/searx/searx/wiki/Milestones#milestone-14---network

How to test this PR locally?

Tests need to be rewritten.

Author's checklist

Related issues

dalf added 16 commits March 22, 2021 18:51
Catch exceptions when httpx tries to close an existing already closed connection
Note: error appears after a long running instance on the wikipedia engine.
settings.yml:

* outgoing.networks:
   * can contains network definition
   * propertiers: enable_http, verify, http2, max_connections, max_keepalive_connections,
     keepalive_expiry, local_addresses, support_ipv4, support_ipv6, proxies, max_redirects, retries
   * retries: 0 by default, number of times searx retries to send the HTTP request (using different IP & proxy each time)
   * local_addresses can be "192.168.0.1/24" (it supports IPv6)
   * support_ipv4 & support_ipv6: both True by default
     see searx#1034
* each engine can define a "network" section:
   * either a full network description
   * either reference an existing network

* all HTTP requests of engine use the same HTTP configuration (it was not the case before, see proxy configuration in master)
@dalf dalf closed this Apr 22, 2021
@dalf dalf deleted the httpx_networks branch April 27, 2021 06:54
@kvch kvch mentioned this pull request May 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant