Add custom http.Agent / https.Agent support ( Tor, Socks5, various proxy support )

[ghost]

fixes #2709, #2775

If you want this feature before this PR is merged, there is a custom, unofficial ethers provider to use right now https://github.com/ayanamitech/ethers-axios-provider

Why adding ability to use custom http.agent / https.agent is important?

One of the missing features of ethers.js compared with web3.js is that injecting http.agent is impossible although the web3.js have been supported using custom http.agent for more than 2 years.

Example: https://web3js.readthedocs.io/en/v1.2.11/include_package-core.html?highlight=agent#configuration

@types/node definition of RequestOptions which ethers.js use for initiating remote RPC providers

https://microsoft.github.io/PowerBI-JavaScript/interfaces/_node_modules__types_node_http_d_._http_.requestoptions.html

One of the key features of the ability to inject custom http.agent supports is that it could not only provide essential network connectivity from the censored internet environment but also secures the connection with a secure, private, and faster proxy interface without using VPN.

See also: https://www.coindesk.com/policy/2022/03/03/metamask-infura-block-certain-areas-amid-crypto-sanctions-fury/

Related issues / PR from web3.js

web3/web3.js#887

web3/web3.js#2827

web3/web3.js#2946

web3/web3.js#2980

As you could see, many chinese developers use this feature to evade network firewalls across the border.

Example use case of using web3.js with socks5 proxy ( Tor connection )

Tornado-cli ( Which supports censorship resistant, ip concealing connection with Tor Network )

https://github.com/tornadocash/tornado-cli/blob/master/cli.js#L1205

What does this PR do?

By implementing the minimal interface of http.agent which is also compatible with @types/node package ( as used by ethers.js package ), any RPC provider from ethers.js could provide remote connectivity wrapped by various type of proxies with native way without the need of routing the packets of nodejs or browser workers.

Testing this PR

This PR could be tested against various proxy agents, here is the example that connects Infura with Tor Network using the modified local built ethers.js package.

$ git clone -b custom-http-agent-support https://github.com/0xAyanami/ethers.js
$ cd ethers.js
$ npm i
$ npm i --save socks-proxy-agent
$ npm run build
$ npm i
$ nano test.js

and write this file to test.js

const { SocksProxyAgent } = require('socks-proxy-agent');
const { JsonRpcProvider } = require('./packages/providers/lib/index');

const test = async () => {
  // Infura provider URL
  const infuraUrl = "https://goerli.infura.io/v3/9aa3d95b3bc440fa88ea12eaa4456161";
  // Use Tor Browser User Agent
  const torBrowserAgent = 'Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0';
  // Default tor port for Tor Browser
  const torPort = 9150;

  const ethersOptions = {
    url: infuraUrl,
    headers: { 'User-Agent': torBrowserAgent },
    agent: { https: new SocksProxyAgent('socks5h://127.0.0.1:' + torPort) }
  }

  const provider = new JsonRpcProvider(ethersOptions);
  console.log(provider.connection);
  const [ blockNumber, getBlock ] = await Promise.all([
    provider.getBlockNumber(),
    provider.getBlock('latest')
  ]);
  console.log(blockNumber);
  console.log(getBlock);
}
test();

Change the torPort value if you are using Tor with service without installing it from Tor Browser,

Run node test.js to check if it works, also try turning off the Tor to check if the SocksProxyAgent injected with ethers.js only works with tor connection only.

Example output

{
  url: 'https://goerli.infura.io/v3/9aa3d95b3bc440fa88ea12eaa4456161',
  headers: {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0'
  },
  agent: {
    http: SocksProxyAgent {
      _events: [Object: null prototype] {},
      _eventsCount: 0,
      _maxListeners: undefined,
      timeout: null,
      maxFreeSockets: 1,
      maxSockets: 1,
      maxTotalSockets: Infinity,
      sockets: {},
      freeSockets: {},
      requests: {},
      options: {},
      lookup: false,
      proxy: [Object],
      tlsConnectionOptions: {},
      [Symbol(kCapture)]: false
    }
  }
}
6566829
{
  hash: '0xf937b8918b7f8b1241ee6e2c296a1e1f9488c105c663177ab80e3108c1ec5c1b',
  parentHash: '0x6794ae12c54db26b59bad93cd91a76d6cd5df26f9999a7e237c69485724ed46e',
  number: 6566829,
  timestamp: 1647730522,
  nonce: '0x0000000000000000',
  difficulty: 1,
  gasLimit: BigNumber { _hex: '0x01c9c364', _isBigNumber: true },
  gasUsed: BigNumber { _hex: '0x18aee5', _isBigNumber: true },
  miner: '0x0000000000000000000000000000000000000000',
  extraData: '0xd883010a0d846765746888676f312e31372e32856c696e75780000000000000035a7d2301afad39826afdab78c054d72accca8f1c9d4721d6a9848c7a34cea9065c4a37b4692623e2d2ef6a444a66573b46e4c0269c6b4b923ffcdae1af01d8a00',
  transactions: [
    '0x0d5fdd192ed877be181e38eb719b11dd6eeb1eab891d6eb6e5b77e182810d8be',
    '0xaeb30a3054e78480503620fae5290c09200c22d7be5e0f5236ff90b146ec2ae2',
    '0x7564760f0dcfeee04a9a21998cc0ea7412f664e5afaa81adafd5aa7a3f251630',
    '0x1feb97d3e025eeb1690fac0740686eb550c750533704eb19edc552e81d5ed1b2',
    '0x9dfb08e2d1638a858ae423475f09472a1a492e67a896ce29c16b8de0ed6f7a9e',
    '0x371d0b8a085f7bf7c7373b18e6f9935ba21b172183a55928ece6a6d9d7918bfe',
    '0x1779f9f8e72d1e89d80cdaa655000a2f0b52cef40db07e9ee504bd2a3195b8f4'
  ],
  baseFeePerGas: BigNumber { _hex: '0x07', _isBigNumber: true },
  _difficulty: BigNumber { _hex: '0x01', _isBigNumber: true }
}

[ghost]

This code is also being used by Web3.js https://github.com/ChainSafe/web3.js/blob/1.x/packages/web3-providers-http/types/index.d.ts#L32 as well as many typescript implementations using http.Agent.

[ghost]

I remember that @types/node checks that this should only be one of the Agent supported by http or otherwise it is a boolean.

[NBMSacha]

Hello @ayanamidev,
Thanks a lot for this PR :)

I'm in need for exactly this, so I guess I won't wait for ethers maintainers to merge this - I would like to start using it right now. Not sure about how to add the HTTP agent (in my case, would be a proxy) to ethers - could you document (or link me to the documentation about) how to do so?

Thanks again!

[ghost]

@NBMSacha Actually, it is hard to document how to use the feature without merging this PR since ethers.js have complicated build system. When I have time, I will code the standalone ethers.js compatible HTTP provider that supports powerful featues such as automated client-side fault tolerant load balance features, HTTP / Socks5 proxy support, etc.

[NBMSacha]

Thanks for your reply!! So there is no way I can get started with the codebase from your branch and use an http agent before you find some time to code the standalone version? :/

[ghost]

@NBMSacha Can't tell this one would be lighter than the native ethers provider, however, if you would find some temporary workout until the changes are reflected for v6, here is something for you.

https://github.com/ayanamitech/ethers-axios-provider

[ludekvodicka]

What happened with ethers-axios-provider repository? Was there any issue/security issue with this library?

[skyf0cker]

Does ethers v6 support custom http proxy now?

[cryptothink629]

hello, still not implemented in V6? anybody know?

[ricmoo]

It’s available. The Request.registerGetUrl static method can be used to replace the fetching operations used. I’m planning on getting some example up later.

[cryptothink629]

thank you so much!
Could you show us a quick code snippet example here? Please

[ghost]

@cryptothink629 I have posted the example code which works with V6 #4336