Skip to content

Latest commit

 

History

History
631 lines (455 loc) · 26.9 KB

middleware.rst

File metadata and controls

631 lines (455 loc) · 26.9 KB
Raw

Middleware

Web3 manages layers of middlewares by default. They sit between the public Web3 methods and the providers, which handle native communication with the Ethereum client. Each layer can modify the request and/or response. Some middlewares are enabled by default, and others are available for optional use.

Each middleware layer gets invoked before the request reaches the provider, and then processes the result after the provider returns, in reverse order. However, it is possible for a middleware to return early from a call without the request ever getting to the provider (or even reaching the middlewares that are in deeper layers).

When integrating middleware with your provider, please ensure you're choosing the right version. For AsyncWeb3 users, select the version prefixed with async, such as async_attrdict_middleware. On the other hand, Web3 users should opt for versions lacking the async prefix. If an async version isn't listed, it implies it hasn't been made available yet.

More information is available in the "Internals: internals__middlewares" section.

Default Middleware

Middlewares are added by default if you don't add any.

Sync middlewares include:

  • gas_price_strategy
  • name_to_address
  • attrdict
  • validation
  • abi
  • gas_estimate

Async middlewares include:

  • gas_price_strategy
  • name_to_address
  • attrdict
  • validation
  • gas_estimate

The defaults are found in default_middlewares and async_default_middlewares methods in web3/manager.py.

AttributeDict

This middleware recursively converts any dictionary type in the result of a call to an AttributeDict. This enables dot-syntax access, like eth.get_block('latest').number in addition to eth.get_block('latest')['number'].

Note

Accessing a property via attribute breaks type hinting. For this reason, this feature is available as a middleware, which may be removed if desired.

.eth Name Resolution

This middleware converts Ethereum Name Service (ENS) names into the address that the name points to. For example w3.eth.send_transaction <web3.eth.Eth.send_transaction> will accept .eth names in the 'from' and 'to' fields.

Note

This middleware only converts ENS names on chains where the proper ENS contracts are deployed to support this functionality. All other cases will result in a NameNotFound error.

Gas Price Strategy

Warning

Gas price strategy is only supported for legacy transactions. The London fork introduced maxFeePerGas and maxPriorityFeePerGas transaction parameters which should be used over gasPrice whenever possible.

This adds a gasPrice to transactions if applicable and when a gas price strategy has been set. See Gas_Price for information about how gas price is derived.

Buffered Gas Estimate

This adds a gas estimate to transactions if gas is not present in the transaction parameters. Sets gas to: min(w3.eth.estimate_gas + gas_buffer, gas_limit) where the gas_buffer default is 100,000

HTTPRequestRetry

This middleware is a default specifically for HTTPProvider that retries failed requests that return the following errors: ConnectionError, HTTPError, Timeout, TooManyRedirects. Additionally there is a whitelist that only allows certain methods to be retried in order to not resend transactions, excluded methods are: eth_sendTransaction, personal_signAndSendTransaction, personal_sendTransaction.

Validation

This middleware includes block and transaction validators which perform validations for transaction parameters.

Configuring Middleware

Middleware can be added, removed, replaced, and cleared at runtime. To make that easier, you can name the middleware for later reference. Alternatively, you can use a reference to the middleware itself.

Middleware Order

Think of the middleware as being layered in an onion, where you initiate a web3.py request at the outermost layer of the onion, and the Ethereum node (like geth) receives and responds to the request inside the innermost layer of the onion. Here is a (simplified) diagram:

New request from web3.py

            |
            |
            v

    `````Layer 2``````

``

| ```

v

``

. `Layer 1 .`

`` .

. | .`

. v .

. . ` .

.` `Layer 0` .`

`.` . .`

. | . . . `.` | . .

. . JSON-RPC call . . . . . | . ` .

. . v . . . . .` . . . . . . Ethereum node . . . . . . . . . . . | . . . . . . | . . . . . . Response . . . . . . | . . . . . |`. . . `.` v`. . . Layer 0 `. . `. `.` `. . `. | `.` `. .` | .` `. v ` `.` ` .` Layer 1 . ` ` | v ``Layer 2

v

Returned value in web3.py

The middlewares are maintained in Web3.middleware_onion. See below for the API.

When specifying middlewares in a list, or retrieving the list of middlewares, they will be returned in the order of outermost layer first and innermost layer last. In the above example, that means that w3.middleware_onion.middlewares would return the middlewares in the order of: [2, 1, 0].

See "Internals: internals__middlewares" for a deeper dive to how middlewares work.

Middleware Stack API

To add or remove items in different layers, use the following API:

Optional Middleware

Web3 ships with non-default middleware, for your custom use. In addition to the other ways of Modifying_Middleware, you can specify a list of middleware when initializing Web3, with:

Web3(middlewares=[my_middleware1, my_middleware2])

Warning

This will replace the default middlewares. To keep the default functionality, either use middleware_onion.add() from above, or add the default middlewares to your list of new middlewares.

Below is a list of available middlewares which are not enabled by default.

Stalecheck

This middleware checks how stale the blockchain is, and interrupts calls with a failure if the blockchain is too old.

  • allowable_delay is the length in seconds that the blockchain is allowed to be behind of time.time()

Because this middleware takes an argument, you must create the middleware with a method call.

two_day_stalecheck = make_stalecheck_middleware(60 * 60 * 24 * 2)
web3.middleware_onion.add(two_day_stalecheck)

If the latest block in the blockchain is older than 2 days in this example, then the middleware will raise a StaleBlockchain exception on every call except web3.eth.get_block().

Cache

Simple Cache Middleware

These simple cache constructor methods accept the following arguments:

param cache

Must be an instance of the web3.utils.caching.SimpleCache class. If a cache instance is not provided, a new instance will be created.

param rpc_whitelist

Must be an iterable, preferably a set, of the RPC methods that may be cached. A default list is used if none is provided.

param should_cache_fn

Must be a callable with the signature fn(method, params, response) which returns whether the response should be cached.

Constructs a middleware which will cache the return values for any RPC method in the rpc_whitelist.

Ready to use versions of this middleware can be found at web3.middleware.simple_cache_middleware and web3.middleware.async_simple_cache_middleware. These are the equivalent of using the constructor methods with the default arguments.

Time-based Cache Middleware

Proof of Authority

Note

It's important to inject the middleware at the 0th layer of the middleware onion: w3.middleware_onion.inject(geth_poa_middleware, layer=0)

The geth_poa_middleware is required to connect to geth --dev or the Goerli public network. It may also be needed for other EVM compatible blockchains like Polygon or BNB Chain (Binance Smart Chain).

If the middleware is not injected at the 0th layer of the middleware onion, you may get errors like the example below when interacting with your EVM node.

web3.exceptions.ExtraDataLengthError: The field extraData is 97 bytes, but should be
1.  It is quite likely that you are connected to a POA chain. Refer to
http://web3py.readthedocs.io/en/stable/middleware.html#proof-of-authority
for more details. The full extraData is: HexBytes('...')

The easiest way to connect to a default geth --dev instance which loads the middleware is:

>>> from web3.auto.gethdev import w3

# confirm that the connection succeeded
>>> w3.client_version
'Geth/v1.7.3-stable-4bb3c89d/linux-amd64/go1.9'

This example connects to a local geth --dev instance on Linux with a unique IPC location and loads the middleware:

>>> from web3 import Web3, IPCProvider

# connect to the IPC location started with 'geth --dev --datadir ~/mynode'
>>> w3 = Web3(IPCProvider('~/mynode/geth.ipc'))

>>> from web3.middleware import geth_poa_middleware

# inject the poa compatibility middleware to the innermost layer (0th layer)
>>> w3.middleware_onion.inject(geth_poa_middleware, layer=0)

# confirm that the connection succeeded
>>> w3.client_version
'Geth/v1.7.3-stable-4bb3c89d/linux-amd64/go1.9'

Why is geth_poa_middleware necessary?

There is no strong community consensus on a single Proof-of-Authority (PoA) standard yet. Some nodes have successful experiments running though. One is go-ethereum (geth), which uses a prototype PoA for its development mode and the Goerli test network.

Unfortunately, it does deviate from the yellow paper specification, which constrains the extraData field in each block to a maximum of 32-bytes. Geth's PoA uses more than 32 bytes, so this middleware modifies the block data a bit before returning it.

Locally Managed Log and Block Filters

This middleware provides an alternative to ethereum node managed filters. When used, Log and Block filter logic are handled locally while using the same web3 filter api. Filter results are retrieved using JSON-RPC endpoints that don't rely on server state.

>>> from web3 import Web3, EthereumTesterProvider >>> w3 = Web3(EthereumTesterProvider()) >>> from web3.middleware import local_filter_middleware >>> w3.middleware_onion.add(local_filter_middleware)

#  Normal block and log filter apis behave as before.
>>> block_filter = w3.eth.filter("latest")

>>> log_filter = myContract.events.myEvent.build_filter().deploy()

Signing

This middleware automatically captures transactions, signs them, and sends them as raw transactions. The from field on the transaction, or w3.eth.default_account must be set to the address of the private key for this middleware to have any effect.

  • private_key_or_account A single private key or a tuple, list or set of private keys.

    Keys can be in any of the following formats:

    • An eth_account.LocalAccount object
    • An eth_keys.PrivateKey object
    • A raw private key as a hex string or byte string
>>> from web3 import Web3, EthereumTesterProvider
>>> w3 = Web3(EthereumTesterProvider)
>>> from web3.middleware import construct_sign_and_send_raw_middleware
>>> from eth_account import Account
>>> acct = Account.create('KEYSMASH FJAFJKLDSKF7JKFDJ 1530')
>>> w3.middleware_onion.add(construct_sign_and_send_raw_middleware(acct))
>>> w3.eth.default_account = acct.address

Hosted nodes<local_vs_hosted> (like Infura or Alchemy) only support signed transactions. This often results in send_raw_transaction being used repeatedly. Instead, we can automate this process with construct_sign_and_send_raw_middleware(private_key_or_account).

>>> from web3 import Web3
>>> w3 = Web3(Web3.HTTPProvider('HTTP_ENDPOINT'))
>>> from web3.middleware import construct_sign_and_send_raw_middleware
>>> from eth_account import Account
>>> import os
>>> acct = w3.eth.account.from_key(os.environ.get('PRIVATE_KEY'))
>>> w3.middleware_onion.add(construct_sign_and_send_raw_middleware(acct))
>>> w3.eth.default_account = acct.address

>>> # use `eth_sendTransaction` to automatically sign and send the raw transaction
>>> w3.eth.send_transaction(tx_dict)
HexBytes('0x09511acf75918fd03de58141d2fd409af4fd6d3dce48eb3aa1656c8f3c2c5c21')

Similarly, with AsyncWeb3:

>>> from web3 import AsyncWeb3
>>> async_w3 = AsyncWeb3(AsyncHTTPProvider('HTTP_ENDPOINT'))
>>> from web3.middleware import async_construct_sign_and_send_raw_middleware
>>> from eth_account import Account
>>> import os
>>> acct = async_w3.eth.account.from_key(os.environ.get('PRIVATE_KEY'))
>>> async_w3.middleware_onion.add(await async_construct_sign_and_send_raw_middleware(acct))
>>> async_w3.eth.default_account = acct.address

>>> # use `eth_sendTransaction` to automatically sign and send the raw transaction
>>> await async_w3.eth.send_transaction(tx_dict)
HexBytes('0x09511acf75918fd03de58141d2fd409af4fd6d3dce48eb3aa1656c8f3c2c5c21')

Now you can send a transaction from acct.address without having to build and sign each raw transaction.

When making use of this signing middleware, when sending dynamic fee transactions (recommended over legacy transactions), the transaction type of 2 (or '0x2') is necessary. This is because transaction signing is validated based on the transaction type parameter. This value defaults to '0x2' when maxFeePerGas and / or maxPriorityFeePerGas are present as parameters in the transaction as these params imply a dynamic fee transaction. Since these values effectively replace the legacy gasPrice value, do not set a gasPrice for dynamic fee transactions. Doing so will lead to validation issues.

# dynamic fee transaction, introduced by EIP-1559:
>>> dynamic_fee_transaction = {
...     'type': '0x2',  # optional - defaults to '0x2' when dynamic fee transaction params are present
...     'from': acct.address,  # optional if w3.eth.default_account was set with acct.address
...     'to': receiving_account_address,
...     'value': 22,
...     'maxFeePerGas': 2000000000,  # required for dynamic fee transactions
...     'maxPriorityFeePerGas': 1000000000,  # required for dynamic fee transactions
... }
>>> w3.eth.send_transaction(dynamic_fee_transaction)

A legacy transaction still works in the same way as it did before EIP-1559 was introduced:

>>> legacy_transaction = {
...     'to': receiving_account_address,
...     'value': 22,
...     'gasPrice': 123456,  # optional - if not provided, gas_price_strategy (if exists) or eth_gasPrice is used
... }
>>> w3.eth.send_transaction(legacy_transaction)