| number | title | author | discussions-to | status | type | category | created |
|---|---|---|---|---|---|---|---|
7571 |
Shadow Events |
Emily Hsia (@emilyhsia), Alvin Hsia (@alvinhsia), Jon Becker (@beckerrjon), Ethen Pociask (@ethen_not_ethan), Georgios Konstantopoulos (@gakonst), Storm Slivkoff (@notnotstorm), Mark Toda (@marktoda), Sara Reynolds (@saraareynolds), Austin Adams (@AustinAdams10) |
Draft |
Standards Track |
ERC |
2023-11-09 |
A standard for shadow events in Ethereum smart contracts, enabling enhanced onchain data indexing, analytics, and reduced gas costs through an offchain logging mechanism.
This draft is a starting point for discussion and refinement. It aims to lay the foundation for a standard that could be implemented across the Ethereum ecosystem, enhancing the capabilities of smart contracts while maintaining efficiency and security.
This standard proposes a system for shadow events, which are events that are generated offchain in a shadow fork of an Ethereum compatible execution client. Shadow forks provide a mechanism to emit richer, custom, and more detailed event data without incurring the gas costs associated with onchain event logging on mainnet.
Shadow events are emitted by shadow contracts in an offchain shadow fork environment, and would be accessible via standard JSON RPC calls. Shadow events, shadow contracts, and the types of shadow forks are described in more detail in the Specification section below.
This standard does not intend to prescribe what event data should be logged on mainnet as opposed to a shadow fork; that decision is ultimately left to the smart contract developer.
The motivation for shadow events is threefold:
- Richer Analytics: Shadow events enable the emission of detailed data that would be prohibitively expensive to emit onchain, thus providing developers and analysts with deeper insights into contract interactions.
- Open Data Access: Anyone can write a shadow contract implementation without affecting mainnet state, allowing the data accessibility of smart contracts to be improved upon permissionlessly.
- Gas Efficiency: By allowing protocols to emit shadow events offchain, smart contracts can reduce their onchain footprint, leading to lower gas costs for users and more efficient use of blockspace.
A shadow fork is an offchain execution environment that mirrors Ethereum mainnet state; either in realtime or monotonically over some historical block range. Shadow forks bypass gas and contract size restrictions when executing transactions, enabling the emission of detailed event data without incurring gas costs for users on mainnet, and allowing developers to more efficiently utilize the 24,576 byte contract size limit.
Because the changes made on a shadow fork are read-only and the post-transaction state does not change when shadow events are emitted, they can easily stay in sync with mainnet with no risk of halting; assuming the use of either an appropriately set confirmation depth or a mechanism for handling chain reorgs.
Shadow forks execute all transactions in some origin block while only persisting lightweight post-state entities (i.e. receipts, logs) and disregarding heavier data (i.e. resultant MPT state). This allows shadow forks to be cost effective while still maintaining necessary information for rich downstream indexing and ingestion. Shadow forks only bypass gas and contract size restrictions for transactions that succeed on mainnet.
A shadow contract is a modified version of a deployed mainnet contract, which is deployed as new bytecode onto a shadow fork using offchain infrastructure. Modifications are made on a shadow contract in order to get additional event data that is not logged on the mainnet contract, or event data that would be otherwise uneconomical to generate on mainnet. Shadow contracts have no effect on their mainnet counterparts, can be updated at any time, and anyone can write a custom implementation permissionlessly.
Shadow events are emitted by shadow contracts within a shadow fork of Ethereum mainnet. These logs are not included in logsBloom of mainnet nodes, relieving them of storage and computation requirements. Shadow events are emitted in response to transactions that are executed by EVM in the shadow fork.
Shadow code that should only be run in a shadow fork environment can be specified using shadow blocks or shadow annotated comments within the Solidity source code.
Shadow blocks are the preferred syntax to allow developer tools to recognize and process shadow events.
A shadow block is marked by shadow { ... }, where the code inside the curly braces is Solidity, or Yul for inline assembly. This pattern is heavily inspired by Solidity’s existing Inline Assembly syntax.
Example of a shadow block:
shadow {
emit LogEventName(param1, param2, param3);
}Similar to Solidity’s Inline Assembly syntax, the inline shadow code can access local Solidity variables. However, unlike inline assembly blocks, different inline shadow blocks share the same namespace, i.e. it is possible to access a variable defined in a different inline shadow block.
Tooling that builds support for this syntax can enable the shadow version of the contract for their own purposes. For example, a smart contract testing framework can add shadow contract support that would allow developers to write unit tests for their shadow contract implementation.
For backwards compatibility with existing compilers, tooling can introduce a preprocessing step to remove the shadow block parentheses before running the compiler to compile the shadow version of the contract.
int128 liquidityNet =
ticks.cross(
step.tickNext,
(zeroForOne ? state.feeGrowthGlobalX128 : feeGrowthGlobal0X128),
(zeroForOne ? feeGrowthGlobal1X128 : state.feeGrowthGlobalX128),
cache.secondsPerLiquidityCumulativeX128,
cache.tickCumulative,
cache.blockTimestamp
);
shadow {
// Capture the state of the tick after it is crossed
Tick.Info storage info = ticks[step.tickNext];
// Math to convert X128 to E18 for downstream data storage
uint256 feeGrowthOutside0E18 = (info.feeGrowthOutside0X128 * 10**18) >> 128;
uint256 feeGrowthOutside1E18 = (info.feeGrowthOutside1X128 * 10**18) >> 128;
// Emit TickCrossed event
emit TickCrossed(
step.tickNext,
info.liquidityGross,
info.liquidityNet,
feeGrowthOutside0E18,
feeGrowthOutside1E18,
info.tickCumulativeOutside,
info.secondsPerLiquidityOutsideX128,
info.secondsOutside
);
}
if (zeroForOne) liquidityNet = -liquidityNet;Example: Shadow block for emitting a new TickCrossed event in a Uniswap V3 pool contract This syntax can be used with supported tooling.
To ensure backwards compatibility, we also define a syntax using annotated comments that guarantees compatibility with all existing tools.
Example of an shadow annotated comment:
/* shadow {
emit LogEventName(param1, param2, param3)
} */Tooling that supports shadow contracts should have the ability to transform between shadow blocks and annotated comments. These tools would transform shadow blocks into annotated comments when interacting with other tools and services that do not offer first-class shadow contract support. For example, the Foundry toolkit could convert the shadow blocks into shadow annotated comments before uploading the contract source code to Etherscan or Sourcify for verification, enabling users of those services to easily see the source code that was used to compile both the mainnet and shadow versions of the deployed contract.
int128 liquidityNet =
ticks.cross(
step.tickNext,
(zeroForOne ? state.feeGrowthGlobalX128 : feeGrowthGlobal0X128),
(zeroForOne ? feeGrowthGlobal1X128 : state.feeGrowthGlobalX128),
cache.secondsPerLiquidityCumulativeX128,
cache.tickCumulative,
cache.blockTimestamp
);
/* shadow {
// Capture the state of the tick after it is crossed
Tick.Info storage info = ticks[step.tickNext];
// Math to convert X128 to E18 for downstream data storage
uint256 feeGrowthOutside0E18 = (info.feeGrowthOutside0X128 * 10**18) >> 128;
uint256 feeGrowthOutside1E18 = (info.feeGrowthOutside1X128 * 10**18) >> 128;
// Emit TickCrossed event
emit TickCrossed(
step.tickNext,
info.liquidityGross,
info.liquidityNet,
feeGrowthOutside0E18,
feeGrowthOutside1E18,
info.tickCumulativeOutside,
info.secondsPerLiquidityOutsideX128,
info.secondsOutside
);
} */
if (zeroForOne) liquidityNet = -liquidityNet;Example: Shadow annotated comment for emitting an new TickCrossed event in a Uniswap V3 pool contract. Compatible with all existing tooling.
The access pattern for shadow events will adhere to the standard JSON RPC eth_getLogs interface. We propose leveraging the standard JSON RPC eth_getLogs interface to minimize the integration overhead for consumers of this data.
Shadow fork environment should provide a JSON RPC interface for the shadow fork. Implementers of shadow fork environments will have the discretion over the exact implementation of the shadow fork environment, the storage format of shadow events, and the implementation of the JSON RPC service.
We propose introducing a new JSON RPC endpoint (e.g. shadow_getAddresses) that returns the list of addresses that are shadowed for the given RPC service. The shadowed bytecode for each address can be subsequently fetched via the standard JSON RPC eth_getCode interface.
Users have spent >330K ETH on gas fees to emit events. In periods of high activity, this has spiked to ~21K ETH spent on event gas fees in a single month. Because of this, developers increasingly choose to log minimal events to reduce gas costs for their end users. As a result, the majority of data augmentation being done today in complex offchain indexing pipelines.
The proposed standard for shadow events aims to balance the need for rich onchain data with the constraints of blockchain resource costs. By establishing a standard, we can ensure that shadow events are emitted and accessed in a consistent manner, facilitating broader adoption and integration with existing tools.
Shadow events are fully backwards compatible as they do not affect the execution or the state of the mainnet contracts. They are an offchain addition that can be adopted incrementally by the community.
While shadow events do not affect onchain state, care must be taken to ensure that the offchain infrastructure for emitting and accessing shadow events is secure and reliable. Additionally, developers must be thoughtful about the reliance on shadow events, as to not introduce severe centralized trust risks.
Shadow fork implementations should take care to verify executional correctness of transactions. Similar to node clients, shadow execution environments can be open sourced and reviewed by the community.
Different code paths can be reached when re-executing a mainnet transaction on a shadow fork EVM that removes the use of contract size and gas cost restrictions (e.g. a contract that reaches recursive call-depth 10 on the origin chain but reaches the MAX_STACK_LIMIT when ran in a shadow environment). Furthermore, opcodes specific to consensus metadata (e.g. DIFFICULTY) could express different values than that of the mainnet transaction’s execution.
Strict verifications should be instilled into shadow execution environments to ensure that re-executed transactions don’t traverse code paths that largely differ from their mainnet counterparts.
The term "shadow fork” has been used in the past to refer to update shadow forks. This ERC refers to a second type of shadow fork – an application shadow fork:
-
Application shadow forks - Environment for altering data availability (receipt logs) via an offchain execution environment that allows for the execution of arbitrary EVM bytecode. These are critical for reducing user data footprints while also minimizing transaction gas costs.
-
Upgrade shadow forks - Environment for testing backwards incompatible node upgrades via a mirrored read-only node environment. These are critical for ensuring that protocol upgrades don’t introduce liveness failures or security vulnerabilities and can demonstrate correctness in a pseudo-production environment.
Copyright and related rights waived via CC0.