Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use AtomicView in all services #1612

Merged
merged 68 commits into from Jan 27, 2024
Merged

Use AtomicView in all services #1612

merged 68 commits into from Jan 27, 2024

Conversation

xgreenx
Copy link
Collaborator

@xgreenx xgreenx commented Jan 21, 2024

Related to #1589 and preparation for #1583.

All services that use the database for read-only purposes use AtomicView now instead of direct access to the database.

  • Removed Relayer from the Verifier because if it is not used for now, plus it may not be needed because of the shared sequencer and its consensus rules.
  • Added verification of the transactions root hash into Verifier.
  • Removed requesting of the one block from p2p, it is possible to use range request for that purposes.
  • Removed not used get_sealed_header and get_sealed_block method.
  • Added the latest_height method to AtomicView because the database always knows its latest height.
  • Added customisation of the Height used by the AtomicView. In the case of the relayer, we want to use DaBlockHeight as a primary key to create a snapshot, while in the case of the Fuel's databases, we want to use BockHeight as a primary key.
  • Renamed Executor into ExecutionInstance and changed it to be a one-time usable instance. It consumes the self to perform the execution of the block. The instance has a view of the database and execution options.

xgreenx and others added 30 commits December 24, 2023 04:08
…ion-to-own-crate' into feature/move-storage-implementation-to-own-crate
# Conflicts:
#	CHANGELOG.md
#	crates/fuel-core/src/executor.rs
#	crates/services/executor/src/executor.rs
#	crates/services/executor/src/ports.rs
#	crates/services/importer/src/config.rs
#	crates/services/importer/src/importer.rs
…-crate

# Conflicts:
#	CHANGELOG.md
#	crates/fuel-core/src/database.rs
#	crates/fuel-core/src/database/storage.rs
#	crates/storage/src/tables.rs
…key' into feature/block-height-as-primary-key

# Conflicts:
#	CHANGELOG.md
@xgreenx xgreenx requested a review from a team January 21, 2024 11:43
@xgreenx xgreenx self-assigned this Jan 21, 2024
# Conflicts:
#	crates/services/consensus_module/poa/src/verifier/tests.rs
#	crates/types/src/blockchain/header.rs
# Conflicts:
#	crates/fuel-core/src/executor.rs
}

fn view_at(&self, _: &BlockHeight) -> StorageResult<Self::View> {
// TODO: Unimplemented until of the https://github.com/FuelLabs/fuel-core/issues/451
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this is closed now.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without state rewind we can't implement it properly=) It uses the latest for now to simulate the current behavior=)

#[cfg(feature = "relayer")]
{
use fuel_core_relayer::ports::RelayerDb;
// TODO: The database should track the latest da height inside of the database object
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a follow-up issue fore this?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, #1589

@xgreenx xgreenx requested a review from a team January 26, 2024 19:31
@@ -85,6 +133,16 @@ pub struct BlockImporterAdapter {
Arc<fuel_core_importer::Importer<Database, ExecutorAdapter, VerifierAdapter>>,
}

impl BlockImporterAdapter {
pub fn events(&self) -> BoxStream<SharedImportResult> {
use futures::StreamExt;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have a reason why you'd put the import here rather than at the top. I see this sometimes and I'm curious what the justification for it is. I don't have anything against it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is used only in one place in that file. If you decide to remove that code later, you don't need to clean up imports too=) Plus it creates fewer conflicts with other people.

/// In production mode, block fields like transaction commitments are set based on the executed txs.
/// In validation mode, the processed block commitments are compared with the proposed block.
/// The executor is used for block production and validation.
/// It acts like a factory creating an `ExecutionInstance` valid only for one block.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is an implementation detail. I don't know if it caller needs to know this.

/// In production mode, block fields like transaction commitments are set based on the executed txs.
/// In validation mode, the processed block commitments are compared with the proposed block.
#[derive(Clone, Debug)]
pub struct ExecutionInstance<R, D> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be public? I only see uses in this file, but "ctrl-f" says there are 8 more usages in the code that it's not showing me lol. So maybe they are somewhere on this page and I just don't see them.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, it can be private=)

@MitchTurner
Copy link
Member

MitchTurner commented Jan 26, 2024

Added customisation of the Height used by the AtomicView. In the case of the relayer, we want to use DaBlockHeight as a primary key to create a snapshot, while in the case of the Fuel's databases, we want to use BockHeight as a primary key.

I don't think I want to block the PR for this, but this is an example of what I was talking about with business domain. the relayer and Fuel's db might be isomorphic, but they serve different roles in the domain of our code. I feel like the abstraction should be for the function, rather than the shape.

@xgreenx xgreenx enabled auto-merge (squash) January 26, 2024 23:39
@xgreenx
Copy link
Collaborator Author

xgreenx commented Jan 26, 2024

the relayer and Fuel's db might be isomorphic, but they serve different roles in the domain of our code.

It is true. The AtomicView provides customization for the Height not only because of how DB works. Mostly, it is dictated by the Executor and BlockHeader.

Because the BlockHeader has DaBlockHeight and BlockHeight,it creates two data sources for the Executor. The first one is on-chain storage, and the other one is relayer storage. It is why Executor wants to get the view of the relayer at DaBlockHeight and the on-chain storage at BlockHeight.

Relayer doesn't need to be a database to implement the AtomicView trait. You can see that example in the tests. The main ideas is that with this trait relayer should provide an immutable view of the relayer at DaBlockHeight.

@xgreenx xgreenx requested review from MitchTurner and a team January 26, 2024 23:59
Copy link
Member

@MitchTurner MitchTurner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay. This looks good to me, although I do have a small reservation about reusing the abstractions.

@xgreenx xgreenx merged commit a05ce48 into master Jan 27, 2024
32 checks passed
@xgreenx xgreenx deleted the feature/atomic-view-everywhere branch January 27, 2024 03:16
@xgreenx xgreenx mentioned this pull request Mar 4, 2024
xgreenx added a commit that referenced this pull request Mar 4, 2024
## Version v0.23.0

### Added

- [#1713](#1713): Added
automatic `impl` of traits `StorageWrite` and `StorageRead` for
`StructuredStorage`. Tables that use a `Blueprint` can be read and
written using these interfaces provided by structured storage types.
- [#1671](#1671): Added a new
`Merklized` blueprint that maintains the binary Merkle tree over the
storage data. It supports only the insertion of the objects without
removing them.
- [#1657](#1657): Moved
`ContractsInfo` table from `fuel-vm` to on-chain tables, and created
version-able `ContractsInfoType` to act as the table's data type.

### Changed

- [#1723](#1723): Notify about
imported blocks from the off-chain worker.
- [#1717](#1717): The fix for
the [#1657](#1657) to include
the contract into `ContractsInfo` table.
- [#1657](#1657): Upgrade to
`fuel-vm` 0.46.0.
- [#1671](#1671): The logic
related to the `FuelBlockIdsToHeights` is moved to the off-chain worker.
- [#1663](#1663): Reduce the
punishment criteria for mempool gossipping.
- [#1658](#1658): Removed
`Receipts` table. Instead, receipts are part of the
`TransactionStatuses` table.
- [#1640](#1640): Upgrade to
fuel-vm 0.45.0.
- [#1635](#1635): Move
updating of the owned messages and coins to off-chain worker.
- [#1650](#1650): Add api
endpoint for getting estimates for future gas prices
- [#1649](#1649): Add api
endpoint for getting latest gas price
- [#1600](#1640): Upgrade to
fuel-vm 0.45.0
- [#1633](#1633): Notify
services about importing of the genesis block.
- [#1625](#1625): Making
relayer independent from the executor and preparation for the force
transaction inclusion.
- [#1613](#1613): Add api
endpoint to retrieve a message by its nonce.
- [#1612](#1612): Use
`AtomicView` in all services for consistent results.
- [#1597](#1597): Unify
namespacing for `libp2p` modules
- [#1591](#1591): Simplify
libp2p dependencies and not depend on all sub modules directly.
- [#1590](#1590): Use
`AtomicView` in the `TxPool` to read the state of the database during
insertion of the transactions.
- [#1587](#1587): Use
`BlockHeight` as a primary key for the `FuelsBlock` table.
- [#1585](#1585): Let
`NetworkBehaviour` macro generate `FuelBehaviorEvent` in p2p
- [#1579](#1579): The change
extracts the off-chain-related logic from the executor and moves it to
the GraphQL off-chain worker. It creates two new concepts - Off-chain
and On-chain databases where the GraphQL worker has exclusive ownership
of the database and may modify it without intersecting with the On-chain
database.
- [#1577](#1577): Moved
insertion of sealed blocks into the `BlockImporter` instead of the
executor.
- [#1574](#1574): Penalizes
peers for sending invalid responses or for not replying at all.
- [#1601](#1601): Fix
formatting in docs and check that `cargo doc` passes in the CI.
- [#1636](#1636): Add more
docs to GraphQL DAP API.

#### Breaking

- [#1725](#1725): All API
endpoints now are prefixed with `/v1` version. New usage looks like:
`/v1/playground`, `/v1/graphql`, `/v1/graphql-sub`, `/v1/metrics`,
`/v1/health`.
- [#1722](#1722): Bugfix: Zero
`predicate_gas_used` field during validation of the produced block.
- [#1714](#1714): The change
bumps the `fuel-vm` to `0.47.1`. It breaks several breaking changes into
the protocol:
- All malleable fields are zero during the execution and unavailable
through the GTF getters. Accessing them via the memory directly is still
possible, but they are zero.
- The `Transaction` doesn't define the gas price anymore. The gas price
is defined by the block producer and recorded in the `Mint` transaction
at the end of the block. A price of future blocks can be fetched through
a [new API nedopoint](#1641)
and the price of the last block can be fetch or via the block or another
[API endpoint](#1647).
- The `GasPrice` policy is replaced with the `Tip` policy. The user may
specify in the native tokens how much he wants to pay the block producer
to include his transaction in the block. It is the prioritization
mechanism to incentivize the block producer to include users
transactions earlier.
- The `MaxFee` policy is mandatory to set. Without it, the transaction
pool will reject the transaction. Since the block producer defines the
gas price, the only way to control how much user agreed to pay can be
done only through this policy.
- The `maturity` field is removed from the `Input::Coin`. The same
affect can be achieve with the `Maturity` policy on the transaction and
predicate. This changes breaks how input coin is created and removes the
passing of this argument.
- The metadata of the `Checked<Tx>` doesn't contain `max_fee` and
`min_fee` anymore. Only `max_gas` and `min_gas`. The `max_fee` is
controlled by the user via the `MaxFee` policy.
- Added automatic `impl` of traits `StorageWrite` and `StorageRead` for
`StructuredStorage`. Tables that use a `Blueprint` can be read and
written using these interfaces provided by structured storage types.

- [#1712](#1712): Make
`ContractUtxoInfo` type a version-able enum for use in the
`ContractsLatestUtxo`table.
- [#1657](#1657): Changed
`CROO` gas price type from `Word` to `DependentGasPrice`. The dependent
gas price values are dummy values while awaiting updated benchmarks.
- [#1671](#1671): The GraphQL
API uses block height instead of the block id where it is possible. The
transaction status contains `block_height` instead of the `block_id`.
- [#1675](#1675): Simplify GQL
schema by disabling contract resolvers in most cases, and just return a
ContractId scalar instead.
- [#1658](#1658): Receipts are
part of the transaction status.
Removed `reason` from the `TransactionExecutionResult::Failed`. It can
be calculated based on the program state and receipts.
Also, it is not possible to fetch `receipts` from the `Transaction`
directly anymore. Instead, you need to fetch `status` and its receipts.
- [#1646](#1646): Remove
redundant receipts from queries.
- [#1639](#1639): Make Merkle
metadata, i.e. `SparseMerkleMetadata` and `DenseMerkleMetadata` type
version-able enums
- [#1632](#1632): Make
`Message` type a version-able enum
- [#1631](#1631): Modify api
endpoint to dry run multiple transactions.
- [#1629](#1629): Use a
separate database for each data domain. Each database has its own folder
where data is stored.
- [#1628](#1628): Make
`CompressedCoin` type a version-able enum
- [#1616](#1616): Make
`BlockHeader` type a version-able enum
- [#1614](#1614): Use the
default consensus key regardless of trigger mode. The change is breaking
because it removes the `--dev-keys` argument. If the `debug` flag is
set, the default consensus key will be used, regardless of the trigger
mode.
- [#1596](#1596): Make
`Consensus` type a version-able enum
- [#1593](#1593): Make `Block`
type a version-able enum
- [#1576](#1576): The change
moves the implementation of the storage traits for required tables from
`fuel-core` to `fuel-core-storage` crate. The change also adds a more
flexible configuration of the encoding/decoding per the table and allows
the implementation of specific behaviors for the table in a much easier
way. It unifies the encoding between database, SMTs, and iteration,
preventing mismatching bytes representation on the Rust type system
level. Plus, it increases the re-usage of the code by applying the same
blueprint to other tables.
    
It is a breaking PR because it changes database encoding/decoding for
some tables.
    
    ### StructuredStorage
    
The change adds a new type `StructuredStorage`. It is a wrapper around
the key-value storage that implements the storage
traits(`StorageInspect`, `StorageMutate`, `StorageRead`, etc) for the
tables with blueprint. This blueprint works in tandem with the
`TableWithBlueprint` trait. The table may implement `TableWithBlueprint`
specifying the blueprint, as an example:
    
    ```rust
    impl TableWithBlueprint for ContractsRawCode {
        type Blueprint = Plain<Raw, Raw>;
    
        fn column() -> Column {
            Column::ContractsRawCode
        }
    }
    ```
    
It is a definition of the blueprint for the `ContractsRawCode` table. It
has a plain blueprint meaning it simply encodes/decodes bytes and
stores/loads them into/from the storage. As a key codec and value codec,
it uses a `Raw` encoding/decoding that simplifies writing bytes and
loads them back into the memory without applying any serialization or
deserialization algorithm.
    
If the table implements `TableWithBlueprint` and the selected codec
satisfies all blueprint requirements, the corresponding storage traits
for that table are implemented on the `StructuredStorage` type.
    
    ### Codecs
    
Each blueprint allows customizing the key and value codecs. It allows
the use of different codecs for different tables, taking into account
the complexity and weight of the data and providing a way of more
optimal implementation.
    
That property may be very useful to perform migration in a more easier
way. Plus, it also can be a `no_std` migration potentially allowing its
fraud proving.
    
    An example of migration:
    
    ```rust
    /// Define the table for V1 value encoding/decoding.
    impl TableWithBlueprint for ContractsRawCodeV1 {
        type Blueprint = Plain<Raw, Raw>;
    
        fn column() -> Column {
            Column::ContractsRawCode
        }
    }
    
    /// Define the table for V2 value encoding/decoding.
    /// It uses `Postcard` codec for the value instead of `Raw` codec.
    ///
    /// # Dev-note: The columns is the same.
    impl TableWithBlueprint for ContractsRawCodeV2 {
        type Blueprint = Plain<Raw, Postcard>;
    
        fn column() -> Column {
            Column::ContractsRawCode
        }
    }
    
    fn migration(storage: &mut Database) {
        let mut iter = storage.iter_all::<ContractsRawCodeV1>(None);
        while let Ok((key, value)) = iter.next() {
            // Insert into the same table but with another codec.
            storage.storage::<ContractsRawCodeV2>().insert(key, value);
        }
    }
    ```
    
    ### Structures
    
The blueprint of the table defines its behavior. As an example, a
`Plain` blueprint simply encodes/decodes bytes and stores/loads them
into/from the storage. The `SMT` blueprint builds a sparse merkle tree
on top of the key-value pairs.
    
Implementing a blueprint one time, we can apply it to any table
satisfying the requirements of this blueprint. It increases the re-usage
of the code and minimizes duplication.
    
It can be useful if we decide to create global roots for all required
tables that are used in fraud proving.
    
    ```rust
    impl TableWithBlueprint for SpentMessages {
        type Blueprint = Plain<Raw, Postcard>;
    
        fn column() -> Column {
            Column::SpentMessages
        }
    }
                     |
                     |
                    \|/
    
    impl TableWithBlueprint for SpentMessages {
        type Blueprint =
Sparse<Raw, Postcard, SpentMessagesMerkleMetadata,
SpentMessagesMerkleNodes>;
    
        fn column() -> Column {
            Column::SpentMessages
        }
    }
    ```
    
    ### Side changes
    
    #### `iter_all`
The `iter_all` functionality now accepts the table instead of `K` and
`V` generics. It is done to use the correct codec during
deserialization. Also, the table definition provides the column.
    
    #### Duplicated unit tests
    
The `fuel-core-storage` crate provides macros that generate unit tests.
Almost all tables had the same test like `get`, `insert`, `remove`,
`exist`. All duplicated tests were moved to macros. The unique one still
stays at the same place where it was before.
    
    #### `StorageBatchMutate`
    
Added a new `StorageBatchMutate` trait that we can move to
`fuel-storage` crate later. It allows batch operations on the storage.
It may be more performant in some cases.

- [#1573](#1573): Remove
nested p2p request/response encoding. Only breaks p2p networking
compatibility with older fuel-core versions, but is otherwise fully
internal.

## What's Changed
* Fix the example tx in README.md by @Dentosal in
#1543
* Weekly `cargo update` by @github-actions in
#1557
* Update LICENSE by @Voxelot in
#1558
* Remove duplicating logic in the `KeyValueStore` trait by @xgreenx in
#1559
* Move `KeyValueStore` to the `fuel-core-storage` crate by @xgreenx in
#1566
* Update `libp2p` from `0.50.0` to `0.53.1` by @MitchTurner in
#1379
* Weekly `cargo update` by @github-actions in
#1575
* Weekly `cargo update` by @github-actions in
#1578
* Moved insertion of the blocks into the `BlockImporter` instead of the
executor by @xgreenx in #1577
* Let `NetworkBehaviour` macro generate `FuelBehaviorEvent` in p2p by
@MitchTurner in #1585
* Remove RustSec Ignore for RUSTSEC-2022-0093 by @MitchTurner in
#1586
* Simplify p2p request/response message serialization by @Dentosal in
#1573
* Weekly `cargo update` by @github-actions in
#1588
* Depend on libp2p directly instead of submodules by @MitchTurner in
#1591
* Make `Block` versionable by @MitchTurner in
#1593
* Switch katyo/publish-crates@v2 to xgreenx/publish-crates@v1 for the CI
by @Dentosal in #1603
* Weekly `cargo update` by @github-actions in
#1599
* Fix cargo doc, check for cargo doc in the CI by @Dentosal in
#1601
* Move storage traits implementation to the `fuel-core-storage` crate by
@xgreenx in #1576
* Extract off chain logic from the executor by @xgreenx in
#1579
* Applying comments from the PR for the storage crate by @xgreenx in
#1610
* Use `BlockHeight` as a primary key for the `FuelsBlock` table by
@xgreenx in #1587
* Use `AtomicView` in the `TxPool` by @xgreenx in
#1590
* Unify namespacing in P2P codebase by @MitchTurner in
#1597
* Weekly `cargo update` by @github-actions in
#1611
* Make `Consensus` version-able by @MitchTurner in
#1596
* feat: add api endpoint to retrieve messages by nonce by @matt-user in
#1613
* Update Docker tags for Flux integration by @liviudm in
#1617
* Remove dev consensus key by @MujkicA in
#1614
* Make `BlockHeader` a versionable enum by @MitchTurner in
#1616
* feat: Versionable CompressedCoin by @bvrooman in
#1628
* Use `AtomicView` in all services by @xgreenx in
#1612
* Notify services about importing of the genesis block by @xgreenx in
#1633
* Weekly `cargo update` by @github-actions in
#1634
* feat: Versionable `Message` by @bvrooman in
#1632
* Making relayer independent from the executor by @xgreenx in
#1625
* Upgrade to the fuel-vm 0.44.0 by @Dentosal in
#1600
* Upgrade to fuel-vm 0.45.0 by @MitchTurner in
#1640
* feat: Versionable Merkle metadata by @bvrooman in
#1639
* Add docs for GraphQL DAP endpoints by @Dentosal in
#1636
* Use a separate database for each data domain by @xgreenx in
#1629
* feat: dry run multiple transactions by @matt-user in
#1631
* Weekly `cargo update` by @github-actions in
#1644
* Decrease peer reputation on request timeouts and decode errors by
@Dentosal in #1574
* refactor: remove redudant receipts queries by @matt-user in
#1646
* Latest gas price endpoint by @MitchTurner in
#1649
* Estimate gas price API Endpoint by @MitchTurner in
#1650
* Fix for the race condition with tx status and receipts by @xgreenx in
#1658
* Bump up rust to 1.75 by @MitchTurner in
#1656
* Weekly `cargo update` by @github-actions in
#1655
* Duplication of the `0.22.1` release by @xgreenx in
#1665
* Move updating of the owned messages and coins to off-chain worker by
@xgreenx in #1635
* Weekly `cargo update` by @github-actions in
#1682
* Remove nested contract resolvers in GQL schema by @Voxelot in
#1675
* Soften txpool p2p reputation requirements by @Voxelot in
#1663
* Remove failing deployment job by @Voxelot in
#1692
* chore: format cargo tomls and add ci check by @segfault-magnet in
#1691
* Extraction of the `FuelBlockSecondaryKeyBlockHeights` table to an
off-chain worker by @xgreenx in
#1671
* Weekly `cargo update` by @github-actions in
#1707
* Increase liveness probe timeout to 5 minutes by @rfuelsh in
#1709
* feat: Add `ContractInfo` table and versionable `ContractInfoType` to
on-chain storage by @bvrooman in
#1657
* feat: Versionable `ContractUtxoInfo` for the `ContractsLatestUtxo`
table by @bvrooman in #1712
* Weekly `cargo update` by @github-actions in
#1719
* Bump `fuel-vm` dep to 0.47.1 by @MitchTurner in
#1714
* Bugfix: Calling of the contract with enabled `utxo_validation` fails
by @bvrooman in #1717
* Bugfix: Zero `predicate_gas_used` field during validation of the
produced block by @xgreenx in
#1722
* Notify about imported blocks from the off-chain worker by @xgreenx in
#1723
* Added versioning suffix to the GraphQL endpoint by @xgreenx in
#1725

## New Contributors
* @liviudm made their first contribution in
#1617

**Full Changelog**:
v0.22.0...v0.23.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants