Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(data-parser): data-parser implementation #111

Merged
merged 35 commits into from
Aug 13, 2024
Merged

feat(data-parser): data-parser implementation #111

merged 35 commits into from
Aug 13, 2024

Conversation

0xterminator
Copy link
Contributor

@0xterminator 0xterminator commented Jul 30, 2024 •
edited
Loading

Ref: #110

Scope of the MR:

  • - added new crate data-parser
  • - added benchmark tests for data-parser
  • - moved all benches in the same crate
  • - integrated data-parser into fuel-streams publihser
  • - reworked fuel-streams-publisher to be compatible with core
  • - extended benchmark tests with new data parser and S2 compression from nats
  • - integrated data-parser into benchmarks
  • - updated fuel-core to v.0.32 as specified in: Update to use v0.32 of fuel-core on Publisher #96 (comment)

@0xterminator 0xterminator force-pushed the ep/feat/data-parser branch 2 times, most recently from fc59439 to 602d147 Compare August 5, 2024 11:57
@0xterminator 0xterminator marked this pull request as ready for review August 5, 2024 11:58
@0xterminator 0xterminator changed the title feat(parser): initial commit for data-parser feat(data-parser): initial commit for data-parser Aug 5, 2024
@0xterminator 0xterminator changed the title feat(data-parser): initial commit for data-parser feat(data-parser): data-parser implementation Aug 5, 2024
0xterminator and others added 19 commits August 7, 2024 17:55
@0xterminator
feat(parser): initial commit for data-parser
2934f49
@0xterminator
feat(data-parser): cleanup
13d2085
@0xterminator
feat(repo): added prost parser
6abd2d8
@0xterminator
feat(repo): added criterion
8f5fe78
@0xterminator
feat(data-parser): extended benchmarks
f3df353
@0xterminator
feat(data-parser): added blocks and txs types
db6fd6b
@0xterminator
feat(data-parser): fixed Cargo.toml and lock
c2e4922
@0xterminator
feat(data-parser): fixed criterion tests
4d2a826
@0xterminator
feat(repo): integrated data parser in publisher
743fede
@pedronauck @0xterminator
test(repo): add test project for publishing
03de678
@pedronauck @0xterminator
test(repo): improve benchmarks
564b027
@pedronauck @0xterminator
test(repo): improve benchmarks algorithms
48deab9
@pedronauck @0xterminator
fix: pre-commit warnings
e81ab0d
@pedronauck @0xterminator
test: add benchmarks
9ebd9bf
@pedronauck @0xterminator
fix: clippy warnigs
41c739d
@pedronauck @0xterminator
docs: add readme on how to run benchs
de3e8b8
@pedronauck @0xterminator
docs: update README.md
fc24c95
@0xterminator
feat(repo): integrated fuel-data-parser further in scope added benches
b4fef4a
@0xterminator
feat(repo): exposed selected from fuel-data-parser
bd14994
pedronauck
pedronauck requested changes Aug 8, 2024
View reviewed changes
crates/data-parser/Cargo.toml Outdated Show resolved Hide resolved
crates/data-parser/Cargo.toml Outdated Show resolved Hide resolved
crates/fuel-data-parser/Cargo.toml Outdated Show resolved Hide resolved
crates/fuel-data-parser/src/error.rs Show resolved Hide resolved
crates/fuel-streams-core/src/types.rs Outdated Show resolved Hide resolved
crates/fuel-streams-core/src/types.rs Outdated Show resolved Hide resolved
Jurshsmith
Jurshsmith previously approved these changes Aug 8, 2024
View reviewed changes
Copy link
Member

@Jurshsmith Jurshsmith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work here 🚀. Looks good to merge already. However, we can increase our data sample space to improve the confidence level of the bench metrics.

benchs/bench-consumers/Cargo.toml Outdated Show resolved Hide resolved
benchs/bench-consumers/src/runners/benchmark_results.rs Outdated Show resolved Hide resolved
benchs/bench-consumers/src/runners/benchmark_results.rs Outdated Show resolved Hide resolved
benchs/bench-consumers/src/runners/runner_consumer.rs Show resolved Hide resolved
benchs/data-parser/Cargo.toml Outdated Show resolved Hide resolved
benchs/data-parser/src/lib.rs Outdated Show resolved Hide resolved
benchs/nats-publisher/config/nats.conf Show resolved Hide resolved
pedronauck
pedronauck approved these changes Aug 13, 2024
View reviewed changes
crates/fuel-streams-publisher/src/lib.rs Outdated Show resolved Hide resolved
@pedronauck pedronauck merged commit 8bc4de6 into main Aug 13, 2024
7 checks passed
@pedronauck pedronauck deleted the ep/feat/data-parser branch August 13, 2024 17:03
This was referenced Aug 14, 2024
Closed
Closed
Closed
Closed
pedronauck pushed a commit that referenced this pull request Aug 16, 2024
@0xterminator @pedronauck
feat(data-parser): Data-parser implementation (#111)
7475cb2 
* feat(parser): initial commit for data-parser

* feat(data-parser): cleanup

* feat(repo): added prost parser

* feat(repo): added criterion

* feat(data-parser): extended benchmarks

* feat(data-parser): added blocks and txs types

* feat(data-parser): fixed Cargo.toml and lock

* feat(data-parser): fixed criterion tests

* feat(repo): integrated data parser in publisher

* test(repo): add test project for publishing

* test(repo): improve benchmarks

* test(repo): improve benchmarks algorithms

* fix: pre-commit warnings

* test: add benchmarks

* fix: clippy warnigs

* docs: add readme on how to run benchs

* docs: update README.md

* feat(repo): integrated fuel-data-parser further in scope added benches

* feat(repo): exposed selected from fuel-data-parser

* feat(core): adjustments for new imports

* feat(repo): improved parser unit tests and annotations

* feat(core): added tx status in publisher

* feat(data-parser): fixed readme

* feat(repo): integrated data-parser into benchmarks

* refactor(benches): some improvments to the bench runners

* fix(benches): renamed Cargo.toml description

* fix(data-parser): clean up of Cargo.toml

* refactor(data-parser): cleanup unused deps

* feat(data-parser): added publish to Cargo.toml file

* feat(repo): dependencies cleanup

* feat(data-parser): added entropy to data

* fix(publisher): added admin profile to publisher

* refactor(repo): bumped crates versions, added script

* fix(publisher): use conn stores instead of arbitrary stores
pedronauck pushed a commit that referenced this pull request Aug 16, 2024
@0xterminator @pedronauck
feat(data-parser): Data-parser implementation (#111)
a6f25d5 
* feat(parser): initial commit for data-parser

* feat(data-parser): cleanup

* feat(repo): added prost parser

* feat(repo): added criterion

* feat(data-parser): extended benchmarks

* feat(data-parser): added blocks and txs types

* feat(data-parser): fixed Cargo.toml and lock

* feat(data-parser): fixed criterion tests

* feat(repo): integrated data parser in publisher

* test(repo): add test project for publishing

* test(repo): improve benchmarks

* test(repo): improve benchmarks algorithms

* fix: pre-commit warnings

* test: add benchmarks

* fix: clippy warnigs

* docs: add readme on how to run benchs

* docs: update README.md

* feat(repo): integrated fuel-data-parser further in scope added benches

* feat(repo): exposed selected from fuel-data-parser

* feat(core): adjustments for new imports

* feat(repo): improved parser unit tests and annotations

* feat(core): added tx status in publisher

* feat(data-parser): fixed readme

* feat(repo): integrated data-parser into benchmarks

* refactor(benches): some improvments to the bench runners

* fix(benches): renamed Cargo.toml description

* fix(data-parser): clean up of Cargo.toml

* refactor(data-parser): cleanup unused deps

* feat(data-parser): added publish to Cargo.toml file

* feat(repo): dependencies cleanup

* feat(data-parser): added entropy to data

* fix(publisher): added admin profile to publisher

* refactor(repo): bumped crates versions, added script

* fix(publisher): use conn stores instead of arbitrary stores
pedronauck pushed a commit that referenced this pull request Aug 16, 2024
@0xterminator @pedronauck
feat(data-parser): Data-parser implementation (#111)
9310be4 
* feat(parser): initial commit for data-parser

* feat(data-parser): cleanup

* feat(repo): added prost parser

* feat(repo): added criterion

* feat(data-parser): extended benchmarks

* feat(data-parser): added blocks and txs types

* feat(data-parser): fixed Cargo.toml and lock

* feat(data-parser): fixed criterion tests

* feat(repo): integrated data parser in publisher

* test(repo): add test project for publishing

* test(repo): improve benchmarks

* test(repo): improve benchmarks algorithms

* fix: pre-commit warnings

* test: add benchmarks

* fix: clippy warnigs

* docs: add readme on how to run benchs

* docs: update README.md

* feat(repo): integrated fuel-data-parser further in scope added benches

* feat(repo): exposed selected from fuel-data-parser

* feat(core): adjustments for new imports

* feat(repo): improved parser unit tests and annotations

* feat(core): added tx status in publisher

* feat(data-parser): fixed readme

* feat(repo): integrated data-parser into benchmarks

* refactor(benches): some improvments to the bench runners

* fix(benches): renamed Cargo.toml description

* fix(data-parser): clean up of Cargo.toml

* refactor(data-parser): cleanup unused deps

* feat(data-parser): added publish to Cargo.toml file

* feat(repo): dependencies cleanup

* feat(data-parser): added entropy to data

* fix(publisher): added admin profile to publisher

* refactor(repo): bumped crates versions, added script

* fix(publisher): use conn stores instead of arbitrary stores
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pedronauck pedronauck pedronauck approved these changes

@Jurshsmith Jurshsmith Jurshsmith left review comments

@luizstacio luizstacio Awaiting requested review from luizstacio luizstacio is a code owner

@lostman lostman Awaiting requested review from lostman

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@0xterminator @pedronauck @Jurshsmith