Epic: Modularization: Updated Roadmap

[nickpoorman]

Updated Roadmap

• Move ServicesMain to hedera-app #4689
• Add feature flag to enable new workflows #4761
• Modularized HCS implementation #4945
• Update to use PBJ generated classes #4712
• Modularize hedera-util-service #5522
• Epic: Implement the hedera-admin-service modules #4220
• Modularize RecordCache #6555
• Implement query workflow with adapters #4713
• Implement pre-handle workflow #4715
• Implement handle workflow using adapter #4718
• MerkleHederaState - Use new merkle tree in hedera-app, adapt mono-repo #4691
• Fee Facility - Create APIs for fee calculations #4732
• Properties Facility - Create APIs for properties #4734
• Properties Facility - Implement properties APIs backed by mono-repo code #4735
• Revisit UncheckedSubmit #5001
• Disable all logging in CI #4914
• Implement handle() method in transaction handlers in each module to delegate to mono-service #4719
• Epic: Modularize concerns of signing requirements and signature verification #4034

---

Triage

• Implement the transaction handling workflow #4211

---

June 9th - Final Date before beginning E2E Testing

• Epic: Implement the hedera-app module #4198
• Epic: Implement ingestion workflow with adapters #4714
• Epic: Special Files Facility: Handle updates to special files #6856
• Epic: Throttle Facility - Implement throttle APIs backed by mono-repo code #4729
• Epic: Fee Facility - Implement fee APIs backed by mono-repo code #4733
• Epic: Implement the hedera-smart-contract-service modules #4217
• Epic: Implement the hedera-util-service modules #4221
• Epic: Implement the hedera-app-spi module #4224
• Epic: Modularize hedera-file-service #5517
• Epic: Modularize consensus-service #6019
• Epic: Modularize hedera-network-admin-service #5518
• Epic: Modularize hedera-schedule-service #5519
• Epic: Modularize hedera-smart-contract-service for handle dependencies #5520
• Epic: Modularize Token Service Implementation #5767
• Epic: Modularize hedera-staking-service #5523
• Epic: Add handle implementations for all Crypto operations (Other than CryptoTransfer) #5770
• Intermediate FeeCalculator in HandleContext #6702

Remaining Sprint Backlog Items Ready to Be Pulled

• Implement prehandle and handle in TokenUpdateHandler #6340
• Implement handle for TokenAccountWipeHandler #6389
• Intermediate FeeCalculator in HandleContext #6702
• File service bi-directional translators #6848
• Network Admin service bi-directional translators #6849
• Token service bi-directional translators #6850
• Crypto service bi-directional translators #6851
• Smart contracts service bi-directional translators #6852
• Consensus service bi-directional translators #6853
• Schedule service bi-directional translators #6854
• Bi-directional translators - Probably some miscellaneous others #6855

---

E2E Testing

• Design Facility for privileged transaction checks for modular services #6131
• Implement a base framework for blackbox testing
• Write/Audit blackbox tests for query workflow #4720
• Write/Audit blackbox tests for ingestion workflow #4721
• Write/Audit blackbox tests for pre-handle workflow #4722
• Write/Audit blackbox tests for handle workflow #4723
• Handle migration from mono service state to modularized state #6858
• Revisit the order of initialization around schemas and SchemaRegistry #5120

---

Iterate++

• Modularize expire and renewal #6421
• Implement a facility for Throttling #4785
• Create APIs for all facilities/utilities needed for handle workflow #4778

---

Final Testing

• Review Validations and Error / Exceptions #5012
• Add more RecordStreamValidator implementations #4529

---

Future

• Epic: Build improvements for modularity #4222
• Epic: Restructor structure of the repo #4243
• Benchmark exceptions in non-happy path and determine if we need to refactor them to return explicit responses. #4756
• Update long term schedule transactions feature #3994
• Change the pricing for Long term schedule transactions based on minutesUntilScheduledTime and sizeOfScheduledTransaction #4154

---

Old Items

At this point, we can take the workflows into production. We won’t have new code for the handlers, and we won’t be using any of the states or other APIs, but we will have the workflows done and in production. At this point we are using the new gRPC handlers and not the ones in mono-repo.

• State Facility - Implement stores for each module #4725
• State Facility - Implement other methods needed in ReadableStores #4755
• Write/Audit blackbox tests for pre-handle for each transaction #4726
• Implement new pre-handle for each transaction (file this issue and create a sub issue for each transaction type, and for each tx type it depends on the “write/audit blackbox test” of the corresponding transaction type. So the pre-handle crypto create issue would depend on the test crypto create pre-handle issue) #4727

At this point, all pre-handle work will be in the new code and not in the mono-repo. We can actually ship when ready for individual pre-handles, we don’t have to have them all done to ship any. We can ship them as they are available.

• Throttle Facility - Create APIs for throttles #4728
• Write/Audit blackbox tests for each query (one top level issue and one sub issue for each query type) #4730
• Implement new query for each query (one top level issue and one sub issue for each query type, with dependency on the corresponding test issue filed above) #4731

At this point, we will have all queries and all pre-handle running on the production network. In theory, these could be done in parallel with the pre-handle work, we don’t have to finish all pre-handle to get started on queries, but practically it might make sense to have everybody focus on one type of transaction at a time.

• Record Facility - Create API for records #4736
• Record Facility - Implement records APIs backed by mono-repo code #4737
• Write/Audit blackbox tests for handle for each transaction (one top level issue and one sub issue for each transaction type) #4738
• Implement new handle for each transaction (one top level issue and one sub issue for each transaction type which depends on the corresponding testing issue) #4739

At this point all transaction handling code, pre-handle, query, gRPC is all being handled by new business logic, but still on the current merkle tree data structure.

• For each service, design new merkle state layout and migrate and update stores #4740
• Switch to new States backend implementation #4741

At this point, we can use the Schema APIs for handling schema migrations. Up to this point, we have to use the mono-service migration APIs for handling migrations.

• Implement improved record handling implementation #4742
• Implement improved throttle system #4743
• Implement improved fee system #4744
• Handle sanitization of inputs at API layer and remove pre-check where we reject the transaction if the memo contains the unicode NUL (0x00) byte. #4753
• Remove adapters #5524

Future

• Remove the default constructor provided for ConstructableRegistry #4860
• Implement MerkleHederaState#getAddressBookCopy() to return the AddressBook #4861

Tests and CI

• Write a test to ensure the error returned when a query cannot be parsed does not change. #4865

Merge Platform and Services code together

• Rearrange Services directories to make way for Platform directories #4889

Metrics and Monitoring

• Gradle build for launching the Grafana dashboards #4890

---

Core Facilities:

• Properties
• States
• Fees
• Throttles
• Records
• Signature Verification

Final state