Proposed Re-architecture of the JS-SDK!

[MParlikar]

Stay tuned for a proposed re-architecture of the JS-SDK. We are aware that in the beginning, the SDK was an outgrowth of the creation of the Signer. A re-work is desired to help make the SDK easier to use. @hoffmannjan will be presenting his proposal for this here.

[hoffmannjan]

Hey guys, after working two years at Casperlabs and maintaining this repo, I collected a lot of feedback about DX regarding this SDK.
Also, a few weeks ago, I started making comparisons / thinking about possible improvements in js-sdk architecture.

I reviewed Web3.js, @solana/web3.js and near-api-js.

At first sight, we're not so far from Solana's SDK, but some places are:

• overengineered
• code added on demand (which generates mess, and we should avoid this)
• some parts are so old and was not touched since the library was published (about 3y)
• it acts as a golden standard of the SDK, but today I doubt some of the design decisions

You can check my SDK comparsions but here is TLDR:

• we're pretty close to how Ethereum / Solana SDKs are structured
• I like the user-centric attitude that is in near-js-api

I'm a fan of composition & functional approach, so I prepared a roadmap for how we should develop the whole thing.
Also, we would like to share every milestone with the community and wait for their acceptance.

List of rules that we should keep in mind developing SDK:

• Keep the low-level methods so more advanced users can achieve whatever they want (e.g., users can build the deployment in multiple steps)
• Keep all RPC methods stateless (we can then build CasperClient etc, but it should be super easy to query something over RPC without any boilerplate code)
• On top of that, build some convenient abstractions.
• I would consider separating casper-types as a standalone package (like [@solana/buffer-layout] in Solana) - (some developers might gain from just using it and sending the serialized data to the backend without the whole RPC logic) (TBD)

Roadmap:

• Specify a list of all Keys-related methods (draft new Keys class)
• Implement all RAW RPC methods
  • raw functions
  • class containing all of them
• Top level API should look like
  • Account - user can interact with his Account and with other Accounts. He can provide only PublicKey or KeyPair - depending on this, he can call different methods (like .sign vs. .getBalance).
  • Keys
  • Contract - similar to what we have now, it should be able to consume ABI that will help interact with the contract.
  • Wallet - integration with all wallets that are on the market atm.
  • EventStream - we should have EventStream class but still would be happy to hide it for most users under something like PromiEvent
  • Utils

This is only high level overview, I'm pretty sure when we start the work we will be able to specify more details and share it with community.

[bpr]

I notice that the Solana JSON RPC suite has 53 methods. Casper has about 15 IIRC. That's a big enough difference to make me curious. Is their API bloated with unnecessary stuff, is ours too spare, are they really incomparable so nothing to see here?

[hoffmannjan]

That's true, maybe we should review their RPC endpoints and compare to ours. This is just high-level overview of how the new SDK will be built - so IMHO comparsion of RPC is different subject. But good point, maybe there is also an area for improvements!

[asladeofgreen]

• Please review polkadotjs ... it's very high quality.
• Also review structure of python Casper SDK ... might be useful.
• Would like to see a medium level design document plus a slide deck, going deeper into structural design.

[hoffmannjan]

Thanks for the great suggestions @asladeofgreen - I will start reviewing polkadotjs next week, I'm also familiar with your SDK but for sure I'll take a look again.
And yeah, after that I'll prepare a more detailed structural design document than can be next converted for tickets.

[hoffmannjan]

Casper-js-sdk v3.0.0

Smaller modules

Proposition is to split casper-js-sdk into smaller dependencies which will be easier to maintanance and release independently.
Currently when there is a bug in one part of the library we need to re-release the whole library instead of just one module.
Also using current version of NPM we can use new feature which is called peerDependencies which basically eliminates the problem with multiple version of dependencies required by different packages.

The only downside of this solution is that the end-user needs to be more explicite when installing SDK.
Instead of calling npm i casper-js-sdk one will need to do npm i @casperjs/rpc @casperjs/cltypes @casperjs/keys @casperjs/tools.
Here is a top-level description of the single dependencies:

@casperjs/api

• contains all rpc methods

• has a wrapper class as well as a single functions for every rpc method

  • example usage with wrapper

  import { RPCClient } from '@casperjs/rpc`;
  const rpcClient = new RPCClient(__rpcUrl__);
  const currentBlockHeight = rpcClient.getCurrentBlockHeight(configuration*); // "79811"
  

  • example usage with single functions

  import { getCurrentBlockHeight } from '@casperjs/rpc`;
  const currentBlockHeight = await getCurrentBlockHeight(__rpcUrl__, configuration); // "79811"
  

  • its possible to pass configuration object which supports following options

  responseFormat: 'TypedJSON'*, 'parsedJSON', 'raw'
  retryOnFailure: false* | { numOfRetries: 10, delayBetween: 1000 (in `ms`)
  

  • responseFormat:
    - TypedJSON* default option which parses and reconstructs the RPC reponse to typed data structure which is more convinient to use especially in TypeScript application
    - parsedJSON - returns just parsed JSON (calls JSON.parse on response from RPC)
    - raw - returns raw, unformatted response object

  • retryOnFailure:
    - false by default which means that if the call will fail it will just try an error and handling it stays on developer side
    - configuration object conatining:
    - numOfRetries - number of retries before the error will be thrown
    - delayBetween - number of ms between every retry

  • Reimplement HTTP EventStream using EventSource Already done in 2.15.4

  • Over the raw layer of EventParser there should be few more developer friendly tools like watching for some specific Deploys (currently DeployWatcher which needs improvements)

@casperjs/deploy

• contains functions which mutates the deploy in certains ways (eg sign, add ttl) as well as the holds properties of the Deploy object
• there are some rules applied only to specic deploy type eg Transfer.
• this module must include easy ways to create major deploy types which are:
  • transfer
  • entry point call
  • deploy containing session code
• runtime arguments should be easy to create and modify
• we should improve Typescript typings here and reconsider usage of TypedJSON library

@casperjs/clvalue

• CLValue implementation is pretty new so it will mainly require changes prior to new protocol 2.0
• Switch to more modern library (Big.js) operating on big numbers (not supported natively by JS runtime)

@casperjs/keys

• we should follow the current implementation but we should make it more consistent (use same base64 format everywhere etc)

@casperjs/utils

• Some boilerplate code for common use-cases
• Extended RPCClient which is able to utilize provided EventStream

Technological decisions

• Start from scratch to upgrade all of the core dependencies to newest, stable versions
• We decided to use Turbo stack because it has good opinions and we need something battle tested in terms of managing multiple npm packages in one repository.
• Implement the api module to make sure all of the core concepts saddle
• At the same time, setup testing env and start writing tests. Maybe part of the code can be written in TDD manner? It would be nice to share some tapes containing recorded JSON responses with different sdks
• There should be also some tests operating on mainnet and testnet. At least in terms of reading, as blockchain is immutable we can be sure how block 40 look.
• Move the clvalues module and make sure it works as expected.

Development best practises

• Treat Guidance for JSON-RPC SDK Compliance as a source of truth. If you think there is an mistake or something can be described in a clearer way - feel free to share the feedback with us [contact]
• Implement tests
• Remember what Uncle Bob says

[zie1ony]

This looks like all good decisions.
One think I can ask about is the Contract concept. Do you plan to provide a class that would make easy to interact with deployed contract?

[hoffmannjan]

Sure thing, it should be part of the deploy module and it will be extended version what we currently have. In perfect world it should be able to consume metadata provided in a JSON and help developer interact with the contract even in runtime (we can easily create a REPL to interact with the network in nodejs environment).