Optimizing IDE Integration for Chain Interactions: Balancing DX with Performance and Bundle Size

[josepot]

Introduction

Enhancing Developer Experience with Improved Chain Interaction Selection in IDEs

When initiating this project, our goal was to offer a flexible interface for developers to select key chain interactions for their dApps, either through an interactive CLI or a web-based UI. However, developer feedback suggests a need for a different approach.

Feedback and Challenges

Shifting to Integrated Development Environment (IDE) Based Solutions

Developers have expressed a preference for having all possible chain interactions accessible directly within the IDE, rather than through separate tools. This shift presents two primary challenges:

1. Performance Concerns: Integrating all chain interactions could slow down TypeScript server performance, impacting IntelliSense responsiveness.
2. Bundle Size: A comprehensive integration leads to larger bundle sizes, potentially degrading user experience (UX).

Despite these concerns, the demand for IDE-based chain interaction selection is strong.

Current Solutions and Limitations

Progress with Code Generation and Bundle Size Optimization

Our recent efforts in code generation have significantly improved TypeScript server performance, even with extensive chain interactions. However, despite having decreased the bundle size significantly, it still remains a concern. For example, using Kusama metadata for descriptor generation can increase the bundle size for a typical dApp by approximately 80Kb, which is less than ideal.

Proposed Solutions

Exploring Options for Efficient Code Management

Option 1: Manual Descriptor Import

• Approach: Developers manually import each descriptor, allowing unused code to be automatically trimmed. The code would look like this:

import { createClient } from "@polkadot-api/client"
import {
  StorageSessionValidators,
  StorageStakingValidators
} from "./codegen/ksm"
import { getProviderChain } from "./provider"

const client = createClient(getProviderChain.connect)

const sessionValidators = await client.query(StorageSessionValidators).getValue();
const validatorsData = await client.query(StorageStakingValidators).getValues(sessionValidators);

• Drawback: This contradicts the desired developer experience, as it limits the discovery of available chain interactions within the client.

Option 2: Advanced Tree Shaking Solutions

The code looks like this:

import { createClient } from "@polkadot-api/client"
import ksm from "./codegen/ksm"
import { getProviderChain } from "./provider"

const client = createClient(getProviderChain.connect, ksm)

const sessionValidators = await client.query.Session.Validators.getValue();
const validatorsData = await client.query.Staking.Validators.getValues(sessionValidators);

Option 2a: Bundler Specific Plugins

• Approach: Develop plugins for various bundlers (Vite, Rollup, Webpack, etc.) to analyze the AST and identify used interactions for custom tree-shaking.

Option 2b: Bundler Agnostic Solution

• Approach: Create a tool that inputs a bundled JS file and its source map, performs analysis similar to Option 2a, and outputs a new, optimized JS file with an updated source map.
• Challenge: This approach is complex and requires significant development effort.

Seeking Community Input

Collaboration on Plugin Development and Solution Refinement

I propose initially developing a plugin for Vite/Rollup as a starting point. I invite the community's thoughts on these solutions and encourage contributions towards developing plugins for other bundlers.

Conclusion and Next Steps

Balancing Developer Experience with Performance and UX

Our aim is to strike a balance between providing a seamless developer experience and maintaining optimal performance and UX. I look forward to the community's feedback and suggestions on these proposed solutions.

[ryanleecode]

For option 2A, my findings are its possible to do a naive AST search to determine which descriptors are used but there are many pitfalls and drawbacks that make it error prone and brittle. An alternative approach might be to figure out how "Find All References" is in implemented for typescript and run it over each descriptor.

More details here: https://gist.github.com/ryanleecode/82418096a6590fd45b44df294a21513f

[Polkadot-Forum]

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/polkadot-api-2023-q4-update/5318/1

[forgetso]

Having written a "plugin", of sorts, for removing excess stuff from PolkadotJS, I found that many things you would expect to be removed by tree shaking were still included when bundling (e.g. the bytes.js file containing the giant WASM blob was never tree shaken even when specifying onlyJS throughout our code). AST couldn't work out that we didn't want to use the WASM version. I've found a reliable way to tree shake is to have deep package exports in the package.json.

I'm considering our options for bundling JS for interacting with ink! contracts and so far scale-ts looks like a good way to create the required encoding / decoding functions. The key thing when creating a small bundle for a single contract will be including the minimal amount of encoders. Currently, we're using Typechain to output contract bindings but this outputs a huge amount of code, requiring the ABI to be bundled with the JS.

In future, we are aiming to develop a Typechain style code-generation tool that creates the required encoders for a contract, but, critically, doesn't output the entire encoding library (scale-ts / polkadot-api). This means the ABI won't be required and neither will the 80% of other type definitions required by substrate pallets.

Do you think polkadot-api could help us achieve this goal?

Anyway, I don't mean for this issue to go on a tangent. I thought the above context might be of some use to you when considering tree shaking in the context of Dapps.

[xlc]

I prefer option 1. Anything requires plugin to bundler is just going to be costly.

Drawback: This contradicts the desired developer experience, as it limits the discovery of available chain interactions within the client.

I don't really understand this. People should discover available interactions via docs, not auto completes.

Besides, we can do this

import { createClient } from "@polkadot-api/client"
import * as ksm from "./codegen/ksm"
import { getProviderChain } from "./provider"

const client = createClient(getProviderChain.connect)

const sessionValidators = await client.query(ksm.StorageSessionValidators).getValue();
const validatorsData = await client.query(ksm.StorageStakingValidators).getValues(sessionValidators);

so auto completes still works

(not 100% sure if this play nicely with tree shaking but to me it looks significantly easier for bundler to optimize compare to option 2)

[josepot]

I prefer option 1. Anything requires plugin to bundler is just going to be costly.

Drawback: This contradicts the desired developer experience, as it limits the discovery of available chain interactions within the client.

I don't really understand this. People should discover available interactions via docs, not auto completes.

Besides, we can do this

import { createClient } from "@polkadot-api/client"
import * as ksm from "./codegen/ksm"
import { getProviderChain } from "./provider"

const client = createClient(getProviderChain.connect)

const sessionValidators = await client.query(ksm.StorageSessionValidators).getValue();
const validatorsData = await client.query(ksm.StorageStakingValidators).getValues(sessionValidators);

so auto completes still works

(not 100% sure if this play nicely with tree shaking but to me it looks significantly easier for bundler to optimize compare to option 2)

We have actually made A LOT of iterations on this since the last time that we updated this issue 😅.

In fact @voliva is just finishing this latest iteration. Once it's done, then we will have the best of both worlds: super-small bundle-sizes, plus kick-ass code-competition. Also, the user won't be importing the types from a custom folder, but from a subpath of the library...

The thing is that the only thing that adds size the the bundle is the registry of checksums. However, this registry of checksums will be automatically imported dynamically by the library. Which means that all modern bundlers (parcel, vitest, webpack, etc) will create a separate asset/chunk just with the registry of the checksums, which will be required in the background... That's totally fine, because we only need the checksums after we have compiled the latest runtime, which is also an async process. So, by having a separate asset that just contains the checksums and that is loaded dynamically, we are speeding the overall loading time quite significantly.