Application: Subcoin

[liuchengxu]

Project Abstract

Subcoin is a Bitcoin full node in Substrate. The primary goal of this grant is to implement a prototype of a Bitcoin node that is capable of syncing the Bitcoin network from the genesis to the chain tip using the advanced state sync strategy offered by the Substrate framework.

Grant level

• Level 1: Up to $10,000, 2 approvals
• Level 2: Up to $30,000, 3 approvals
• Level 3: Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

• The application template has been copied and aptly renamed (project_name.md).
• I have read the application guidelines.
• Payment details have been provided (Polkadot AssetHub (DOT, USDC & USDT) address in the application and bank details via email, if applicable).
• I understand that 30% of each milestone will be paid in vested DOT, to the Polkadot address listed in the application.
• I am aware that, in order to receive a grant, I (and the entity I represent) have to successfully complete a KYC/KYB check.
• The software delivered for this grant will be released under an open-source license specified in the application.
• The initial PR contains only one commit (squash and force-push if needed).
• The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
• I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

[takahser]

Hi @liuchengxu

Thanks for applying again to our grants program (previous application see here, for those unaware).

I see that you've pivoted to developing a Bitcoin full node rather than a light client, when compared to your previous application. While I think this is a very interesting idea, I'm not fully convinced that it would greatly benefit the Polkadot Ecosystem. If you disagree, please explain why.

[liuchengxu]

@takahser Thanks for your feedback. I'd like to address your concerns and clarify the broader vision for this project.

While this project might seem indirectly related to Polkadot at first glance, its implications are quite significant for the Substrate framework (polkadot-sdk) as a whole, and by extension, Polkadot.

By leveraging Substrate’s advanced state sync strategy, we can achieve faster and more efficient full node implementations. This not only benefits Bitcoin by increasing its decentralization but also demonstrates the power and flexibility of the Substrate framework, attracting more developers to the polkadot-sdk ecosystem, which in turn strengthens Polkadot’s position in the blockchain space.

[liuchengxu]

I also want to add that by integrating a Bitcoin full node in Substrate, new possibilities are opened up for developers to build on Substrate with integrated Bitcoin functionalities, enhancing the overall utility and appeal of Bitcoin-related projects in the Polkadot ecosystem. For instance, the existing BTC bridge projects like ChainX and interlay using a BTC light client both require the role of relayer, with the Bitcoin networking feature developed in this application, we can directly pull blocks and transmit the transaction to the nodes of the Bitcoin network, without any need for intermediaries that might censor them.

[PieWol]

Hey @liuchengxu, thanks for the application. I would be interested in what you deem the biggest challenges when implementing this decentralized bitcoin full node.

[liuchengxu]

@PieWol Thank you for your comment. This is an interesting point. The biggest challenge so far is achieving block execution performance comparable to other leading Bitcoin full node implementations. The state sync feature offered by Subcoin, while advantageous, introduces additional overhead due to the state trie, which significantly impacts block execution performance.

Based on my local experiments, after processing 300,000 blocks, the block execution becomes increasingly slower, executing a single transaction takes over one millisecond, and the average block time exceeds one second. Consequently, the initial full block sync required by bootstrap nodes may take several days, which is considerably longer than the best performance of around 8+ hours, as detailed in this post.

There are ongoing efforts to enhance Merkle trie performance, such as the work being done by thrumdev/nomt. These improvements could significantly help on this front. I will also focus on performance optimization once the prototype is fully developed, although not every Subcoin node has to do the full block sync.

[takahser]

By leveraging Substrate’s advanced state sync strategy, we can achieve faster and more efficient full node implementations. This not only benefits Bitcoin by increasing its decentralization but also demonstrates the power and flexibility of the Substrate framework, attracting more developers to the polkadot-sdk ecosystem, which in turn strengthens Polkadot’s position in the blockchain space.

While I agree that there might be a benefit for Substrate and the Polkadot SDK, I think the benefits are rather indirect than direct. Personally, I don't think it's a strong enough case to get support through the grants program.

For instance, the existing BTC bridge projects like ChainX and interlay using a BTC light client both require the role of relayer, with the Bitcoin networking feature developed in this application, we can directly pull blocks and transmit the transaction to the nodes of the Bitcoin network, without any need for intermediaries that might censor them.

That sounds interesting indeed. Does that mean the interlay relayers are centralised? Could you explain both mechanisms in detail, so we can get a better understanding?

[liuchengxu]

That sounds interesting indeed. Does that mean the interlay relayers are centralised? Could you explain both mechanisms in detail, so we can get a better understanding?

The relayers of a Bitcoin bridge have two main tasks: a) They listen to the Bitcoin network, submit Bitcoin headers, and deposit transactions to the target chain (such as Interlay or ChainX) as needed. b) They broadcast withdrawal transactions from the target chain to the Bitcoin network.

The relayers for Interlay and ChainX operate as peripheral programs. They connect to a remote Bitcoin Core node using RPC. This setup theoretically allows for a permissionless system since the code is open-source, meaning anyone can build the program and run a relayer. However, in practice, the situation tends to be more centralized. Despite the open-source nature, the motivation to run the relayer program is often low, leading to a scenario where only the project builders are actively maintaining the relayers. This creates a single point of failure, as the relayer role is effectively centralized. If all the relayers go offline, the chain is paused.

In contrast, ICP offers a superior integration by implementing the Bitcoin network at the protocol level. ICP nodes can directly communicate with the Bitcoin P2P network without the need for an additional peripheral program, every ICP node serves as a relayer. The Bitcoin network crate developed in Subcoin will facilitate the Bitcoin network integration into a Substrate node.

[takahser]

In contrast, ICP offers a superior integration by implementing the Bitcoin network at the protocol level. ICP nodes can directly communicate with the Bitcoin P2P network without the need for an additional peripheral program, every ICP node serves as a relayer. The Bitcoin network crate developed in Subcoin will facilitate the Bitcoin network integration into a Substrate node.

Ohh, so the pallet basically directly encapsulates the bitcoin light client, similar to how BEEFY can bridge with EVM chains, right? But unidirectional, meaning we can bridge BTC to your substrate-based chain and back, but we can't bridge DOT or your native token to Bitcoin. Is that correct?

[liuchengxu]

Ohh, so the pallet basically directly encapsulates the bitcoin light client, similar to how BEEFY can bridge with EVM chains, right? But unidirectional, meaning we can bridge BTC to your substrate-based chain and back, but we can't bridge DOT or your native token to Bitcoin. Is that correct?

That's correct. The on-chain pallet of interlay/chainx consists of a Bitcoin light client and the logic of processing BTC transfers. The bridge focuses on BTC only, there is no reason to bridge DOT or any other native token to Bitcoin anyway, it's not suitable for busy on-chain activities after all :P

[takahser]

The bridge focuses on BTC only, there is no reason to bridge DOT or any other native token to Bitcoin anyway, it's not suitable for busy on-chain activities after all :P

Makes sense. Yes, I tend to agree with that lol.

What are your long-term plans for this project, in case we decide to fund it?

[liuchengxu]

@takahser Regarding the long-term plan for this project, I have several goals. First and foremost, I aim to make its performance, particularly in syncing, comparable to other leading Bitcoin full node implementations. This will ensure it becomes a viable option for running a Bitcoin node.

Secondly, I want to identify and address use cases for Subcoin. I have noticed some complaints that Bitcoin developers are currently facing with Bitcoin Core, and this project could potentially offer solutions to some of these inconveniences.

Lastly, I plan to seek financial support for its development and maintenance in the long run to ensure its sustainability. Your agreement and support would greatly benefit the success of this project as an endorsement.

[PieWol]

@PieWol Thank you for your comment. This is an interesting point. The biggest challenge so far is achieving block execution performance comparable to other leading Bitcoin full node implementations. The state sync feature offered by Subcoin, while advantageous, introduces additional overhead due to the state trie, which significantly impacts block execution performance.

Based on my local experiments, after processing 300,000 blocks, the block execution becomes increasingly slower, executing a single transaction takes over one millisecond, and the average block time exceeds one second.

Hey @liuchengxu ,
thank you for the insights and the article you linked. So basically unless there is a solution found to the state trie problem the whole project is infeasable as we have steadily growing times to sync a block? How can this issue be fixed so that the chain is able to scale with the increasing bitcoin block height?

[liuchengxu]

@PieWol Not necessarily. As I mentioned earlier, this issue primarily affects the Subcoin bootstrap node. However, not everyone needs to run a Subcoin bootstrap node using the full sync method. Newly joined nodes can leverage state sync to sync to the tip of the network quickly. Therefore, most users can still benefit from Subcoin's fast sync feature even if the state trie problem persists. The Bitcoin state sync in a decentralized manner is the Subcoin's most significant contribution.

Based on current experimental estimations, Subcoin's initial full sync performance is unlikely to be the worst, even with the additional state trie overhead. Other implementations like BTCD and Libbitcoin Server also take several days to complete the initial sync.

[liuchengxu]

Hey, @takahser. No rush, but is there anything I can do to help move this PR forward?

[takahser]

@liuchengxu sorry for the delay, have been busy recently and I'm now catching up with unread notifications.

Secondly, I want to identify and address use cases for Subcoin. I have noticed some complaints that Bitcoin developers are currently facing with Bitcoin Core, and this project could potentially offer solutions to some of these inconveniences.

Do you happen to have any links to share here?

[liuchengxu]

Do you happen to have any links to share here?

I don't have any specific links at the moment. My observations are based on discussions with developers within my network, which are wallet-related topics if I recall correctly. I anticipate that we'll see some specific demand from developers once we achieve a solid implementation in Substrate.

[liuchengxu]

@PieWol I did some Bitcoin-specific optimization within the substrate framework recently, mostly from the conventional engineering aspects like using an in-memory backend and off-runtime execution, the initial full sync performance of the bootstrap node has been significantly improved by more than 5x, and now the initial full sync is expected to take less than one day. This is not the end, chances are great that there will be more optimizations in the future, even if the state trie issue persists.

Ping @takahser I'm happy to chat offline if needed.

[PieWol]

Hello @liuchengxu ,
I would like to have a chat with you to better understand the application. Could you please send a meeting invite for a short meeting to piet@web3.foundation if you are interested? Thanks.

[liuchengxu]

Hey @PieWol You should have received the invite in the email, feel free to suggest another time if that does not work for you, I'm pretty flexible.

[takahser]

@liuchengxu sorry for the delay here, I was planning to review this in more depth, but I think a call sounds good as well. I'm planning to join the call as well. 👍

[liuchengxu]

@PieWol @takahser Thanks a lot for the meeting! The application has been updated with minor changes based on the offline discussion, please see the commit message for the main points. Let me know if anything is still missing.

[takahser]

@liuchengxu thanks for your time in today's call. Just one last question: Could you add more details on how a parachain is going to be able to interact with the bitcoin network using your client?

[liuchengxu]

@PieWol @takahser Please help move this PR forward when you have time. I'm more than happy to have another call if you have any further questions. Thank you!

[takahser]

@liuchengxu thanks for your time today. Given the bitcoin blocks are wrapped and passed to the runtime's execute_block function and any Substrate-based chain can re-use it, I'm happy to add my approval as well.

[PieWol]

Thanks for explaining your project further in our meetings. I'm approving it.

[semuelle]

Hi @liuchengxu, welcome back. Thanks for the application. I am also happy to support the project. I only have one request: Beside the unit tests, could you also provide tutorials for each milestone, e.g. how to author and send transactions?

Are you familiar with Tuxedo? You might be able to leverage some of their work.

[liuchengxu]

Beside the unit tests, could you also provide tutorials for each milestone, e.g. how to author and send transactions?

Hey @semuelle This was deliberately removed in 8823785 as authoring and sending transactions does not make a lot of sense here. In the meetings with PieWol and takahser, I clarified that Subcoin does not expose any callable extrinsics to users, we'll only support broadcasting a constructed Bitcoin transaction to Bitcoin network via subcoin RPC in this application, as one deliverable in M3. The core of this application is to implement a Bitcoin full node that can do an initial full sync from the Bitcoin network and provide the fast sync feature for the newly joined nodes. Constructing a Bitcoin transaction is a feature of a Bitcoin wallet, which is out of the scope for this application.

Tuxedo looks interesting, but with a quick look, we have much different goals. Subcoin is essentially a replica of the Bitcoin chain, the subcoin runtime is a minimalist Substrate runtime, containing only one storage item for the UTXO set, while Tuxedo primarily focuses on developing a brand new chain using the UTXO model. Thanks for the link anyway. We'll see if there is anything we could leverage and collaborate.

[semuelle]

we'll only support broadcasting a constructed Bitcoin transaction to Bitcoin network via subcoin RPC in this application

Thanks for the quick reply. That's fine. I'm just looking for ways to test the deliverables beyond unit tests. But if that's only available in M3, then the spec of 2.0b doesn't make sense ("spin up one of our Substrate nodes and send test transactions"). It looks like all 0b specs are still the template text, could you review them?

[liuchengxu]

Good catch! @semuelle That's the leftover from the last commit. I have fixed it in the lastest push. We'll provide the instructions to run a Subcoin node, but no sending test transactions.

[semuelle]

Thanks for the update. Happy to approve.

[takahser]

re-approving

[PieWol]

Thanks for refining the deliverables @semuelle and @liuchengxu.

[github-actions]

Congratulations and welcome to the Web3 Foundation Grants Program! Please refer to our Milestone Delivery repository for instructions on how to submit milestones and invoices, our FAQ for frequently asked questions and the support section of our README for more ways to find answers to your questions.

Before you start, take a moment to read through our announcement guidelines for all communications related to the grant or make them known to the right person in your organisation. In particular, please don't announce the grant publicly before at least the first milestone of your project has been approved. At that point or shortly before, you can get in touch with us at grantsPR@web3.foundation and we'll be happy to collaborate on an announcement about the work you’re doing.

Also, if you haven't yet, consider signing up for the Polkadot Alpha Program. The program offers plenty of resources for people building on Polkadot. Lastly, please remember to let us know in case you run into any delays or deviate from the deliverables in your application. You can either leave a comment here or directly request to amend your application via PR. We wish you luck with your project! 🚀

[semuelle]

Hi @liuchengxu, could you fill out the KYC form (assuming you are applying as an individual, otherwise please use this form)?

[liuchengxu]

Hey @semuelle I have finished the form and the verification was successful.

[semuelle]

Confirmed!