Move blockchain-facing logic to virtualchain

[jcnelson]

In an effort to better wrangle Blockstack's packages, especially in light of interfacing with multiple blockchains, I propose doing the following:

• make blockstack-virtualchain the package you install to:
  • read and write transactions
  • connect to specific blockchains
  • query blockchain state
• make keylib the package you install to:
  • create blockchain keys and addresses

blockstack-virtualchain would also subsume blockstack-utxo.

[muneeb-ali]

I agree with this in general.

Debating if "query blockchain state" should be in virtualchain or not.

I was actually thinking of having a package that just has networking interfaces i.e., support to talk to popular bitcoin API providers for UTXOs etc, along with support to talk to bitcoind, even DHT etc.

Historically, there was a "coinrpc" package (bad name!) that was merged into pybitcoin and now we're thinking about taking that functionality out and separating it again.

Thoughts?

[muneeb-ali]

In other words, the bigger question is that should virtualchain have support for fetching information from a wide variety of sources or should it import some package, let's call it networking-interfaces for now, to fetch that information. One way to think about it is are there other packages that would need to import this networking-interfaces package.

[muneeb-ali]

I think the answer is yes e.g., let's say I'm building a monitoring service and I need to fetch information over the network but I don't need any virtualchain functionality.

[jcnelson]

That's fine with me. I'm taking a crack at removing the pybitcoin dependency in virtualchain, and so far I've gotten away with including only a few files from pybitcoin (hash.py, merkle.py, opcodes.py). No need for utxo clients yet, but blockstack-server will need them (maybe those can just go into blockstack-utxo? or blockstack-blockchain-clients? or the like).

[muneeb-ali]

Yeah so blockstack-utxo is already the kind of "networking interface" library I had in mind.

How about we rename it to blockstack-rpc or something along those lines? It can be the single repository where we implement all remote API / RPC functionality. Even the client can import this later on.

The library only deals with fetching data from a remote endpoint and delivering it to a client and that's it. No other logic needs to be in this package. This can help us get rid of the "basicrpc" dependency as well. What do you think?

[jcnelson]

Sounds good. How about renaming it to "blockstack-blockchain-rpc"? It will contain all the UTXO clients, as well equivalent RPC clients for other blockchain services.

[muneeb-ali]

Would there be any other rpc support in it other than blockchains? I was thinking of even merging the blockstack proxy / basicrpc stuff in this. If non-blockchain stuff is also included, then we can have a structure like:

blockstack-rpc/blockchains/bitcoin.py
blockstack-rpc/blockchains/ethereum.py
blockstack-rpc/blockstack/blockstack_server.py
blockstack-rpc/blockstack/blockstack_dht.py
blockstack-rpc/services/blockcypher.py

and so on

[jcnelson]

I was thinking of keeping the low-level blockchain proxies in virtualchain (i.e. proxies that talk directly to blockchain daemons like bitcoind and geth), and putting high-level service proxies (i.e. block explorers) in blockstack-blockchain-rpc. Maybe blockstack-blockchain-services is a better name.

To answer your other question, the organization I'm taking with virtualchain is:

blockstack-virtualchain/virtualchain/lib/blockchain/bitcoin/...
blockstack-virtualchain/virtualchain/lib/blockchain/ethereum/...
blockstack-virtualchain/virtualchain/lib/blockchain/hyperledger/...

The approach I'd take with blockstack-blockchain-services would be:

blockstack-blockchain-services/blockchain-services/blockcypher/...
blockstack-blockchain-services/blockchain-services/blockchaininfo/...
blockstack-blockchain-services/blockchain-services/ethchain/...

[muneeb-ali]

Ah got it.

Going back to the monitoring service example. Let's say I want to build a service that

a) checks the current bitcoind block from a public bitcoin server
b) checks the current bitcoind block from blockcypher
c) checks the current bitcoind block returned by blockstack-server

I'll need to import virtualchain and blockchain-services and some third library (probably blockstack-client) to be able to do this correct?

[jcnelson]

In this factoring of the world, yes.

I could see our organization going one of two ways. The first is that we group functionality by interface (which is what I'm proposing). We put all the low-level blockchain RPC stuff in one package, and we put all the higher-level RESTful service clients (e.g. UTXO providers) into another package.

The other possible organizing principle is to group functionality by blockchain. We'd package all the UTXO service clients along with the Bitcoin blockchain clients, we'd put all the Ethereum service APIs alongside the low-level Ethereum blockchain client, etc. I think this is closer to the way things are with pybitcoin today, and we'd also create a "pyethereum" package to complement pybitcoin.

In both universes, the blockstack-client would be a separate package, since it needs to aggregate both multiple blockchains and multiple services anyway (plus it needs to talk to blockstack-server, which is doesn't really fall into either category).

[jcnelson]

Elaborating on this a bit more, I think the factor-by-interface approach makes packaging easier than factor-by-blockchain. This is because the blockchain interfaces are lower in the stack, have simple implementations, have less dependencies, and are better maintained in practice. Also, more software needs the blockchain interfaces than the service interfaces.

Service interfaces are definitely higher in the stack, and in the limit can be installed on an as-needed basis if packaging gets too hard (i.e. we might eventually forgo an all-services-in-one package in favor of an a-la-carte blockstack install service-RPC-package approach). If blockstack spans multiple blockchains in the default case, then there probably won't ever be a case where you install blockchain clients a-la-carte; you'll need all of them for the system to work in the first place.

[muneeb-ali]

I see the difference between actual RPC vs. higher-level RESTful and why that division can make sense.

Question: do you think that virtualchain is going to be a "heavy package" with lots of dependencies or a light package that I can easily install e.g., I just want to talk to bitcoind for whatever reason.

If it's going to be a light package then it's fine. I can do

from virtualchain import bitcoin_client

Or something like that. Logically we can think of virtualchain as something that has cross-chain support.

[jcnelson]

I'm aiming to make it as light as possible, but no matter what we do it will depend on whatever supporting packages are needed by the blockchain clients themselves. I'm banking on these being small in practice--for example, virtualchain will only depend on bitcoin (formerly pybitcointools), python-bitcoinrpc, and utilitybelt for Bitcoin support (and python-bitcoinrpc is so small and simple and stable we could just fork it and keep our own internal version within blockstack-virtualchain if need be). If we can keep it to one or two small packages per blockchain, we'd be doing better than even a single RESTful API client.

[muneeb-ali]

Sounds good, +1 from my side. We can let others chime in as well.

[grctest]

Really looking forwards to new changes to pybitcoin/blockstack-virtualchain, I was looking to integrate gridcoin into pybitcoin (https://github.com/grctest/pybitcoin) in order to match several thousand users BOINC CPIDs to alternate crypto accounts/addresses using virtualchain on the Gridcoin blockchain, I'll hold off until blockchain-facing logic has been moved around.

Keep up the good work guys, your blockstack tech looks great! 👍

[jcnelson]

Update: blockstack-virtualchain will also depend on protocoin once we move away from Bitcoin's RPC interface and start talking to them directly on their peer-facing interface (which will be more efficient). Also, it paves the way for moving SPV and SNV implementations out of the blockstack client, since SPV is a Bitcoin-specific protocol that belongs in virtualchain anyway (so we can validate an untrusted txindex bitcoind node using a locally-trusted bitcoind node) and since SNV will need to be offered for each blockchain (since it's specific to the virtualchain protocol itself).

[grctest]

Hey @jcnelson @muneeb-ali ,

How is Blockstack getting on moving the blockchain facing logic from here to blockstack-virtualchain?

I am very interested in potentially running Blockstack on the Gridcoin blockchain but have been holding off from attempting to do so because the blockchain facing logic was moving about.

Cheers guys :)

[muneeb-ali]

You should create an issue on the Virtualchain repo: https://github.com/blockstack/blockstack-virtualchain/issues

In general, we are thinking about supporting multiple blockchains. For each blockchain that we support we need to write a "blockchain driver" in virtualchain.

Would be great to chat about how you're thinking about writing such a driver for Gridcoin.