-
Notifications
You must be signed in to change notification settings - Fork 668
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature Request] Modification of Mining Module #1969
Comments
Thanks Gavin! Just to echo the above, we think some of the changes we're making a lot of sense to be offered officially by Blockstack PBC. The first we're in the process of working on, and the second we've created one possible solution for. Appreciate the consideration @muneeb-ali @diwakergupta @kantai @jcnelson |
Hey @tyGavinZJU @xanbots, appreciate your input. This is something that @kantai, @lgalabru, and myself have discussed in the past (as early as April), and we have a pretty good idea about what mining is going to look like in the future. If you want to help us with this, there's a specific architecture we had in mind that we'd like you to use. The biggest difference between how the node behaves right now and how the node will be have in the future is that most mining activity will occur in a separate program (#1441). The mining component was never intended to be part of the Stacks node. It's only there now because it was the fastest way to enable mining on the testnet. What we intend to do -- and would accept PRs for -- is to split the mining component out of the Stacks node to create a separate mining program, and expand the RESTful API to make it possible for the mining program to generate and broadcast Stacks blocks and Bitcoin transactions. A Stacks mining program would need to do the following:
All of this currently happens within the Stacks testnet software, and could be split out into a separate Rust binary. However, the Stacks node still needs the following RESTful API endpoints at a minimum to make this happen:
|
A quick question, does the
|
Synchronizing your own regtest bitcoind locally with the one krypton uses is easy: bitcoin.conf
Start bitcoind
and to query with bitcoin-cli:
not sure why it can't pick up the rpcport from the conf file, but whatever network operations are whitelisted are sufficient for the node to synchronize, and now you have access to all rpc calls. |
Here are are some thoughts on how I would approach mining, and incrementally splitting some of the mining responsibilities. First of all, the mining process needs to be driven by the node, which reacts to other nodes when it receives new btc blocks, new stacks blocks and new transactions. So when considering an RPC API, the miner is the server, and the node a client to the miner, in the same fashion the stacks-blockchain-api node is a server and the stacks-blockchain node a client. Secondly, the more granular you make the api, the more serialization/deserialization cost you'll be incurring, slowing down the process, and making it more complex (and also more brittle), so you need to keep the exchanges to a minimum (when speed may matter). I started writing this thinking you'd split the current mining code into smaller chunks, the node sending data needed for each basic operation (register key, generate proof, build block, sign and commit, new block received, new transaction received, ...), but it turns out building a block outside of the node is not really feasible without pulling most of the node code (Clarity VM, including its state for example), so here's a more nimble proposal. The node keeps managing the Leader Key, and assembles the block, and once the block is assembled, calls the miner to let it decide whether it wants to mine the block (based on the current chain tip, the tx count, the tx fees, and a few other parameters) and how much it wants to transfer to stackers, and what fee to pay to the btc network, or the miner can decide to skip this block. The node will call the miner each time it detects a new chain tip, which can be more than once per epoch (btc block), but gives the miner the best chance of mining a block on the best known fork for the current epoch as the node learns about it. This should be fairly simple and only a few modifications to the current node code, and a few extra configuration parameters (miner's address and port may be sufficient), and can be modeled on the event dispatcher (or even be an extension of it with a few more events). A second version could let the miner pick which transactions to include, and in what order (a first call the node would make to the miner), then let the node assemble the block, and again, the node would call the miner to let it decide once the block has been assembled, whether to commit to it or not. The miner could already reuse the existing events used to get new transactions and blocks so it could manage its own mempool and does not need to get a copy of the mempool for each block). This does not take microblocks into accounts, but once this has been enabled, the miner would also need to decide which transaction to mine in a microblock, and when to stop mining microblocks for the current epoch, some fairly simple addition to the protocol. |
Interesting idea @psq ! There's some elegance in re-using the event dispatcher pattern to "push" updates to the miner software. In this model, would the miner program would still make API calls into the node to upload the block etc (basically the set of endpoints in Jude's earlier comment)? I think a quick diagram (whiteboard picture would also work) might help clarify what you have in mind. @blockstack/blockchain-team please chime in with your thoughts! 🙏🏽 |
All you need to do to get the node to produce a block template is add a RPC endpoint that takes a VRF proof, a microblock public key hash, and a list of txids to mine as input, and spits out an unsigned Stacks block that includes those transactions in the order given by the txids (or, returns an error of the node is missing one or more of those transactions or if the transactions cannot be applied in the given order). Internally, the node would use a There's absolutely no need at all to involve the event-dispatcher. It's the mining program's job to identify transactions for inclusion, not the node's. Moreover, the node shouldn't be producing blocks that the mining program didn't ask for, and it shouldn't be polluting the event stream (to which the side-car is attached) with data that it didn't want in the first place. Making a block template is expensive and time-consuming, and should only be done at the request of the miner (implying that this RPC endpoint should be privileged, and/or accessible only from Let's not over-complicate the design. |
@jcnelson the miner is most likely going to need specific events that the explorer would not find interesting (btc blocks information for example, and yes, it could connect directly to bitcoind, but duplicating work is not ideal), so the explorer will eventually need to subscribe to a subset of the events available (that's already something that is available), so it is not much of a stretch, if push is a model that would work better, to extend the event model, and I don't see how that would be "polluting" per se. And it seems anyway we are saying very similar things, and mostly disagree on whether the node is making the http requests or the miner. @diwakergupta in both proposals, the block is still assembled by the node (which has the state and the Clarity VM), but the miner decides which transactions to include and in what order (by providing an ordered tx list). Except my first proposed step that would skip that choice to get to a working solution faster, and the only decision the miner would make is whether to mine a block or not for a given epoch, and how much to pay for it. With all that said, now that I understand more precisely what @jcnelson was suggesting, I think I could create a crude prototype for a miner in a few days by adding one RPC call, and maybe a few extra events that can then be extended by @xanbots and @tyGavinZJU. That prototype miner would pick the transactions to mine, set the btc spend and decide when and whether to mine a new block. I was planning to work on #1981 tomorrow, but I'd be happy to get started on the above right away. Let me know. |
Awesome discussion and summary @psq. Several quick questions @psq:
Thank you! |
@jcnelson Very clear diagram! Thanks. As we discussed before in zoom, I am working on adding 2 API calls in stacks-node:
Developing two APIs above will give great convenience to the community to participate in mining and it's not that hard to implement compared to mining-program. I want to make sure if there is a big conflict between @blockstack/blockchain-team's proposal and what I am working on. If there is, I will stop and discuss in-depth. @jcnelson |
@tyGavinZJU
If you create a new repo, a fork essentially, you'll forgo the improvements or have to keep merging the changes from the upstream repo, so changes should go in stacks-node.
Just adding these 2 api calls would most likely result in throwaway code, and would not allow implementing more than a basic miner, and if that's what you intend to do short term, then this might be best done in a separate repo, despite what I said above.
The crude part would be on the mining logic, to let you implement what you mentioned (start/stop mining, and set the burn fee. Picking the transactions, and I'm simplifying, means providing the equivalent of what |
Thanks everyone for the discussion. To summarize and ask for a point of clarification:
What Daemon wants to do is first release a mining bot that makes it easy for anyone to mine in time for mainnet launch, as we've planned. We will then work on a full release of the mining bot that can use the separate mining program/module that will be developed later. @diwakergupta @jcnelson @psq What do you think might be the difference in experience STX Mining at mainnet launch compared to the experience mining today on testnet? I appreciate it may be a tough question to answer at the moment, but any information will give me a clearer picture of what Daemon can try to build for mainnet launch. Right now we're still not clear enough on this to start any development. Thank you. |
I think it's best to keep the issue focused on what sort of "miner interface" will exist as part of stacks-blockchain proper in time for mainnet: it's not germane whether that's developed by PBC or others. The expectation is that there will be set of stable API endpoints in stacks-blckchain in time for mainnet that would allow anyone interested to create their own miners with relatively modest effort.
There's alignment on the architecture described earlier in this issue, and I believe @psq is actively working on implementing that. @xanbots does that clarify? |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
This is important to me. |
|
@jcnelson |
Hi @tyGavinZJU, (1) I'm not sure I understand the ask? If operated with an event observer, the node will synchronously POST all events to the observer as it processes blocks. It's up to the observer to do useful things with this data, such as store it in a database and serve it later via a Web server. The node does not (and cannot) know what the downstream sinks need. (2) That can be added. Can you an open an issue for it? I'll try and get it in for the next release. (3) The success response is HTTP 200, with a body that is a JSON blob with the following schema:
The
The (4) Right now, the mock miner does not attempt to mine microblocks because its block never wins. We'd have to add that functionality. |
@jcnelson, it is the current diagram above. Several questions below: |
(1) The observer should be posting mempool events to (2) I think there needs to be a new API endpoint for reading snapshot state, given either a burnchain hash or a burnchain height. Can you open an issue for it? (3) You will get an HTTP 200 response on success, with a JSON body that contains the index block hash of the block and a boolean flag indicating whether or not the block was accepted into the database (you will get |
Is there an observer option in the |
Yeah -- it looks like this:
For example, you'd want Your config file can have multiple |
Closing due to inactivity. This is being taken up by https://github.com/stacksgov/Stacks-Grant-Launchpad/issues/790 |
Hello @jcnelson ! We will build on top of the miner module so that it will read from the smart contract, know how much the miner should spend on each transaction, and the miners will communicate with each other. But we are not restructuring the miner module code itself. We are not developing the functionality of dynamically running/stopping the miners for each block, that would also be counterproductive for the mining pool flow as they should leave the smart contract pool before stopping the BTC contributions sent in each block. The amount sent by each miner is picked from the smart contract as the total contribution amount of the pool is split evenly between the miners, so the amount will not be dynamic in that sense. |
@BowTiedDeployer the smart contract will be queried for each tx? If I provide an api similar to the smart contract, then I could change the bitcoin commit for each tx? |
Is your feature request related to a problem? Please describe.
Right now the official code for a Stacks 2.0 node does not readily expose information/data/API that is both necessary and desired by STX Miners. This extensively limits the operations an individual miner can easily execute, as well as the ability for third parties to provide STX Mining solutions that make it easy for anyone to participate. The two biggest issues are described in detail below:
The operating mechanism of existing nodes is to read configuration information from the configuration file of .toml, such as whether to mine, miner address, etc. This method of statically reading files limits the operations related to mining. For example, starting and stopping mining are currently controlled by opening and closing the node program, and the node controller cannot change the behavior during the node operation. (For example, only normal node status at the beginning of the node, and then perform mining operation later).
Specifically, the two missing features that are of highest priority are:
Describe the solution you'd like
Daemon Technologies has, up until now, been modifying the official Stacks Blockchain code in order to facilitate the above. These changes, however, are unofficial/not part of master, and therefore need to be redone every time Blockstack PBC changes the official code. Given that the changes we propose will be necessary, and large value add for any and all STX Miners, in addition to data that will be important for other products such as the Explorer, Daemon Technologies would like to propose working with Blockstack PBC to merge a solution to the issues described above to the master/official Stacks Blockchain code.
The initial solution Daemon Technologies is proposing to the issues above is as follows:
Since the pre-run check and runtime state changes of existing nodes are both in run_loop, the following points need to be adjusted at this stage:
Describe alternatives you've considered
If Blockstack PBC is not willing to work with Daemon Technologies to merge these features to master at this stage(maybe because of time/energy/priority), Daemon plans to continue it's development, but with a delayed timeline. If these features are not offered in the official Stacks chain code, Daemon will need to take more time to develop and release the STX Mining bot. If these changes are merged to master and officially offered, we believe we can release a fully usable version of the mining bot by mainnet launch. If not, we anticipate the official release of the mining bot occuring 1-2 months post mainnet launch.
Additional context
The text was updated successfully, but these errors were encountered: