Bitcoin Core 0.18 RPC

[cwaldron97]

RPC for Bitcoin Core 0.18.0
All new RPCs have been implemented for V18 currently working on changing deprecated calls currently in the code base. There are tests for most of the RPCs I might be missing 1 or 2 of the node/network RPCs as those are a bit tricky to test. Need to implement newly enumerated warning messages as mentioned in https://github.com/bitcoin/bitcoin/blob/eb7daf4d600eeb631427c018a984a77a34aca66e/src/rpc/protocol.h#L32
Release notes:
https://bitcoincore.org/en/releases/0.18.0/

Update (8/26)
This PR addresses all V18 RPC calls and 18.1 has been considered when writing. There are also organizational things in the code base to make testing and things like that possible. As of now all tests are passing except getnodeaddresses and technically updateutxo as both of those require 2 nodes to be spun up and currently that requires generatetoaddress but we currently use generate. In a new PR I will fix the usage of generate to make to usable for future releases as it is now removed so moving forward it will be rather important we have it in the code base. Also within the new PR once I have implemented that I will fix the getnodeaddresses and updateutxo test so they are more useful/will pass. Current state is that when testing with CI V16 or V17 will fail a few random tests because of changes implemented for V18 for example (getpeerinfo ) but with otherwise pass everything else.

#734 closes this

[nkohen]

I don't think it currently solves #423, and would you mind linking to the release notes that mention this new RPC?

[Christewart]

In general, look through the CI logs after they are done running and you can verify that your implementation doesn't have compiler errors 👍

[torkelrogstad]

Would be great if you could open and save BitcoindInstance.scala, RpcPsbtResult.scala, JsonSerializers.scala and BitcoindRpcTestUtil.scala. We will hopefully end up with a diff that's more readable:-)

[torkelrogstad]

This test doesn't really test the functionality of utxoupdatepsbt. From looking at the docs, it looks like it takes in a PSBT, and updates it with information found in our node. This is probably a bit hard to test, so I'd just leave this out for now, and we can come back to it later.

[torkelrogstad]

Because you added a field to Peer you have to change some JSON readers. You'll have to add this in JsonSerializers:

  implicit val satsPerKbReads: Reads[SatoshisPerKiloByte] =
    new Reads[SatoshisPerKiloByte] {

      def reads(json: JsValue): JsResult[SatoshisPerKiloByte] =
        SerializerUtil.processJsNumber(num =>
          SatoshisPerKiloByte(Satoshis(Int64(num.toBigInt()))))(json)
    }

And then change peerReads to:

  implicit val peerReads: Reads[Peer] = Json.reads[Peer]

[Christewart]

984d8bf does not compile for me

[error]     Json.reads[SubmitHeaderResult]
[error]                ^
[warn] /home/chris/dev/bitcoin-s-core/bitcoind-rpc/src/main/scala/org/bitcoins/rpc/serializers/JsonSerializers.scala:5:29: Unused import
[warn] import java.util.concurrent.TimeUnit
[warn]                             ^
[warn] /home/chris/dev/bitcoin-s-core/bitcoind-rpc/src/main/scala/org/bitcoins/rpc/serializers/JsonSerializers.scala:34:34: Unused import
[warn] import scala.concurrent.duration.FiniteDuration
[warn]                                  ^
[warn] 13 warnings found
[error] three errors found

[nkohen]

In the None case this turns into "null", is this what you want? Or should you just implement the two public getNodeAddresses calls individually?

[cwaldron97]

Torkel suggested I write it this way to have all possible cases I believe. If the two calls individually work better I can implement that

[nkohen]

I could be wrong but I was under the impression that @torkelrogstad implemented a type for PSBTs?

[nkohen]

same comment applies for the two defs below

[cwaldron97]

He did not implement a type for PSBTs it was in a list of his todo's he wanted to add but never got to it. I believe there is an open issue for it right now

[Christewart]

#551

[nkohen]

ping

[Christewart]

Attempted to pull down and run f027c23 this morning and these were my error messages

07:38:35.143 ERROR BitcoindV18RpcClient - Error when parsing result of 'getdescriptorinfo': {"obj.hasprivatekey":[{"msg":["error.path.missing"],"args":[]}]}!JSON: {"descriptor":"pk(0279be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798)#gn28ywm7","isrange":false,"issolvable":true,"hasprivatekeys":false}
[info] BitcoindV18RpcClientTest:
[info] BitcoindV18RpcClient
[info] - should be able to start a V18 bitcoind instance
[info] - should return active rpc commands *** FAILED ***
[info] akka.stream.scaladsl.TcpIdleTimeoutException: TCP idle-timeout encountered on connection to [localhost:24734], no bytes passed in the last 5 minutes
[info] - should return a list of wallets *** FAILED ***
[info] 0 was not greater than or equal to 25 (BitcoindV18RpcClientTest.scala:48)
[info] org.scalatest.exceptions.TestFailedException:
[info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)
[info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529)
[info] at org.scalatest.AsyncFlatSpec.newAssertionFailedException(AsyncFlatSpec.scala:2219)
[info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:503)
[info] at org.bitcoins.rpc.v18.BitcoindV18RpcClientTest.$anonfun$new$12(BitcoindV18RpcClientTest.scala:48)
[info] at scala.util.Success.$anonfun$map$1(Try.scala:255)
[info] at scala.util.Success.map(Try.scala:213)
[info] at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[info] at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[info] at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[info] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[info] at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
[info] at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92)
[info] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info] at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
[info] at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92)
[info] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
[info] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
[info] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[info] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[info] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[info] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[info] - should analyze a descriptor *** FAILED ***
[info] java.lang.IllegalArgumentException: Could not parse JsResult: {"descriptor":"pk(0279be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798)#gn28ywm7","isrange":false,"issolvable":true,"hasprivatekeys":false}! Error: Error when parsing result of 'getdescriptorinfo': {"obj.hasprivatekey":[{"msg":["error.path.missing"],"args":[]}]}!
[info] at org.bitcoins.rpc.client.common.Client.parseResult(Client.scala:334)
[info] at org.bitcoins.rpc.client.common.Client.$anonfun$bitcoindCall$2(Client.scala:249)
[info] at scala.util.Success.$anonfun$map$1(Try.scala:255)
[info] at scala.util.Success.map(Try.scala:213)
[info] at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[info] at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[info] at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[info] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[info] at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
[info] at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92)
[info] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info] at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
[info] at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92)
[info] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
[info] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
[info] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[info] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[info] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[info] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[info] - should get node address given a null parameter *** FAILED ***
[info] java.lang.UnsupportedOperationException: empty.head
[info] at scala.collection.immutable.Vector.head(Vector.scala:189)
[info] at org.bitcoins.rpc.v18.BitcoindV18RpcClientTest.$anonfun$new$18(BitcoindV18RpcClientTest.scala:72)
[info] at scala.util.Success.$anonfun$map$1(Try.scala:255)
[info] at scala.util.Success.map(Try.scala:213)
[info] at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[info] at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[info] at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[info] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[info] at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
[info] at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92)
[info] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info] at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
[info] at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92)
[info] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
[info] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
[info] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[info] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[info] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[info] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[info] - should get node addresses given a count *** FAILED ***
[info] java.lang.UnsupportedOperationException: empty.head
[info] at scala.collection.immutable.Vector.head(Vector.scala:189)
[info] at org.bitcoins.rpc.v18.BitcoindV18RpcClientTest.$anonfun$new$21(BitcoindV18RpcClientTest.scala:82)
[info] at scala.util.Success.$anonfun$map$1(Try.scala:255)
[info] at scala.util.Success.map(Try.scala:213)
[info] at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[info] at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[info] at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[info] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[info] at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
[info] at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92)
[info] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info] at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
[info] at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92)
[info] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
[info] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
[info] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[info] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[info] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[info] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[info] - should successfully submit a header
[info] ScalaTest
[info] Run completed in 5 minutes, 5 seconds.
[info] Total number of tests run: 7
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 2, failed 5, canceled 0, ignored 0, pending 0
[info] *** 5 TESTS FAILED ***
[error] Failed: Total 7, Failed 5, Errors 0, Passed 2
[error] Failed tests:
[error] org.bitcoins.rpc.v18.BitcoindV18RpcClientTest
[error] (Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 315 s, completed Aug 13, 2019 7:38:36 AM

[Christewart]

where is this from? can you link?

[cwaldron97]

I just ran analyzepsbt on it and got the information from my console to make sure we were parsing the correct numbers from the Json

[nkohen]

I think @Christewart is referring to the actual PSBT and where you got that from, like was it from a doc some place or did you generate it or?

[cwaldron97]

See the link at the top for BIP 174, that has all my test vectors included. I will post the description of it so it is easy to search.

[Christewart]

is this pulled from some where? Can you link?

[cwaldron97]

Link is at the top of the file. Do you think I should add the link to more places? All the test vectors I used are contained in that link. I'll change the link to hyperlink specifically to test vectors.

[Christewart]

#551

[nkohen]

should probably assert that it contains at least one more to make the test a little more robust

[cwaldron97]

The problem is that a joined PSBT doesn't necessarily contain both inside of it. It uses parts of each to create a brand new PSBT and its not exactly clear how I can pattern search.

[nkohen]

gotchya, in that case can you manually run this join and find out what is in it and assert on at least one thing that is contributed from each of the inputs?

[Christewart]

Looks like you still have tests failing https://travis-ci.org/bitcoin-s/bitcoin-s/jobs/577071987#L1213

[torkelrogstad]

All tests pass locally! 🎉

But you need to rebase. Probably a good idea to squash first, seeing as this is 56 commits. I can help you with the rebase process if you'd like, I think it's going to be a somewhat hairy process.

[nkohen]

I think @Christewart is referring to the actual PSBT and where you got that from, like was it from a doc some place or did you generate it or?

[nkohen]

gotchya, in that case can you manually run this join and find out what is in it and assert on at least one thing that is contributed from each of the inputs?