Asyncio based on develop #261

mcelrath · 2014-08-26T15:00:45Z

No description provided.

adamkrellenstein · 2014-08-26T15:21:47Z

Do you have any estimates on what the optimal value for Bitcoin Core's rpcthreads parameter is? Is it much higher than 4, say, for the initial DB synchronization?

mcelrath · 2014-08-26T15:28:06Z

I suspect there's a bug in bitcoin core. I didn't see it use more than 4
threads even when I increased rpcthreads.

As for optimal, that comes down to disk speed, really. I think I was
substantially slowed down because my bitcoin database is on a spinning-rust
style drive (So bitcoind is I/O bound regardless of the number of threads). If
you have an SSD I suspect a higher value of rpcthreads would make it even
faster, since its less likely to be I/O bound and can then use multiple cores...

robby-d · 2014-08-26T15:59:34Z

in API.rst, you are committing a conflict (thus the <<< >>> symbols)... can you add a commit to this to clean it up?

robby-d · 2014-08-26T16:00:06Z

actually, looking at the commits, that is MY fault. Jeez! I can get that fixed once we merge this in

mcelrath · 2014-08-26T16:34:49Z

Aaah that explains why I had so many problems with that file. ;-) I kept trying to remove the merge conflict...

robby-d · 2014-08-26T16:37:41Z

yeah, looks like a merge gone bad on my part...if you can fix it, feel free to do so (the right version is the signature without the lock= parameter)

mcelrath · 2014-08-26T16:41:08Z

Will do. I'm not entirely sure how this happened, and it might not be your
fault. I did lots of gymnastics trying to rebase on develop.

adamkrellenstein · 2014-08-28T18:20:13Z

lib/bitcoin.py

@@ -384,9 +385,10 @@ def sort_unspent_txouts(unspent, allow_unconfirmed_inputs):
    return unspent

 def private_key_to_public_key (private_key_wif):
-    # allowable_wif_prefixes = [
+    allowable_wif_prefixes = [ b'\x80', b'\xef' ] 


Still need to fix this---these values should be set in config.py.

adamkrellenstein · 2014-08-28T18:21:49Z

Can you rename aiorun()? Let's merge this in and do some more testing.

mcelrath · 2014-08-29T13:54:34Z

Sure, how about {{{aio_wait_for()}}}?

adamkrellenstein · 2014-08-29T14:24:04Z

I guess I'd prefer something like synch_run(), but of course it doesn't matter much. 'wait for' sounds like it wants a time argument.

mcelrath · 2014-08-29T14:27:40Z

It does take an (optional) time argument. ;-)

I just wanted the name to somehow refer to the fact that it has to do with aiohttp and asyncio. It's your code, if you really want another name, tell me what to use.

adamkrellenstein · 2014-08-29T14:29:10Z

No, sure, that makes sense. How about aio_run_synch()?

mcelrath · 2014-08-29T14:31:50Z

Done.

adamkrellenstein · 2014-08-29T14:54:10Z

See #235.

The only thing left is allowable_wif_prefixes, I think.

mcelrath · 2014-08-29T14:55:49Z

I'm not sure what you have in mind there...

adamkrellenstein · 2014-08-29T14:57:47Z

Just instead of hardcoding those two bytes in bitcoin.py, use config.ADDRESSVERSION_MAINNET, etc..

adamkrellenstein · 2014-08-29T15:01:31Z

Rather:

if config.TESTNET:
    allowable_wif_prefixes = [config.PRIVATEKEY_VERSION_TESTNET,]
else:
    allowable_wif_prefixes = [config.PRIVATEKEY_VERSION_MAINNET,]

mcelrath · 2014-08-29T15:08:53Z

Done. Note that this is due to an interface change in pycoin, and you'll have to upgrade it. (I didn't want to have to include this patch in my asyncio PR, but I don't know what version of pycoin you guys are running and I had to do this to get it to run)

btcdrak · 2014-08-29T15:10:46Z

It's safe to upgrade to pycoin 0.50 btw.

adamkrellenstein · 2014-08-29T20:49:10Z

pip-requirements.txt

@@ -18,7 +18,7 @@ pytest==2.5.1

 pycoin==0.26


Is this line correct, then?

May want >= instead... For the windows build... Dont know if it will matter

Bad idea. If they later push a breaking change it would wreak havok.

What should the value be? 0.50?

We are using 0.50 with no problems in clearinghoused that was released 3 weeks ago. Looks like he's upgraded to 0.51 but there is no 0.51 tag so I cant tell if the changes are OK. 0.50 is definitely fine.

0.51 has been tagged, it's safe to use.

adamkrellenstein · 2014-09-03T17:20:40Z

@mcelrath, bump

mcelrath · 2014-09-03T17:25:38Z

@PhantomPhreak what do you need?

adamkrellenstein · 2014-09-03T17:30:18Z

@mcelrath, can you fix the pycoin version to 0.51?

mcelrath · 2014-09-04T13:29:58Z

Sure thing, pushed.

Asyncio based on develop

adamkrellenstein · 2014-09-11T01:01:59Z

@mcelrath, I'm having some trouble getting aiohttp to work with SSL, when that is enabled for Bitcoin Core. The only workaround I can figure out is passing connector=aiohttp.TCPConnector(force_close=True, verify_ssl=False) to aiohttp.request() in bitcoin.connect(). The error is: ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)

mcelrath · 2014-09-11T01:27:24Z

I recall having this problem, and fixing it (on a different project). Can you try aiohttp 0.9.1dev and see if it is already fixed there?

FWIW, I would not expose bitcoind to the external world if I were you. It's better to firewall it so it only accepts connections from localhost, in which case, SSL connections are just wasted CPU.

adamkrellenstein · 2014-09-11T01:33:08Z

Also, this message pops up sometimes randomly: poll 5.000 took 2.531 seconds

mcelrath · 2014-09-11T01:41:06Z

I believe the poll message was fixed by Python 3.4.1.

adamkrellenstein · 2014-09-11T01:45:32Z

I tried using SSL with the latest from the aiohttp GitHub repo. No dice.

I agree with you re: SSL, but the feature is there in Bitcoin Core, and some people (most notably exchanges) want it badly.

I'm running Python 3.4.1, and I still see the poll message.

mcelrath · 2014-09-11T01:55:01Z

The "Attempt to decode JSON..." message is from aiohttp. So the fix is to file a PR against aiohttp removing the message, or get web servers to report the correct mine-type. ;-)

adamkrellenstein · 2014-09-11T01:58:48Z

We worked around the JSON problem by just parsing the JSON externally.

Bob McElrath and others added 12 commits August 26, 2014 10:43

Update wif prefixes for changes in pycoin interface

9d7d429

Initial asyncio implementation for talking to bitcoind

4d95ec0

yield from in all the right places

eb1eb18

updated API docs

c415180

more doc tweaks with lock param for create_issuance

6cae8a5

Missed a couple rpc calls and wip fix

de1fe52

aiohttp instead of requests, with aiorun() to emulate synchronicity

e5e5878

Update requirements requests->aiohttp, python 3.4

f328aa6

Yield from functions in bitcoin.py

c8afa25

Tests pass (excluding test_log and test_json_rpc)

89371c0

Use aiorun()

015410a

Print block timing information

c7483fe

mcelrath mentioned this pull request Aug 26, 2014

Asyncio #260

Closed

Fix merge conflict

d3507a9

adamkrellenstein reviewed Aug 28, 2014
View reviewed changes

Rename aiorun to aio_wait_for

2217102

Rename aio_wait_for to aio_run_synch

c405127

Put wif prefixes in config

adc3300

adamkrellenstein reviewed Aug 29, 2014
View reviewed changes

adamkrellenstein mentioned this pull request Sep 3, 2014

Update wif prefixes for changes in pycoin interface #255

Closed

Update pycoin version to 0.51

27406df

adamkrellenstein pushed a commit that referenced this pull request Sep 4, 2014

Merge pull request #261 from mcelrath/asyncio

a3c9236

Asyncio based on develop

adamkrellenstein merged commit a3c9236 into CounterpartyXCP:develop Sep 4, 2014

mcelrath deleted the asyncio branch September 11, 2014 01:27

This was referenced Sep 12, 2014

asyncio upgrades on develop break SSL support for Bitcoin Core #284

Closed

parallelise population of tables blocks and transactions #128

Closed

Asyncio based on develop #261

Asyncio based on develop #261

Conversation

mcelrath commented Aug 26, 2014

adamkrellenstein commented Aug 26, 2014

mcelrath commented Aug 26, 2014

robby-d commented Aug 26, 2014

robby-d commented Aug 26, 2014

mcelrath commented Aug 26, 2014

robby-d commented Aug 26, 2014

mcelrath commented Aug 26, 2014

adamkrellenstein Aug 28, 2014

Choose a reason for hiding this comment

adamkrellenstein commented Aug 28, 2014

mcelrath commented Aug 29, 2014

adamkrellenstein commented Aug 29, 2014

mcelrath commented Aug 29, 2014

adamkrellenstein commented Aug 29, 2014

mcelrath commented Aug 29, 2014

adamkrellenstein commented Aug 29, 2014

mcelrath commented Aug 29, 2014

adamkrellenstein commented Aug 29, 2014

adamkrellenstein commented Aug 29, 2014

mcelrath commented Aug 29, 2014

btcdrak commented Aug 29, 2014

adamkrellenstein Aug 29, 2014

Choose a reason for hiding this comment

robby-d Aug 29, 2014

Choose a reason for hiding this comment

btcdrak Aug 29, 2014

Choose a reason for hiding this comment

adamkrellenstein Aug 30, 2014

Choose a reason for hiding this comment

btcdrak Aug 30, 2014

Choose a reason for hiding this comment

btcdrak Aug 30, 2014

Choose a reason for hiding this comment

adamkrellenstein commented Sep 3, 2014

mcelrath commented Sep 3, 2014

adamkrellenstein commented Sep 3, 2014

mcelrath commented Sep 4, 2014

adamkrellenstein commented Sep 11, 2014

mcelrath commented Sep 11, 2014

adamkrellenstein commented Sep 11, 2014

mcelrath commented Sep 11, 2014

adamkrellenstein commented Sep 11, 2014

mcelrath commented Sep 11, 2014

adamkrellenstein commented Sep 11, 2014