Extract electrumx module #4427

haarts · 2018-06-11T11:07:43Z

This pull request further clarifies the network module API.

A couple of major things have happened here (non of which impacts functionality):

A class Triggers is defined and used as a mixin in the network module. This class deals with GUI notifications.
A class ElectrumX is defined and used instead of the Network class. This class holds (almost) all the previously defined ElectrumX specific API calls. The methods therein are just copied over from the Network class.
All of the helper methods in the Network class have been prefixed with an underscore as to communicate their private nature.

I would understand if these changes are considered controversial. Read with care.

kyuupichan · 2018-06-12T01:28:44Z

You seem to have lost some of the changes I recently made

haarts · 2018-06-12T07:31:07Z

I've identified 4 commits of your to which you are referring:

Number 1 is addressed in e289880 and 071f782.
Number 2 is rebased properly afaict.
Number 3 I indeed changed slightly and I might have missed something there. I have omitted the server version call in the scripts/block_headers as I believe that that message has already been sent in the initialization of the network class here: https://github.com/haarts/electrum/blob/dbea86f39da57fad4f7677b25635ffa94b445682/lib/network.py#L697
Number 4 I did miss passing True once here: https://github.com/spesmilo/electrum/pull/4427/files#diff-cb3587f39324c05e12dc545e531b2300R120. I will fix that.

I might very well be missing other commits or patches. I appreciate the feedback!

kyuupichan · 2018-06-12T17:23:03Z

Grepping for get_chunk still has hits. There are none in master.

Otherwise looks good

SomberNight · 2018-06-12T20:43:38Z

lib/verifier.py

-                else:
-                    if (tx_hash not in self.requested_merkle
-                            and tx_hash not in self.merkle_roots):
+            if tx_height == 0 or tx_height > local_height:


you should not let negative tx_heights through; there are some magic numbers in that range; see

electrum/lib/wallet.py

Lines 77 to 79 in 12c5474

TX_HEIGHT_LOCAL = -2

TX_HEIGHT_UNCONF_PARENT = -1

TX_HEIGHT_UNCONFIRMED = 0

Ah, that is surprising. Thanks for pointing that out. Will fix.

SomberNight · 2018-06-12T20:52:07Z

lib/synchronizer.py

@@ -120,7 +120,7 @@ def on_address_history(self, response):
        tx_fees = dict(filter(lambda x:x[1] is not None, tx_fees))
        # Note if the server hasn't been patched to sort the items properly
        if hist != sorted(hist, key=lambda x:x[1]):
-            self.network.interface.print_error("serving improperly sorted address histories")
+            self.print_error("error: serving improperly sorted address histories")


not that it's criticial but the difference here is that if you called self.network.interface.print_error, it would actually print which server is doing it

Yeah... I know what I did there. The reason I did this was so I could get rid of the last remaining 'interface' reference. I deemed the upside bigger than the downside. I'll leave the code as is if that is OK with you.

SomberNight · 2018-06-12T21:15:27Z

lib/network.py

@@ -525,7 +525,7 @@ def _add_recent_server(self, server):

    def _process_response(self, interface, response, callbacks):
        if self.debug:
-            self.print_error("<--", response)
+            self.print_error(interface.host, "<--", response)


did you test this? I think it's redundant, it happens already.
see

electrum/lib/util.py

Line 172 in 12c5474

print_error("[%s]" % self.diagnostic_name(), *msg)

haarts · 2018-06-13T07:25:04Z

I left the get_chunk call in the electrumX module with a deprecation comment. But I remove it, that indeed is better.

haarts · 2018-06-27T07:17:33Z

What needs to be done to get this branch merged? (aside from resolving the conflicts)

ecdsa · 2018-06-27T13:21:18Z

@haarts I did not have time to look at it.
note that I also plan to update the network code to use asyncio, so I will look at it at that time.

haarts · 2018-06-27T14:10:08Z

This PR can be seen as a prelude to converting the network code to use asyncio.
As a matter of fact, the code I'm currently work on is yet an other step into that direction. I have kept the network module a thread but only to contain the changes to the network module (otherwise much more intensive refactoring is required). This again is a good step in the direction of making it asyncio compatible.

It would help me a lot if this PR got merged, otherwise my works diverges more and more from master. You can follow my progress on my own branch: https://github.com/haarts/electrum/tree/add-pylibbitcoin

SomberNight · 2018-06-27T14:21:07Z

Given you are subclassing Network to do e-x specific stuff, I am guessing you won't allow libbitcoin/e-x heterogeneous networks. What is your plan for libbitcoin? Will the client connect to multiple libbitcoin instances like with e-x currently, or will it only connect to one which we assume to be trusted?

haarts · 2018-06-27T14:43:04Z

Yes, you are correct. I currently do not allow for hetrogeneous networks. Doing the sub classing as implemented however makes it easy to switch implementations. Either network = ElectrumX() or network = Libbitcoin(). Both implementations have the same interface so the Electrum code can use it as a drop in replacement. That's why spent so much time reducing the API of the Network class. This makes the implementation of the Libbitcoin class doable.

The Libbitcoin implementation supports multiple backends and multiple chains.

SomberNight · 2018-06-27T14:47:53Z

I guess you probably already thought this through but is it easier to modify the client to support both backends (e-x and libbitcoin) than, say having some middleware between the electrum client and libbitcoin translate the electrum protocol calls into libbitcoin rpc (or similar)?

SomberNight · 2018-06-27T15:48:12Z

lib/websockets.py

-        return addr, amount
+        scripthash = bitcoin.address_to_scripthash(addr)
+        hash2address[scripthash] = address
+        return scripthash, amount


bitcoin is not imported
instead of hash2address, you want self.hash2address
addr instead of address

I'm a bit disappointed in myself I missed this. Odd. And fixed.

haarts · 2018-06-28T08:21:52Z

As to your middleware question; the thought occurred to me.
And in a way this is what is happening in my next PR, see protocol.py on my development branch.
I feel that which ever road I would have taken this would still have lead to the changes I've done up until this point. All I've done so far is refactor. All with a dual goal; understanding the code base and clearly defining an API. As a pleasant side effect I'm writing tests and gently polish the code somewhat.
Does this make sense?

SomberNight · 2018-06-28T08:44:58Z

lib/websockets.py

@@ -34,6 +34,7 @@
    sys.exit("install SimpleWebSocketServer")

 from . import util
+from lib import bitcoin


use a relative import; especially as the package structure is different when running from source and when having the lib installed
( #4279 )

I did not know that. Will fix.

haarts · 2018-07-02T07:16:22Z

I've rebased this PR again.

Would it perhaps help if I split this PR up in pieces? It has grown quite a bit.

This is again a small step in the right direction. The network module is going to accumulate more and more of these simple methods. Once everything is moved into that module, that module is going to be split. Note that I've left the scripts which use scripts/util.py alone. I suspect the same functionality can be reached when using just lib/network.py and that scripts/util.py is obsolete.

This makes the method easier to read.

This class is a subclass of Network and doesn't do anything different at the moment. The idea is to move ElectrumX specifics to this new class.

Much remains to be done but the demarcation is now clear. To do is moving ElectrumX specific attributes to the electrumx module and making the network configurable somehow.

This was quite a serious bug.

It is starting to be hard to keep track of all the loose ends.

The main motivation of doing so is that the websockets module now no longer needs to access the h2addr field on the network module. Removing the deprecated call was a nice bonus.

The backing ElectrumX call is deprecated in favor of blockchain.scripthash.subscribe.

This was fixed earlier but that fix was lost in an aggresive rebase.

Initially I thought that the caller should be responsible for maintaining the mapping between address and scripthash. That made sense b/c callers accessed the mapping in the network module directly. And the dictionary mainting the mapping was accessed by different callers on potentially different threads. Turns out that maintaining that mapping on the caller site involves quite a bit of work. Besides using address instead of scripthash makes sense from the users perspective. The user never thinks in terms of scripthash. The concurrent access is not a huge deal since we're only adding items and reading singular values.

The class is extracted from the Network class. It's a Seperation of Concerns thing. In the end this will make it easier to change the network module. Note that the lock used here is now specific to the callbacks, it used to be shared with the lock used in sending API calls. I think this is clearer.

Doing so clarifies which methods can be used by others and which are strictly for internal use. I've choosen to use a weaker form of signaling visibility by using only one leading underscore b/c I do not want to deviate from the norm set in this project too much. This diff also introduces the 'server_version' API call.

This way the method call does not wait for a response. It's a little odd that the codebase is using two different ways of dealing with responses. One is with callbacks or blocking and the other is a fire and forget mode in which the responses are later tide to callbacks via message IDs.

I've assumed that having a blockchain (which lives on an interface) implied the network is connected. This is not always the case, therefor we need this additional check.

The new name reflects better what it does.

A transaction height can be negative too, see the wallet modules TX_HEIGHT_LOCAL and TX_HEIGHT_UNCONF_PARENT constants.

haarts · 2018-07-11T19:02:40Z

I can still split it up if you guys find that easier. I'll be sure to do so in the future at least. :)

haarts · 2018-07-12T07:08:05Z

I'm closing this one in favor of smaller PR's.

SomberNight reviewed Jun 12, 2018

View reviewed changes

haarts force-pushed the extract-electrumx-module branch from 25f3d08 to 569826c Compare June 27, 2018 15:12

SomberNight reviewed Jun 27, 2018

View reviewed changes

SomberNight reviewed Jun 28, 2018

View reviewed changes

haarts force-pushed the extract-electrumx-module branch from 16a356a to 73d7445 Compare July 2, 2018 07:15

toxeus mentioned this pull request Jul 6, 2018

rename plugins package to electrum_plugins #4510

Closed

haarts added 12 commits July 11, 2018 16:03

Reduce indentation level

79b7db3

This makes the method easier to read.

Minor linter nits

4623b2c

Add ElectrumX class and reference it

c02911a

This class is a subclass of Network and doesn't do anything different at the moment. The idea is to move ElectrumX specifics to this new class.

Move the majority of the ElectrumX code

e5e7858

Much remains to be done but the demarcation is now clear. To do is moving ElectrumX specific attributes to the electrumx module and making the network configurable somehow.

Pass list of tuples instead of bare tuple

7bb5a1d

This was quite a serious bug.

Add some TODO's

fab126d

It is starting to be hard to keep track of all the loose ends.

Remove deprecated subscribe_to_addresses call

3b733e3

The main motivation of doing so is that the websockets module now no longer needs to access the h2addr field on the network module. Removing the deprecated call was a nice bonus.

Remove subscribe_to_address

5a2404e

The backing ElectrumX call is deprecated in favor of blockchain.scripthash.subscribe.

Return early when callback is not None

151bf0e

This was fixed earlier but that fix was lost in an aggresive rebase.

Remove left over variable

7139ded

haarts added 15 commits July 11, 2018 16:03

Make map_scripthash_to_address private

26de436

Rename 'r' to 'response'

31695e2

Remove mentions interface in non network modules

733cb83

Remove unused is_up_to_date network method

64fae80

Clarify method name, add explaination

735555c

Improve logging

5d0047e

Add connection check

2cf5fe8

I've assumed that having a blockchain (which lives on an interface) implied the network is connected. This is not always the case, therefor we need this additional check.

Rename method

3834e1b

The new name reflects better what it does.

Add lost True parameter

4130e2d

Skip negative tx heights too

1c9f719

A transaction height can be negative too, see the wallet modules TX_HEIGHT_LOCAL and TX_HEIGHT_UNCONF_PARENT constants.

Remove deprecated 'get_chunck' API call

4220d3d

Fix missing import and wrong RHS

ccd6819

haarts force-pushed the extract-electrumx-module branch from 73d7445 to ccd6819 Compare July 11, 2018 14:06

haarts closed this Jul 12, 2018

This was referenced Jul 12, 2018

Preliminaries #4541

Merged

Add electrumx class #4542

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extract electrumx module #4427

Extract electrumx module #4427

haarts commented Jun 11, 2018

kyuupichan commented Jun 12, 2018

haarts commented Jun 12, 2018

kyuupichan commented Jun 12, 2018

SomberNight Jun 12, 2018

haarts Jun 13, 2018

SomberNight Jun 12, 2018

haarts Jun 13, 2018 •

edited

Loading

SomberNight Jun 12, 2018

haarts commented Jun 13, 2018

haarts commented Jun 27, 2018 •

edited

Loading

ecdsa commented Jun 27, 2018

haarts commented Jun 27, 2018

SomberNight commented Jun 27, 2018

haarts commented Jun 27, 2018

SomberNight commented Jun 27, 2018

SomberNight Jun 27, 2018

haarts Jun 28, 2018

haarts commented Jun 28, 2018

SomberNight Jun 28, 2018

haarts Jun 29, 2018

haarts commented Jul 2, 2018

haarts commented Jul 11, 2018

haarts commented Jul 12, 2018

	TX_HEIGHT_LOCAL = -2
	TX_HEIGHT_UNCONF_PARENT = -1
	TX_HEIGHT_UNCONFIRMED = 0

Extract electrumx module #4427

Extract electrumx module #4427

Conversation

haarts commented Jun 11, 2018

kyuupichan commented Jun 12, 2018

haarts commented Jun 12, 2018

kyuupichan commented Jun 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haarts Jun 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haarts commented Jun 13, 2018

haarts commented Jun 27, 2018 • edited Loading

ecdsa commented Jun 27, 2018

haarts commented Jun 27, 2018

SomberNight commented Jun 27, 2018

haarts commented Jun 27, 2018

SomberNight commented Jun 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haarts commented Jun 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haarts commented Jul 2, 2018

haarts commented Jul 11, 2018

haarts commented Jul 12, 2018

haarts Jun 13, 2018 •

edited

Loading

haarts commented Jun 27, 2018 •

edited

Loading