New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Display list of transactions for a given address #8

Closed
janoside opened this Issue Aug 25, 2017 · 32 comments

Comments

Projects
None yet
9 participants
@janoside
Copy link
Owner

janoside commented Aug 25, 2017

No description provided.

@askz

This comment has been minimized.

Copy link

askz commented May 9, 2018

Hi, any work in progress on this?

@janoside

This comment has been minimized.

Copy link
Owner Author

janoside commented May 9, 2018

Still researching the best way to go about implementing address pages. This issue was the first thought I had, but it may not be the best. Out of curiosity: do you run a node with addrindex?

@askz

This comment has been minimized.

Copy link

askz commented May 10, 2018

Not atm. But I can do it fast, I run some small blockchains for test purposes

@janoside janoside changed the title Add address support when interacting with a node with -addrindex Display list of transactions for a given address May 12, 2018

@TheHolyRoger

This comment has been minimized.

Copy link

TheHolyRoger commented May 23, 2018

Have you got any other ideas for bitcoind without addrindex?

@janoside

This comment has been minimized.

Copy link
Owner Author

janoside commented May 23, 2018

The idea floating in my head, but that I've been too busy to research is to interact with the node in a fashion similar to a lite wallet. I believe that an implementation in that vein would involve sending/receiving bloom-filtered P2P messages to the target node and building a list of transactions on the fly.

Again, since a design goal of this project is to be database-free, there will almost certainly be a tradeoff here: spending more time building lists of relevant transactions instead of storing such lists in a database.

Thoughts on this idea?

@askz

This comment has been minimized.

Copy link

askz commented May 24, 2018

Interesting idea you got here.
It would in fact take much longer for each new address, also a cache system would be appreciable.

I'll try in the next days to implement addrindex

@TheHolyRoger

This comment has been minimized.

Copy link

TheHolyRoger commented May 24, 2018

Considering addrindex is not officially implemented in Bitcoin Core or any of the Core node software for most altcoins, I think it would be best avoided.

I won't be running a node with addrindex on my network given the implications of keeping addrindex up to date with Core releases.

This explorer is great and I love that it's database free, but I think an optional database would be best to cache addresses.

Happy to lend a hand to help out with this!

If you go down the addrindex route I'll have to switch to Bitcoin ABE though!

@askz

This comment has been minimized.

Copy link

askz commented May 24, 2018

Maybe we can instead rely on a much lighter database like sqlite ?

@askz

This comment has been minimized.

Copy link

askz commented May 24, 2018

I am for the addrindex because, the coin and its asset chain I use (komodo, KMD) implemented addrindex, spentindex etc.

@janoside

This comment has been minimized.

Copy link
Owner Author

janoside commented May 24, 2018

Thanks for the thoughts/conversation. I agree with you @TheHolyRoger - the lack of addrindex support means I definitely don't want addrindex to be the solution to this problem. However, @askz, I'm leaning toward a flexible/configurable solution where addrindex could be used if desired.

@TheHolyRoger

This comment has been minimized.

Copy link

TheHolyRoger commented May 24, 2018

I think personally I'm going to attempt to scan addresses into a db with nedb, will let you know how i get on!

@volbil

This comment has been minimized.

Copy link

volbil commented May 25, 2018

Any luck @TheHolyRoger :) ?

@TheHolyRoger

This comment has been minimized.

Copy link

TheHolyRoger commented May 25, 2018

Not yet, I've used bitcoin-abe for now.

I started it, but not sure when I'll get round to doing more on it

@karel-3d

This comment has been minimized.

Copy link

karel-3d commented Jun 19, 2018

Look at indexd if you have time

https://github.com/bitcoinjs/indexd

@karel-3d

This comment has been minimized.

Copy link

karel-3d commented Jun 19, 2018

This is not possible without address index, sorry. The question is how to make it simple.

@karel-3d

This comment has been minimized.

Copy link

karel-3d commented Jun 20, 2018

For interest - I am from Trezor, we are having these issues too.

We used to run Insight/bitcore. insight/bitcore does an unholy thing and adds an address index INTO THE SOURCE CODE - so all insight instances actually run a modified version of bitcoin/altcoin. That is a mess to maintain.

We are now migrating to blockbook - https://github.com/trezor/blockbook - but I dont think it is the right solution for this project at all.

@chrisiaut

This comment has been minimized.

Copy link

chrisiaut commented Jul 23, 2018

@karel-3d the "easiest" way to keep track of all balances is to run a full node and using a .dat file parser to extract transactions and store them in a database. MongoDB or Redis would work for example.

To collect all new blocks and transactions going forward you can setup a webhook for new blocks

@karel-3d

This comment has been minimized.

Copy link

karel-3d commented Jul 25, 2018

@chrisiaut nope, you still need to deal with orphaned blocks so you don't have the tx twice for example

so in the end you will end up doing similar stuff as indexd, or insight/bitcore, or blockbook.

@chrisiaut

This comment has been minimized.

Copy link

chrisiaut commented Jul 25, 2018

@karel-3d sure you can use tools like fast-dat-parser to get the longest chain and then parse it.

@karel-3d

This comment has been minimized.

Copy link

karel-3d commented Jul 25, 2018

@frndxyz

This comment has been minimized.

Copy link

frndxyz commented Aug 13, 2018

@janoside but if you used bloom-filtered P2P messages old altcoin not support by explorer. because i see that old coin not support bip37.

@martinschwarz

This comment has been minimized.

Copy link

martinschwarz commented Sep 11, 2018

What about querying local or public Electrum servers (electrumx)? Many fullnode users already use electrum to connect Trezor to their fullnode.

@janoside

This comment has been minimized.

Copy link
Owner Author

janoside commented Sep 11, 2018

I like the ElectrumX server idea.

Thinking out loud...If I go in that direction, I'll want to minimize trust placed in those servers. I think a pretty good middle-ground could be achieved with:

  1. Only query txids for a given address from Electrum, then use the txids to get transaction data from the app's connected node.
  2. Support configuring multiple electrum servers and corroborating the txids-for-address results among them.

I still like the idea that the app could support a few methods of displaying address details, but I like the feel of this one as a first option that I'll try to tackle soon. Thanks for the suggestion.

@askz

This comment has been minimized.

Copy link

askz commented Sep 12, 2018

I like the idea too ! Don't know why we didn't thought about it before.

Maybe we can contribute to add HTTP(S)/WebSocket support to ElectrumX server, so the client will directly query the txids and send it to the explorer. So the explorer server doesn't get overloaded.
Or just implement a cache with invalidation etc.

What do you think?

EDIT:
We actually forked an electrum client for node, to add connection persistence : https://github.com/Monaize/node-electrum-client/tree/electrumx-persistence

janoside added a commit that referenced this issue Sep 21, 2018

Majority of the work on #8 to support querying N ElectrumX servers fo…
…r address balance and address txid history

janoside added a commit that referenced this issue Sep 21, 2018

More work on #8 - showing balances and transaction history for addresses
- in tx history, request/display all inputs to get correct gain/loss values
- cross referencing for txid history from electrum
- include genesis coinbase values for genesis coinbase output address (electrum ignores the genesis coinbase TX and +50 value, but for consistency with the rest of this tool they're included)
- banner describing the electrum trust model
- ui tweaks including showing gain/loss for each address tx history item
@janoside

This comment has been minimized.

Copy link
Owner Author

janoside commented Sep 21, 2018

All, please let me know what you think about the current implementation. As discussed, I went with trust-minimized ElectrumX querying as a first implementation. You can see the feature in action at the link below. It's currently using the "electrum-client" npm package for querying Electrum - I still need to switch to the forked, persistent-connection version @askz referred to (@askz Is that fork available via npm?)

https://btc.chaintools.io/address/3NPGpNyLLmVKCEcuipBs7G4KpQJoJXjDGe

Edit - After deploying to the demo site the electrum connections hung and I went ahead and switched to use the persistent-connection fork of electrum-client. We'll see how it fares on the demo site.

janoside added a commit that referenced this issue Sep 21, 2018

@louiseth1

This comment has been minimized.

Copy link

louiseth1 commented Sep 23, 2018

I saw you were working on this today and did see it live on your demo site.

Seems to work well and all the relevant information appeared. However, when I refresh from time to time I see sometimes the transaction informations are not displayed on the address page.

@janoside

This comment has been minimized.

Copy link
Owner Author

janoside commented Sep 23, 2018

@louiseth1 The sometimes-blank tx data is due to the socket connection to ElectrumX timing out and not reconnecting properly. I've been periodically restarting the demo site as I push new code and the tx data is visible for a while after each restart.

Once I get this socket issue "addressed", I'll be closing this issue...finally.

Thanks for the feedback.

@louiseth1

This comment has been minimized.

Copy link

louiseth1 commented Sep 23, 2018

Thanks for the explanation. I'll then wait until you fixed it to update on my end.

I did try to pull it but had issues building it.

@askz

This comment has been minimized.

Copy link

askz commented Sep 24, 2018

@janoside we didnt deployed it on npm because we wanted to wait the PR to be accepted, but the maintainer seems.... busy/dead.

@louiseth1

This comment has been minimized.

Copy link

louiseth1 commented Sep 26, 2018

I tried deploying fresh today. I had to downgrade my npm version to be able to install the electrum-client. (Seems to have installed properly.)

Only had it working using npm 3.10.10. Newer versions I tried wouldn't work.

Otherwise everything installed fine. Could run the npm build aswell, however having an issue on the address pages (nothing in the logs appeared though):

Failed to load address 1HsRTMvG6fjnqJtrqKQJFa9qxWtnYcFQfk (TypeError: Promise.all(...).catch(...).finally is not a function)

I then tried adding the electrum addresses but it seems I can't get the format right because it shows as undefined in the running log.

Log output shows some issues:
[root@explorer btc-rpc-explorer]# node bin/www Running RPC Explorer for Bitcoin Connecting via RPC to node at 72.10.166.164:8332 Connecting to ElectrumX Server: undefined:undefined OnError:Error: connect ECONNREFUSED 127.0.0.1 Error 137rg023xx7gerfwdd: Error: connect ECONNREFUSED 127.0.0.1, when trying to connect to ElectrumX server at undefined:undefined Error 120387rygxx231gwe40: Error: connect ECONNREFUSED 127.0.0.1 Error 31207ugf4e0fed: Error: connect ECONNREFUSED 127.0.0.1, while initializing ElectrumX API OnClose:undefined
Could you indicate how you would format the following array in config.js?
electrumXServers:[ "electrum.chainhost.io:50002", "electrum-server.ninja:50002" ],

@janoside

This comment has been minimized.

Copy link
Owner Author

janoside commented Sep 27, 2018

@louiseth1 Sorry about this. The whole thing feels like a bit of a mess at the moment and I'm trying to get back to it soon. This really should've been done in a feature branch too...hindsight. Anyway, I believe the code, as is, requires a newer version of node (10+) to deal with the Promise.all().catch().finally. I'd like to remove that requirement and intend to do so soon. Secondly, the format for electrumXServers would be in your example:

electrumXServers:[ {host:"electrum.chainhost.io", port:50002}, {host:"electrum-server.ninja", port:50002} ]

@louiseth1

This comment has been minimized.

Copy link

louiseth1 commented Sep 30, 2018

Got it, updated the servers config line and it worked just fine.

Got an issue because app couldnt find coinConfig.baseCurrencyUnit.multiplier. The value in the config file was CurrencyUnits, I added an array named baseCurrencyUnit with same values to fix the issue I've had.

Everything else worked fine aside from the Balance of the address which displays 0. Haven't seen any errors in console.log or the nodejs stdout. Tried reading the code but not that familiar with node so couldn't figure it out.

@janoside janoside closed this in 3915313 Oct 12, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment