Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Display list of transactions for a given address #8

Closed
janoside opened this issue Aug 25, 2017 · 32 comments
Closed

Display list of transactions for a given address #8

janoside opened this issue Aug 25, 2017 · 32 comments

Comments

@janoside
Copy link
Owner

@janoside janoside commented Aug 25, 2017

No description provided.

@askz
Copy link

@askz askz commented May 9, 2018

Hi, any work in progress on this?

@janoside
Copy link
Owner Author

@janoside janoside commented May 9, 2018

Still researching the best way to go about implementing address pages. This issue was the first thought I had, but it may not be the best. Out of curiosity: do you run a node with addrindex?

@askz
Copy link

@askz askz commented May 10, 2018

Not atm. But I can do it fast, I run some small blockchains for test purposes

@janoside janoside changed the title Add address support when interacting with a node with -addrindex Display list of transactions for a given address May 12, 2018
@TheHolyRoger
Copy link

@TheHolyRoger TheHolyRoger commented May 23, 2018

Have you got any other ideas for bitcoind without addrindex?

@janoside
Copy link
Owner Author

@janoside janoside commented May 23, 2018

The idea floating in my head, but that I've been too busy to research is to interact with the node in a fashion similar to a lite wallet. I believe that an implementation in that vein would involve sending/receiving bloom-filtered P2P messages to the target node and building a list of transactions on the fly.

Again, since a design goal of this project is to be database-free, there will almost certainly be a tradeoff here: spending more time building lists of relevant transactions instead of storing such lists in a database.

Thoughts on this idea?

@askz
Copy link

@askz askz commented May 24, 2018

Interesting idea you got here.
It would in fact take much longer for each new address, also a cache system would be appreciable.

I'll try in the next days to implement addrindex

@TheHolyRoger
Copy link

@TheHolyRoger TheHolyRoger commented May 24, 2018

Considering addrindex is not officially implemented in Bitcoin Core or any of the Core node software for most altcoins, I think it would be best avoided.

I won't be running a node with addrindex on my network given the implications of keeping addrindex up to date with Core releases.

This explorer is great and I love that it's database free, but I think an optional database would be best to cache addresses.

Happy to lend a hand to help out with this!

If you go down the addrindex route I'll have to switch to Bitcoin ABE though!

@askz
Copy link

@askz askz commented May 24, 2018

Maybe we can instead rely on a much lighter database like sqlite ?

@askz
Copy link

@askz askz commented May 24, 2018

I am for the addrindex because, the coin and its asset chain I use (komodo, KMD) implemented addrindex, spentindex etc.

@janoside
Copy link
Owner Author

@janoside janoside commented May 24, 2018

Thanks for the thoughts/conversation. I agree with you @TheHolyRoger - the lack of addrindex support means I definitely don't want addrindex to be the solution to this problem. However, @askz, I'm leaning toward a flexible/configurable solution where addrindex could be used if desired.

@TheHolyRoger
Copy link

@TheHolyRoger TheHolyRoger commented May 24, 2018

I think personally I'm going to attempt to scan addresses into a db with nedb, will let you know how i get on!

@volbil
Copy link

@volbil volbil commented May 25, 2018

Any luck @TheHolyRoger :) ?

@TheHolyRoger
Copy link

@TheHolyRoger TheHolyRoger commented May 25, 2018

Not yet, I've used bitcoin-abe for now.

I started it, but not sure when I'll get round to doing more on it

@karelbilek
Copy link

@karelbilek karelbilek commented Jun 19, 2018

Look at indexd if you have time

https://github.com/bitcoinjs/indexd

@karelbilek
Copy link

@karelbilek karelbilek commented Jun 19, 2018

This is not possible without address index, sorry. The question is how to make it simple.

@karelbilek
Copy link

@karelbilek karelbilek commented Jun 20, 2018

For interest - I am from Trezor, we are having these issues too.

We used to run Insight/bitcore. insight/bitcore does an unholy thing and adds an address index INTO THE SOURCE CODE - so all insight instances actually run a modified version of bitcoin/altcoin. That is a mess to maintain.

We are now migrating to blockbook - https://github.com/trezor/blockbook - but I dont think it is the right solution for this project at all.

@geek-at
Copy link

@geek-at geek-at commented Jul 23, 2018

@karel-3d the "easiest" way to keep track of all balances is to run a full node and using a .dat file parser to extract transactions and store them in a database. MongoDB or Redis would work for example.

To collect all new blocks and transactions going forward you can setup a webhook for new blocks

@karelbilek
Copy link

@karelbilek karelbilek commented Jul 25, 2018

@chrisiaut nope, you still need to deal with orphaned blocks so you don't have the tx twice for example

so in the end you will end up doing similar stuff as indexd, or insight/bitcore, or blockbook.

@geek-at
Copy link

@geek-at geek-at commented Jul 25, 2018

@karel-3d sure you can use tools like fast-dat-parser to get the longest chain and then parse it.

@karelbilek
Copy link

@karelbilek karelbilek commented Jul 25, 2018

@frndxyz
Copy link

@frndxyz frndxyz commented Aug 13, 2018

@janoside but if you used bloom-filtered P2P messages old altcoin not support by explorer. because i see that old coin not support bip37.

@martinschwarz
Copy link

@martinschwarz martinschwarz commented Sep 11, 2018

What about querying local or public Electrum servers (electrumx)? Many fullnode users already use electrum to connect Trezor to their fullnode.

@janoside
Copy link
Owner Author

@janoside janoside commented Sep 11, 2018

I like the ElectrumX server idea.

Thinking out loud...If I go in that direction, I'll want to minimize trust placed in those servers. I think a pretty good middle-ground could be achieved with:

  1. Only query txids for a given address from Electrum, then use the txids to get transaction data from the app's connected node.
  2. Support configuring multiple electrum servers and corroborating the txids-for-address results among them.

I still like the idea that the app could support a few methods of displaying address details, but I like the feel of this one as a first option that I'll try to tackle soon. Thanks for the suggestion.

@askz
Copy link

@askz askz commented Sep 12, 2018

I like the idea too ! Don't know why we didn't thought about it before.

Maybe we can contribute to add HTTP(S)/WebSocket support to ElectrumX server, so the client will directly query the txids and send it to the explorer. So the explorer server doesn't get overloaded.
Or just implement a cache with invalidation etc.

What do you think?

EDIT:
We actually forked an electrum client for node, to add connection persistence : https://github.com/Monaize/node-electrum-client/tree/electrumx-persistence

janoside added a commit that referenced this issue Sep 21, 2018
…r address balance and address txid history
janoside added a commit that referenced this issue Sep 21, 2018
- in tx history, request/display all inputs to get correct gain/loss values
- cross referencing for txid history from electrum
- include genesis coinbase values for genesis coinbase output address (electrum ignores the genesis coinbase TX and +50 value, but for consistency with the rest of this tool they're included)
- banner describing the electrum trust model
- ui tweaks including showing gain/loss for each address tx history item
@janoside
Copy link
Owner Author

@janoside janoside commented Sep 21, 2018

All, please let me know what you think about the current implementation. As discussed, I went with trust-minimized ElectrumX querying as a first implementation. You can see the feature in action at the link below. It's currently using the "electrum-client" npm package for querying Electrum - I still need to switch to the forked, persistent-connection version @askz referred to (@askz Is that fork available via npm?)

https://btc.chaintools.io/address/3NPGpNyLLmVKCEcuipBs7G4KpQJoJXjDGe

Edit - After deploying to the demo site the electrum connections hung and I went ahead and switched to use the persistent-connection fork of electrum-client. We'll see how it fares on the demo site.

janoside added a commit that referenced this issue Sep 21, 2018
@louiseth1
Copy link

@louiseth1 louiseth1 commented Sep 23, 2018

I saw you were working on this today and did see it live on your demo site.

Seems to work well and all the relevant information appeared. However, when I refresh from time to time I see sometimes the transaction informations are not displayed on the address page.

@janoside
Copy link
Owner Author

@janoside janoside commented Sep 23, 2018

@louiseth1 The sometimes-blank tx data is due to the socket connection to ElectrumX timing out and not reconnecting properly. I've been periodically restarting the demo site as I push new code and the tx data is visible for a while after each restart.

Once I get this socket issue "addressed", I'll be closing this issue...finally.

Thanks for the feedback.

@louiseth1
Copy link

@louiseth1 louiseth1 commented Sep 23, 2018

Thanks for the explanation. I'll then wait until you fixed it to update on my end.

I did try to pull it but had issues building it.

@askz
Copy link

@askz askz commented Sep 24, 2018

@janoside we didnt deployed it on npm because we wanted to wait the PR to be accepted, but the maintainer seems.... busy/dead.

@louiseth1
Copy link

@louiseth1 louiseth1 commented Sep 26, 2018

I tried deploying fresh today. I had to downgrade my npm version to be able to install the electrum-client. (Seems to have installed properly.)

Only had it working using npm 3.10.10. Newer versions I tried wouldn't work.

Otherwise everything installed fine. Could run the npm build aswell, however having an issue on the address pages (nothing in the logs appeared though):

Failed to load address 1HsRTMvG6fjnqJtrqKQJFa9qxWtnYcFQfk (TypeError: Promise.all(...).catch(...).finally is not a function)

I then tried adding the electrum addresses but it seems I can't get the format right because it shows as undefined in the running log.

Log output shows some issues:
[root@explorer btc-rpc-explorer]# node bin/www Running RPC Explorer for Bitcoin Connecting via RPC to node at 72.10.166.164:8332 Connecting to ElectrumX Server: undefined:undefined OnError:Error: connect ECONNREFUSED 127.0.0.1 Error 137rg023xx7gerfwdd: Error: connect ECONNREFUSED 127.0.0.1, when trying to connect to ElectrumX server at undefined:undefined Error 120387rygxx231gwe40: Error: connect ECONNREFUSED 127.0.0.1 Error 31207ugf4e0fed: Error: connect ECONNREFUSED 127.0.0.1, while initializing ElectrumX API OnClose:undefined
Could you indicate how you would format the following array in config.js?
electrumXServers:[ "electrum.chainhost.io:50002", "electrum-server.ninja:50002" ],

@janoside
Copy link
Owner Author

@janoside janoside commented Sep 27, 2018

@louiseth1 Sorry about this. The whole thing feels like a bit of a mess at the moment and I'm trying to get back to it soon. This really should've been done in a feature branch too...hindsight. Anyway, I believe the code, as is, requires a newer version of node (10+) to deal with the Promise.all().catch().finally. I'd like to remove that requirement and intend to do so soon. Secondly, the format for electrumXServers would be in your example:

electrumXServers:[ {host:"electrum.chainhost.io", port:50002}, {host:"electrum-server.ninja", port:50002} ]

@louiseth1
Copy link

@louiseth1 louiseth1 commented Sep 30, 2018

Got it, updated the servers config line and it worked just fine.

Got an issue because app couldnt find coinConfig.baseCurrencyUnit.multiplier. The value in the config file was CurrencyUnits, I added an array named baseCurrencyUnit with same values to fix the issue I've had.

Everything else worked fine aside from the Balance of the address which displays 0. Haven't seen any errors in console.log or the nodejs stdout. Tried reading the code but not that familiar with node so couldn't figure it out.

@janoside janoside closed this in 3915313 Oct 12, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants