transactions of an address - example code please

[altsheets]

Please publish working example code for at least one of these questions:

• all transactions TO the address 0xe99356bde974bbe08721d77712168fa070aa8da4
• all transactions FROM the address 0xe99356bde974bbe08721d77712168fa070aa8da4
• the past 10 transactions TO address 0xe99356bde974bbe08721d77712168fa070aa8da4
• the past 11-20 transactions TO address 0xe99356bde974bbe08721d77712168fa070aa8da4
• the past 21-30 transactions TO address 0xe99356bde974bbe08721d77712168fa070aa8da4
• within the blocks 700,000 to 800,000, show all transactions TO the address 0xe99356bde974bbe08721d77712168fa070aa8da4
• within all blocks until the newest, show all transactions TO the address 0xe99356bde974bbe08721d77712168fa070aa8da4

For 'transactions' I need an array of transaction ids
e.g. [0xdf4c18485def00e5a61ea496d66ed0a1880526c291b45e76f91551c6ce0de6ac, 0x1a488b0d71c193d79872a7e0cee43484c3e1c7d1bc794931818cebc8c09880a9, ...].

If you have working code for ONE of those questions, it will probably be enough to extrapolate to most of the others ones. Please share your code, the wiki is explicit only about pending transactions.

Thanks a lot!

Happy New Year :-)

[obscuren]

It isn't easy and will be time consuming.

var n = eth.blocknumber;

var txs = [];
for(var i = 0; i < n; i++) {
    var block = eth.getBlock(i, true);
    for(var j = 0; j < block.transactions; j++) {
        if( block.transactions[j].to == the_address )
            txs.push(block.transactions[j]);
    }
}

[altsheets]

Related to #1749 and #1897 .

[fjl]

You have already asked for example code in #1897.

As much as everyone would like to see support for auto-indexing of raw value transfer transactions,
the answer you'll always receive is that the way to satisfy such queries is to make your account a wallet contract that emits logs for the things you care about.

[fjl]

Did you try the contract option, and if so, what prevents you from using it?

[altsheets]

for(var i = 0; i < eth.blocknumber; i++) {

ouch.

is to make your account a wallet contract that emits logs for the things you care about.

Thanks. Please publish example code for that.

That is faster than traversing 800000 blocks?

[altsheets]

You have already asked for example code in #1897.

yes - and no one answered there. That made me ask here. You see? :-)

Is there perhaps a working way to use the web3.eth.filter function to achieve the same? Would that be faster than the brute-force-checking-all-blocks-ever-myself suggestion of obscuren (thanks!) ?

[altsheets]

what prevents you from using it?

Can it be applied to any address?

i.e.
what if I want to see all transactions going into, or coming from the address of a friend?

[obscuren]

@altsheets use something like this

contract T {
    event NewTx(address indexed from, address indexed to, uint256 value)
    function transact(address to) {
        to.value(msg.value);
        NewTx(msg.from, to, msg.value);
    }
}

var abi = /* abi code */
var vContract = web3.eth.contract(abi);

var wallet = vContract.at(contractAddr);

wallet.NewTx({to: me}, function(error, log) {
});

[obscuren]

Please note this is an issue tracker for go-ethereum. This isn't your average go-to channel for asking help (though I'm happy to provide it). Please find us on gitter :-)

[obscuren]

Gitter also has a bigger user base so you'll likely receive help faster

[altsheets]

Thanks a lot. I was on gitter several times already. Without success.

(though I'm happy to provide it)

Thanks a lot for that.

---

note this is an issue tracker for go-ethereum

I know, and I am sorry. But:

I see this same question ("all tx of an ETH address") in many places all over the internet - and unanswered. That is why I thought it to be of general interest, and asked here.

Also it is meant to hint at a "request for code", for a future version of go-ethereum. Call me a naive dreamer * lol * - but of course I would rather like a simple basic API call --> please study the excellent NXT API function 'getAccountTransactions' - that is like where you want to end up at some moment in your future. You do not want to live much longer without that function, which is so essential for coding a money system.

But until you have created that urgently necessary function in your base code ... I already need its results ... soonish.

That's why I asked for example code here, to create a workaround. Sadly, it really looks as if I will have to create a whole database, and parse the whole ETH chain myself. Only then I can recycle the information from the 800k-loop. Otherwise with each new address I would need to start anew.

Am I right?
What a hassle ... :-(

---

contract T { ...

To create a contract costs money, right?

Just to see the transactions into or out of someone's account ... I have to pay money each time?

I am still a newbie, so please forgive me, if I am understanding some of this stuff completely wrong.

---

P.S.: Is there a better place where I can bring this "request for code" to, where it will be read by the essential deciders, and architects?

[obscuren]

You do not want to live much longer without that function, which is so essential for coding a money system.

Ethereum provides you with the tools and means to code up "features" through contracts. If you require extra functionality you can extend it, at a cost.

If you require indexing of some type ... you code it up through a contract.
If you require an authentication system ... you code it up through a contract.
If you require storing data ... you code it up through a contract.
If you require ... you code it up through a contract.

Ethereum is a generic, extendible, computation platform with consensus and an immutable data structure . It's not your average Bitcoin blockchain with a very specific use case :-)

To create a contract costs money, right?

Creating a contract costs Ether, correct. But you get so much in return! (an extendible, flexible blockchain).

Just to see the transactions into or out of someone's account ... I have to pay money each time?

Storing data on the blockchain cost money. Retrieving that data does not cost money.

[altsheets]

Thanks a lot for your answer. Very interesting.

I like what ETH is adding to the landscape. Otherwise I would not be here.

However, I still don't understand how dropping essential, proven, good concepts is a good thing.

And I don't get the defending of the status quo of an unready system, against the introduction of a such much wanted function, which is needed by so many people? Just google it. And even see the above "... everyone would like to see support for...".

But that's politics, anyways. If something's really needed, it will be built, one day.

[altsheets]

To create a contract costs money, right?

Creating a contract costs Ether, correct.

Ah, so the above strategy would cost money then?

Just to see the transactions into or out of someone's account ... I have to pay money each time?

Storing data on the blockchain cost money. Retrieving that data does not cost money.

Good to know. Not sure I understand how this relates to the case at hand. Creating that above suggested contract would cost money, and later querying it won't cost money anymore? Correct?

What if I wanted to study all the transactions of 10 different accounts? Would I need to create 10 contracts for that, and pay 10 times? And what if I wanted to study 100 accounts? 1000 accounts?

If you require indexing of some type ... you code it up through a contract.

Idea! (-:
Would it be a working strategy if someone created such a contract for each of all the ETH addresses which appeared on the block chain until now - and then everyone could simply query those contracts to get the answer to the "missing function" of in- and out-going transactions? How much would that costs for all ETH addresses in existence now? Perhaps it is cheap, and the solution to our needs could simply be paid for.

Thanks a lot.

[obscuren]

Ah, so the above strategy would cost money then?

Yes. Talking micro cents here tho.

Good to know. Not sure I understand how this relates to the case at hand.

It doesn't. I'm trying to show you that if you require any additional functionality not provided by default you'd have to code it up :-)

Would it be a working strategy if someone ...

It's a nice strategy but isn't required by default, in fact we used to have this functionality but decided to remove it since it simply isn't required for most use cases. It's only interesting in cases where people actually do value transfers. For example I use my <address> as an identity and there's an identity management system on the blockchain that I can interact with. I don't care about the information when I interact with the identity system, I only care that it does it right and any of the in or outputs have no additional value, to me, as a user of the identity "software". Note here that I'm never sending it any value (hence why I not need any additional overhead of "storing" the interaction).

[karalabe]

The confusion I think stems from the fact that we have two types of accounts: user accounts and contracts.

The system was really designed to support complex contracts, where user accounts only initiate a contract function, but otherwise in itself is not interesting. If you use your account in this context, then it's much less interesting what a user did as it "should" just interact with contracts. In the contract itself you could add arbitrarily complex logging to meet your specific needs.

Now the two places where this assumption doesn't hold as strongly as it could is:

• Even though you can code arbitrary contracts to fit your inspection needs, you cannot force this upon someone else. So inspecting 3rd party contracts is not that easy.
• Even though user accounts should only interact with contracts, people obviously use them as direct person-to-person transactions, but we never really targeted this use case so the associated tooling is limited.

Neither of the use cases above are inherently bad, but as they are not the way the system is "meant" to be used, official implementations tend not to focus on them as much. The reason is that it's extra non-insignificant maintenance cost with a potentially significant runtime and storage complexity.

AFAIK, Vitalik's Serenity proposal has a few ideas to move further things into contracts and leave much less complexity in implementations, one of which is to remove the concept of user accounts altogether (haven't spent too much time reading the proposals, so cannot say anything more for sure). In essence, the development direction is to make Ethereum harder to use badly, and inherently ensure that the core functionality can cover all the use cases people what to hit it with, without requiring a constant effort being poured into the implementations themselves to keep up. The more we can push into contracts, the more inherently works in all implementations and less custom code is required client side.

[altsheets]

Thanks for all the background information! Very interesting.

[EvilJordan]

Hey, @altsheets, did you ever figure out a solution for this. You are not wrong in wanting this functionality and I find the intense opposition to it startling.

I really liked the suggestion, in the now infamous #1897, of how a "full" node would have a special admin function or the like to avoid the issue of light nodes not having complete information.

It's very disappointing to me to see that the conversation around this issue hasn't progressed, gets shut down/turned private, and there seem to be no existing solutions out there (if there are, people are keeping them closely guarded for whatever reason).

Anyway, just checking in, two years later, to see if you've made any progress you'd be willing to share!

[dskvr]

Developer 1: "How do I query all transactions belonging to an address?"
Developer 2: "Iterate the entire blockchain and check against the desired address"
Developer 1: "Seems legit" (said no one ever)

[karalabe]

Ethereum does not have any consensus mechanism to verify such an information. You're only option is to reprocess all the blocks and transactions since the genesis and whenever your desired account is touched, you make a note of it. This is completely unrealistic for any user other than block explorers, so that's the reason for the opposition to make all nodes bear such burden.

As long as there's no consensus mechanism, it's impossible for a fast synced, warp synced or light synced node to verify this information. Without being able to verify it, a remote node can feed you arbitrary junk about what your past transactions were and you have absolutely no means to check it.

[EvilJordan]

Lol. You closed it. You guys are doing amazing work, but running away from this is going to bite Ethereum in the ass, hard.

Right now, the only way for any non-custom solution on a developer-owned-full-node running in an external-database in order to view a history of transactions (both "internal" and "external") is via etherscan or ethplorer or similar... the block explorers you mentioned. These are even worse than a remote node, as they are one system, opaque, proprietary, and without a consensus. There's no way to really know if things aren't being omitted or manipulated.

So... the problem you mentioned:

a remote node can feed you arbitrary junk about what your past transactions were and you have absolutely no means to check it

exists right now, it's just not integrated programmatically across the network.

What good is an immutable chain if you can't look back and see what's stored in it? How in the world would a future financial institution/economy run an audit if they can't pull the data and know for sure everyone's looking at the same thing? All of their trust would have to be in one of these block explorers. Isn't that exactly the opposite of what's attempting to be built here?

I'm not trying to be inflammatory or disrespectful, I just don't understand how this is considered a non-issue. Please educate me!

[altsheets]

:-)

[0xae]

@altsheets i'm having the very same problem here as i'm working on a ETH android wallet, processing lots of blocks and all their transactions is a headache even in the testnet, the sugestion regarding contracts is not feasible because it doenst make sense to charge my customers just to show them the transactions, no matter how cheap it is, the only viable solution is a db hosted somewhere indexing interested transactions.

The community needs this, even if it is implemented with a capping or pagination or something else.
This is not secondary, since light wallets are the most comfortable way to interact with the coins and even contracts in my opinion and my customers as well.

Sorry for my poor english, i hope you get it.
Awaiting any reactions...

[abrinckm]

I'm assuming this conversation will continue though the issue has been closed. @EvilJordan I absolutely agree with you.

So from what I understand from reading this thread is that if we as a developer have a need for looking at the transaction ledger for a contract, (say to examine the complete lifecycle of an asset which we have defined in the contract), then we must provide our own functionality for that in the smart contract? Doesn't that become very expensive as it puts O(n) complexity directly into the smart contract?

I have another scenario which involves using the ledger of transactions to back a local cache of data. I would argue that though local cache is centralized, we have the decentralized blockchain to back its trust. (and yes there are reasons for caching data locally: we can search, filter, and query or ask anything about the asset we've defined on the contract. these things could be done if we indexed data in a local db) If the cache were destroyed or was claimed corrupt, we could regenerate the cache totally from the transactions made to the smart contract... but I need those transactions to do this. And I want to regenerate the cache periodically so sifting through big data to filter these out would be a terrible waste of resources.

or perhaps I'm the one missing something here.

[abrinckm]

@karalabe I don't understand. If I'm running a peer, and have a copy of the blockchain, and my peer is synced, then the data is verified up to the last moment it was synced. Couldn't a filter mechanism be implemented in a way that can only be read locally and safely assume the information is correct up to last sync without placing burden on other peers? I'm new to this whole thing.

[mdcoon]

I agree with those asking for GETH to simply provide another indexing scheme within the local DB it already maintains. Instead of just indexing by block/hash, I don't see how it's that much work to index by to/from addresses. The only thing we need is a mapping from address=>block#, txnIndex. Given the block# and transaction index, we can retrieve the raw txn and receipt as normal. Otherwise, as already mentioned, we have to rely on centralized solutions or scan 5M+ blocks just to get our transactions. It seems like the benefits of local address indexing would outweigh the costs.

[altsheets]

Hey, @altsheets, did you ever figure out a solution for this.

No.

It is long ago, and I eventually delivered what I wanted to ... without that whole functionality.
What a pity, though.

When -under time constraints- I considered whether to build that DB myself or not, I decided it is not my job to create a functionality that sees so much demand from everyone. I still think, it is a core functionality, which is missing.

You are not wrong in wanting this functionality

I know. And we are many. Sometimes feature requests are built into great software ... with much less people wanting it.

and I find the intense opposition to it startling.

Yes.

By now I understand that Ethereum itself is built in a way that makes it a bit more complicated to implement this -highly useful- functionality.

But to refuse it right away - has always puzzled me too. And none of the arguments are convincing, apart from:

I agree with the opposition that the client in default mode should NOT provide that mapping address --> [ (blocknumber, tx-index) ], just because that would blow up the storage need massively.

However, to add an additional switch --txindex to geth / parity / ... would not harm Ethereum much, would it? As Ethereum is one of the richest projects on the planet, the only reason I can think of to oppose a technical innovation, is that it would create a future, in which bad things would be happening because of it. But I can think of no bad thing happening, when we developers get one more RPC call eth_listTransactionsOfAccount - or can you?

Imagine I start my geth --txindex, what would happen?

Of course, it would not be able to sync from a snapshot.
Instead it would have to -once- sync the whole chain, to create that txindex. That (soon Terabyte) download might take days, but once it's done, it's done; each new block would just cause a few extra milliseconds & bytes.

And imagine it one step cleverer, with geth --txindex 5000000, I can ask this revolutionary dangerous luxurious new functionality ... to only sync from block 5000000; because I really do not need the transactions from the stone (or ice) age, but e.g. only this year. Then I can control even the HD storage waste.

There could even be a protocol on top which works like the current snapshot syncing, and allows to sync a recent txindex-DB from other nodes.

Security ("consensus mechanism"?) idea:

If you put the address --> [ (blocknumber, tx-index) ] (or address --> { nonce --> (blocknumber, tx-index) } or address --> [ txHash ]) mapping into some Merkle trie structure, then an SPV-like mechanism that you actually got the *consensus-*txindex ... should be possible, or not?

Now in contrast

look at the current situation:

The "opposition" team refused to build a feature into the clients that dozens (hundreds) of developers were asking (searching) for. What happened since then?

Then of course, a minority of devs has solved it themselves, traversing the whole chain, putting everything into some local database - done. Without a doubt some of those solutions look almost identical, so it was a huge waste of humanpower (to that waste also add these endless discussions here); and very probably some implementations are even faulty, and are now causing havoc for endusers. But all those implementations together are ... a minority, a small one.

Another, probably larger minority (and my project from back then falls into that category) - never implemented a functionality which would have been really nice to have; or if it was essential, dropped their whole project, because they couldn't do it with simple enough means. There is a big difference between simply calling a standardized RPC function, and implementing a local DB solution. And when you have an idea for an app, you are busy enough with your own questions.

The largest group however, is the bad outcome that this "opposition" can call itself responsible for:

The vast majority of devs who for years have been asking for this functionality ... is now simply querying centralized services like https://etherscan.io/apis . No need to mention any of the dozens of arguments why this is a dangerous and bad outcome. And ideologically just wrong.

But ... this issue is closed.

[altsheets]

What would be a nice T-Shirt slogan for (or against) the opposition to a eth_listTransactions RPC endpoint?

[EvilJordan]

Thank you for the thoughtful reply, and I agree with everything you've said.

I'm totally in to a t-shirt (for!) slogan!
Perhaps print @obscuren's code block and comment (#2104 (comment)) with a 🙄 emoji...

eth_listTransactions... because srsly?

[altsheets]

I have long moved on, so I am really not bothered by this anymore. But as I had opened this issue, I thought I reply a last time, and give my 2cents, now few years later. Be not surprised if I don't get back to this. But I close with some ...

Constructive suggestions:

• it is not difficult to build it yourself. Example code is above. ((( yes, we want that T-Shirt! Anyone interested in further shortening obscuren's code, so that it nicely fits on the back of a shirt? )))
• more than 2 years ago, @Gustav-Simonsson forked geth, and implemented a solution! And he benchmarked it - 12 hours, 600 MB - at 1 million blocks? No one seemed to have bothered him creating that dangerous functionality (he/his-account is still alive ;-) ) - but they also did not merge it, and the fork is 2500 commits behind now.
• try again: fork geth, and just implement that dangerous-and-unwanted-but-much-asked-for functionality --txindex 5000000 yourself? Anyone? If you do want to try, then perhaps begin by compiling a concept from all the good ideas in e.g. these places: add API to get transactions for accounts #1749 and eth_listTransactions #1897 and Feature-request: _listTransactions()  openethereum/parity-ethereum#3567 and this transactions of an address - example code please #2104 (and stackoverflow, bountysource and many other places - google)
• perhaps https://github.com/paritytech/parity/issues is more open to this idea, and wants to offer more to developers, to gain an edge over the mainstream geth client? Gav's comments did not read like that, back then - but very smart people sometimes do change their mind.
• perhaps blk-io/blk-explorer-free helps? Any other open source blockexplorer DB-backend available?
• an ICO after which etherscan opensources their code? ScanCoin ;-)
• as this one is closed, open a new issue?

[MelbourneDeveloper]

@fjl what is the contract option? Is there a sample you can share?

[MelbourneDeveloper]

@karalabe
@obscuren

I agree with many of the comments here. You guys seem to be coming at this from the perspective of someone who is running a node. But, people who use the JSON RPC won't actually be people running the node - that's the whole point. For apps that need data in a quick way, we need to get a list of transactions. It's that simple.

Anyway, I keep hearing mention of the "contract option". What is the contract option?

My use case is that I need to make a JSON RPC call to find out what contracts are at a given address, and then do a lookup to find out what those tokens are. This is totally necessary. Please don't tell me I have setup my own node and then place an indexed database on top of that to serve up the data.

[EvilJordan]

@MelbourneDeveloper This is the contract option: #2104 (comment)

Basically, the proposal is to deploy a contract to the blockchain per address you'd like to monitor and have the contract emit data to you when a certain event occurs. It's a non-solution.

It sounds like what you want to do for your use case is exactly what Etherscan recently deployed (except for the returning what tokens a contract/address owns... you have to know in advance or have the user provide). But, alas, this is exactly what you fear: a node with an indexed DB on top.

[jdneidig]

My vote is for eth_listTransaction

We went ahead and made a T-shirt :-)
It’s Free https://simbachain.com/community/join-our-community/

[jdneidig]

GET the eth_listTransaction T-shirt :-)

https://simbachain.com/community/join-our-community/

[MelbourneDeveloper]

@EvilJordan , yep, that's the API I need to replicate. But since that API is non-standard, it's got no fail-over redundancy and is therefore useless in apps.

[Bavragor]

Ethereum does not have any consensus mechanism to verify such an information. You're only option is to reprocess all the blocks and transactions since the genesis and whenever your desired account is touched, you make a note of it. This is completely unrealistic for any user other than block explorers, so that's the reason for the opposition to make all nodes bear such burden.

As long as there's no consensus mechanism, it's impossible for a fast synced, warp synced or light synced node to verify this information. Without being able to verify it, a remote node can feed you arbitrary junk about what your past transactions were and you have absolutely no means to check it.

This means with switching to PoS we will have a consensus mechanism, meaning we will be able to do this more efficiently?

Tax tools are really lack luster currently, gonna write sth to support variety of transactions and make it understandable for these tools, so I was searching for a solution to efficiently get all transactions for a wallet.