eth_listTransactions #1897

Closed
tymat opened this Issue Oct 12, 2015 · 130 comments

Comments

Projects
None yet
@tymat

tymat commented Oct 12, 2015

This feature has been discussed on gitter in the past and we would like to see this get implemented for accounts and contracts.

Proposal

eth_listTransactions

Returns a list of transaction hashes for a given external or contract account

Parameters

String - the account or contract address

params: [
  "0x385acafdb80b71ae001f1dbd0d65e62ec2fff055"
]

Returns

DATA - A list of transaction hashes

@gavofyork

This comment has been minimized.

Show comment
Hide comment
@gavofyork

gavofyork Oct 12, 2015

Contributor

This is useless - transactions are merely one way of two ways to send a message to an object ("contract") in Ethereum; CALL would be the other way.

Use logs. It's what they were designed for.

Contributor

gavofyork commented Oct 12, 2015

This is useless - transactions are merely one way of two ways to send a message to an object ("contract") in Ethereum; CALL would be the other way.

Use logs. It's what they were designed for.

@gavofyork

This comment has been minimized.

Show comment
Hide comment
@gavofyork

gavofyork Oct 12, 2015

Contributor

would close.

Contributor

gavofyork commented Oct 12, 2015

would close.

@Gustav-Simonsson

This comment has been minimized.

Show comment
Hide comment
@Gustav-Simonsson

Gustav-Simonsson Oct 14, 2015

Member

@gavofyork It's far from useless. For example, it would be very useful for keeping track of non-contract txs, which cannot use logs.

This feature has been requested from several people. We shouldn't force users to only use the platform in a certain way, and If users desire a API call which doesn't conflict with anything or have any significant drawbacks, we should consider adding it.

I'm in favour of adding this.

Member

Gustav-Simonsson commented Oct 14, 2015

@gavofyork It's far from useless. For example, it would be very useful for keeping track of non-contract txs, which cannot use logs.

This feature has been requested from several people. We shouldn't force users to only use the platform in a certain way, and If users desire a API call which doesn't conflict with anything or have any significant drawbacks, we should consider adding it.

I'm in favour of adding this.

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Oct 14, 2015

Member

So am I. But let the community decide on this. Get enough +1 and it will eventually be considered/added.

Member

obscuren commented Oct 14, 2015

So am I. But let the community decide on this. Get enough +1 and it will eventually be considered/added.

@ethernomad

This comment has been minimized.

Show comment
Hide comment
@ethernomad

ethernomad Oct 15, 2015

Contributor

Maybe messages to non-contract addresses should automatically get a blank log entry? It would store the originating transaction id.

Contributor

ethernomad commented Oct 15, 2015

Maybe messages to non-contract addresses should automatically get a blank log entry? It would store the originating transaction id.

@obscuren obscuren referenced this issue in ethereum/meteor-dapp-wallet Oct 15, 2015

Closed

Confusion: Contract based simple wallets #47

@frozeman

This comment has been minimized.

Show comment
Hide comment
@frozeman

frozeman Oct 15, 2015

Member

👍 I fully agree with gustav here. Even though most of the cool stuff can happen in contracts, being able to simply send money from one account to the other will be heavily used. And then you need a easy way to display its transactions (e.g. the wallet needs that for example :) )

I would add a second parameter, which is the nonce, from which on you want to get transactions. This way you don't need to request the whole list over and over again.

Member

frozeman commented Oct 15, 2015

👍 I fully agree with gustav here. Even though most of the cool stuff can happen in contracts, being able to simply send money from one account to the other will be heavily used. And then you need a easy way to display its transactions (e.g. the wallet needs that for example :) )

I would add a second parameter, which is the nonce, from which on you want to get transactions. This way you don't need to request the whole list over and over again.

@tymat

This comment has been minimized.

Show comment
Hide comment
@tymat

tymat Oct 20, 2015

eth_listTransactions is very much needed for Ether to Ether transactions. For wallet devs we need a way to present this information to users so they know how their Ether was spent and when it was spent and which TX hashes they occurred in. Adding a separate persistence layer to store this data for easy access is cumbersome and would have to be constantly updated on the client side.

This is very useful and much needed.

tymat commented Oct 20, 2015

eth_listTransactions is very much needed for Ether to Ether transactions. For wallet devs we need a way to present this information to users so they know how their Ether was spent and when it was spent and which TX hashes they occurred in. Adding a separate persistence layer to store this data for easy access is cumbersome and would have to be constantly updated on the client side.

This is very useful and much needed.

@slothbag

This comment has been minimized.

Show comment
Hide comment
@slothbag

slothbag Oct 20, 2015

+1 I found myself looking for this feature.. would be great to get it included.

+1 I found myself looking for this feature.. would be great to get it included.

@VoR0220

This comment has been minimized.

Show comment
Hide comment
@VoR0220

VoR0220 Oct 20, 2015

Contributor

👍

Contributor

VoR0220 commented Oct 20, 2015

👍

@VoR0220

This comment has been minimized.

Show comment
Hide comment
@VoR0220

VoR0220 Oct 20, 2015

Contributor

this is much more efficient IMO than the logs...and there's a lot of use cases for this feature for execution in other contracts.

Contributor

VoR0220 commented Oct 20, 2015

this is much more efficient IMO than the logs...and there's a lot of use cases for this feature for execution in other contracts.

@VoR0220

This comment has been minimized.

Show comment
Hide comment
@VoR0220

VoR0220 Oct 20, 2015

Contributor

I would add a third optional parameter for number of transactions from the top or bottom (perhaps something python-esque?)

Contributor

VoR0220 commented Oct 20, 2015

I would add a third optional parameter for number of transactions from the top or bottom (perhaps something python-esque?)

@frozeman

This comment has been minimized.

Show comment
Hide comment
@frozeman

frozeman Oct 20, 2015

Member

Though i +1 this feature @VoR0220 you can't get logs only from transactions, as there can also be logs created when one contract interacts with another.. So only relying on this feature won't work.

I don't like the third parameter actually, as a nonce (from which nonce to get tx on), is more than enough to control the tx flow.

Member

frozeman commented Oct 20, 2015

Though i +1 this feature @VoR0220 you can't get logs only from transactions, as there can also be logs created when one contract interacts with another.. So only relying on this feature won't work.

I don't like the third parameter actually, as a nonce (from which nonce to get tx on), is more than enough to control the tx flow.

@VoR0220

This comment has been minimized.

Show comment
Hide comment
@VoR0220

VoR0220 Oct 20, 2015

Contributor

how does nonce control how many transactions you would like to see?

Contributor

VoR0220 commented Oct 20, 2015

how does nonce control how many transactions you would like to see?

@d11e9

This comment has been minimized.

Show comment
Hide comment

d11e9 commented Oct 21, 2015

👍

@frozeman

This comment has been minimized.

Show comment
Hide comment
@frozeman

frozeman Oct 21, 2015

Member

If you say i want to get all transactions from nonce 50 and there a re a total of 60 tx, you will get 10

Member

frozeman commented Oct 21, 2015

If you say i want to get all transactions from nonce 50 and there a re a total of 60 tx, you will get 10

@bas-vk

This comment has been minimized.

Show comment
Hide comment
@bas-vk

bas-vk Oct 23, 2015

Member

@frozeman, what if you want to retrieve tx [100...150] when there are 10.000+ transactions?

Member

bas-vk commented Oct 23, 2015

@frozeman, what if you want to retrieve tx [100...150] when there are 10.000+ transactions?

@VoR0220

This comment has been minimized.

Show comment
Hide comment
@VoR0220

VoR0220 Oct 23, 2015

Contributor

@frozeman but that's for a nonce in the chain correct? So I would only get the transactions for that block, no (I might be totally off here and feel free to correct me)? However, if I wanted to go and see the past 50 or so transactions for a particular address, how does that nonce benefit me if I want to get specific with the transactions displayed?

Contributor

VoR0220 commented Oct 23, 2015

@frozeman but that's for a nonce in the chain correct? So I would only get the transactions for that block, no (I might be totally off here and feel free to correct me)? However, if I wanted to go and see the past 50 or so transactions for a particular address, how does that nonce benefit me if I want to get specific with the transactions displayed?

@frozeman

This comment has been minimized.

Show comment
Hide comment
@frozeman

frozeman Oct 23, 2015

Member

nonce is an ear increasing number of transactions from an account.
With web3.eth.getTransactionCount('0x12345..') you basically get the next used nonce (as nonces start from 0, like array.length)

And then you can subtract whatever number you want to go back and ask for those tx.

Member

frozeman commented Oct 23, 2015

nonce is an ear increasing number of transactions from an account.
With web3.eth.getTransactionCount('0x12345..') you basically get the next used nonce (as nonces start from 0, like array.length)

And then you can subtract whatever number you want to go back and ask for those tx.

@frozeman

This comment has been minimized.

Show comment
Hide comment
@frozeman

frozeman Oct 23, 2015

Member

@bas-vk you couldn't :)

Member

frozeman commented Oct 23, 2015

@bas-vk you couldn't :)

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Oct 23, 2015

Member

While easy to implement its difficult to do it right. Indexing all transactions would require quite significant storage and most certainly doesn't scale.

One way to do this is as how we MIP mapped LOGs and use a similar query mechanism, but this is an expensive and CPU/IO intensive operation because there are far more transactions than there are LOGs.

Doing this for LOGs is actually fine since you more or less paid for it where as transactions incurs no additional cost in terms of upfront payment. letting the user decide what to map isn't an option either since it just won't work for light clients. Light clients depend on archive and full nodes for data retrieval and can't make use of this "user setting" option because they don't receive any other data than the data that is explicit requested.

I understand this is a much requested feature but understand that it comes at a great, expensive cost and, as I said earlier, difficult to do it right.

Member

obscuren commented Oct 23, 2015

While easy to implement its difficult to do it right. Indexing all transactions would require quite significant storage and most certainly doesn't scale.

One way to do this is as how we MIP mapped LOGs and use a similar query mechanism, but this is an expensive and CPU/IO intensive operation because there are far more transactions than there are LOGs.

Doing this for LOGs is actually fine since you more or less paid for it where as transactions incurs no additional cost in terms of upfront payment. letting the user decide what to map isn't an option either since it just won't work for light clients. Light clients depend on archive and full nodes for data retrieval and can't make use of this "user setting" option because they don't receive any other data than the data that is explicit requested.

I understand this is a much requested feature but understand that it comes at a great, expensive cost and, as I said earlier, difficult to do it right.

@frozeman

This comment has been minimized.

Show comment
Hide comment
@frozeman

frozeman Oct 23, 2015

Member

I don't expect dapps requesting this endpoint often, rather only once when a dapp syncs for the first time. It will then keep this information in a local database anyway. And only requests the last x tx.

Remote nodes would have to rate limit their requests anyway somehow.

Member

frozeman commented Oct 23, 2015

I don't expect dapps requesting this endpoint often, rather only once when a dapp syncs for the first time. It will then keep this information in a local database anyway. And only requests the last x tx.

Remote nodes would have to rate limit their requests anyway somehow.

@m888m

This comment has been minimized.

Show comment
Hide comment
@m888m

m888m Oct 24, 2015

I second the need for transaction listings.
Regarding implementation, to compromise between system load and user friendliness, I suggest to use nonce as single argument, and implement a filter on the gui where users can limit time, counterparties, amounts etc.

m888m commented Oct 24, 2015

I second the need for transaction listings.
Regarding implementation, to compromise between system load and user friendliness, I suggest to use nonce as single argument, and implement a filter on the gui where users can limit time, counterparties, amounts etc.

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Oct 24, 2015

Member

@frozeman that's quite an assumption you are making there and neither solves the issue at hand I described above.

@m888m the nonce is hardly the issue

Member

obscuren commented Oct 24, 2015

@frozeman that's quite an assumption you are making there and neither solves the issue at hand I described above.

@m888m the nonce is hardly the issue

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Oct 24, 2015

Member

Remote nodes would have to rate limit their requests anyway somehow.

It's not about rate limiting either. Please reread

Member

obscuren commented Oct 24, 2015

Remote nodes would have to rate limit their requests anyway somehow.

It's not about rate limiting either. Please reread

@rfikki

This comment has been minimized.

Show comment
Hide comment
@rfikki

rfikki Oct 24, 2015

list of transaction hashes for a given external or contract account. +1

rfikki commented Oct 24, 2015

list of transaction hashes for a given external or contract account. +1

@sammy007

This comment has been minimized.

Show comment
Hide comment
@sammy007

sammy007 Oct 24, 2015

+1, with nonce param.

+1, with nonce param.

@m888m

This comment has been minimized.

Show comment
Hide comment
@m888m

m888m Oct 25, 2015

@obscuren nonce was not my main point, the need for user friendly transaction listings was. To push end user adoption, we definitely need a user-friendly way to show past transactions that happened during a certain time period and with certain counter parties. Furthermore, the transaction fees occurred for a set of transactions must be easily established.

m888m commented Oct 25, 2015

@obscuren nonce was not my main point, the need for user friendly transaction listings was. To push end user adoption, we definitely need a user-friendly way to show past transactions that happened during a certain time period and with certain counter parties. Furthermore, the transaction fees occurred for a set of transactions must be easily established.

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Oct 25, 2015

Member

@m888m mine neither. I understand what everyone has reiterated over and over again but as I've mentioned above in great detail; there are issues with the requested feature. It's not as simple as it seems.

Member

obscuren commented Oct 25, 2015

@m888m mine neither. I understand what everyone has reiterated over and over again but as I've mentioned above in great detail; there are issues with the requested feature. It's not as simple as it seems.

@tymat

This comment has been minimized.

Show comment
Hide comment
@tymat

tymat Nov 16, 2015

Indexing of this can be an optional setting when starting up the node. It doesn't have to be a requirement so clients that do not have CPU power can just turn the feature off.

Right now most of the web wallets and mist based wallets are forced to use some external persistence mechanism to have this data available to the user so any efforts to save CPU/etc... becomes a zero sum effort and you add an additional TCP/socket round-trip just to get the data from the blocks and then another connection to the persistence layer. Is the cost of doing indexing on the node greater than that? Probably not.

tymat commented Nov 16, 2015

Indexing of this can be an optional setting when starting up the node. It doesn't have to be a requirement so clients that do not have CPU power can just turn the feature off.

Right now most of the web wallets and mist based wallets are forced to use some external persistence mechanism to have this data available to the user so any efforts to save CPU/etc... becomes a zero sum effort and you add an additional TCP/socket round-trip just to get the data from the blocks and then another connection to the persistence layer. Is the cost of doing indexing on the node greater than that? Probably not.

@caktux

This comment has been minimized.

Show comment
Hide comment
@caktux

caktux Nov 16, 2015

Contributor

Got my vote. Maybe index the node's accounts by default and all accounts only optionally?

Contributor

caktux commented Nov 16, 2015

Got my vote. Maybe index the node's accounts by default and all accounts only optionally?

@jambtt

This comment has been minimized.

Show comment
Hide comment
@jambtt

jambtt Nov 17, 2015

+1 add getTransactions(address) to API.

jambtt commented Nov 17, 2015

+1 add getTransactions(address) to API.

@karalabe

This comment has been minimized.

Show comment
Hide comment
@karalabe

karalabe Nov 23, 2015

Member

Just a thought. As already mentioned, doing a full index of transactions would be costly both CPU and storage wise. However, if we consider that most people will only ever care about a handful of accounts, we could do a heuristics whereby a node would only index transactions for:

  • Etherbase
  • All accounts reported by eth.accounts
  • Any account previously requested

This way the storage price would be linear to the operations you want to perform (i.e. you only expand CPU/HDD resources on the ops you personally need).

Note, this is not a +1 for this issue. I'm not yet fully sold on this. It's only a discussion idea that we could consider if/when implementing this feature.

Member

karalabe commented Nov 23, 2015

Just a thought. As already mentioned, doing a full index of transactions would be costly both CPU and storage wise. However, if we consider that most people will only ever care about a handful of accounts, we could do a heuristics whereby a node would only index transactions for:

  • Etherbase
  • All accounts reported by eth.accounts
  • Any account previously requested

This way the storage price would be linear to the operations you want to perform (i.e. you only expand CPU/HDD resources on the ops you personally need).

Note, this is not a +1 for this issue. I'm not yet fully sold on this. It's only a discussion idea that we could consider if/when implementing this feature.

@wildbunny

This comment has been minimized.

Show comment
Hide comment
@wildbunny

wildbunny Nov 23, 2015

If you guys are considering adding this, please can we have a 'since block' parameter? That's how exchanges sync their coin's blockchains and it would save them time to have this parameter.

If you guys are considering adding this, please can we have a 'since block' parameter? That's how exchanges sync their coin's blockchains and it would save them time to have this parameter.

@tymat

This comment has been minimized.

Show comment
Hide comment
@tymat

tymat Nov 25, 2015

+1 on the idea of doing a heuristic approach.

tymat commented Nov 25, 2015

+1 on the idea of doing a heuristic approach.

@chrisfranko

This comment has been minimized.

Show comment
Hide comment
@chrisfranko

chrisfranko Dec 17, 2015

+1 would like to see this

+1 would like to see this

@altsheets

This comment has been minimized.

Show comment
Hide comment
@altsheets

altsheets Dec 21, 2015

Still no solution for this? (In September I got an answer "might do that as a weekend project").

You are funny people.

If I cannot simply check for transactions out of and into my account, it is pretty useless as a currency, no?

Until you start listening to your userbase ... please publish a fully working code that achieves the same result, that we can use for the time being.

Thanks.

Still no solution for this? (In September I got an answer "might do that as a weekend project").

You are funny people.

If I cannot simply check for transactions out of and into my account, it is pretty useless as a currency, no?

Until you start listening to your userbase ... please publish a fully working code that achieves the same result, that we can use for the time being.

Thanks.

@altsheets

This comment has been minimized.

Show comment
Hide comment
@altsheets

altsheets Dec 21, 2015

i.e. publish working code for getting all (recent) transactions of an account.
I suppose it is done with this: https://github.com/ethereum/wiki/wiki/JavaScript-API#web3ethfilter ?

Please give working example code not for "pending" but for transactions of an account .

Thanks a lot!

i.e. publish working code for getting all (recent) transactions of an account.
I suppose it is done with this: https://github.com/ethereum/wiki/wiki/JavaScript-API#web3ethfilter ?

Please give working example code not for "pending" but for transactions of an account .

Thanks a lot!

@altsheets

This comment has been minimized.

Show comment
Hide comment
@altsheets

altsheets Dec 21, 2015

Fiddling around with those suboptimal manual instructions.

I am stuck here:

var options = {address: "0xf2cc0eeaaaed313542cb262b0b8c3972425143f0"};

var myfilter= web3.eth.filter(options);
console.log(myfilter);

myfilter.get(function (error, log) {
    console.log("get error:", error); 
    console.log("get log:", log); 
});

results in

f {options: Object, implementation: Object, filterId: null, callbacks: Array[0], pollFilters: Array[0]}

get error: Error: could not decode, invalid type on field : not a number or string(…)
get log: undefined

Any hint highly appreciated, thanks.

Fiddling around with those suboptimal manual instructions.

I am stuck here:

var options = {address: "0xf2cc0eeaaaed313542cb262b0b8c3972425143f0"};

var myfilter= web3.eth.filter(options);
console.log(myfilter);

myfilter.get(function (error, log) {
    console.log("get error:", error); 
    console.log("get log:", log); 
});

results in

f {options: Object, implementation: Object, filterId: null, callbacks: Array[0], pollFilters: Array[0]}

get error: Error: could not decode, invalid type on field : not a number or string(…)
get log: undefined

Any hint highly appreciated, thanks.

@altsheets

This comment has been minimized.

Show comment
Hide comment
@altsheets

altsheets Feb 17, 2016

@frozeman Yes, I read that. Good to know. Has to be added then. Or the user has to be notified that he is seeing incomplete data.

@frozeman Yes, I read that. Good to know. Has to be added then. Or the user has to be notified that he is seeing incomplete data.

@wildbunny

This comment has been minimized.

Show comment
Hide comment
@wildbunny

wildbunny Feb 17, 2016

@frozeman Is there yet a way to prove that funds from a contract actually entered any given account? None of the blockchain explorers support this the last time I checked, making it impossible to support in my application.

@frozeman Is there yet a way to prove that funds from a contract actually entered any given account? None of the blockchain explorers support this the last time I checked, making it impossible to support in my application.

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Feb 17, 2016

Member

Yes, by using logs.

Member

obscuren commented Feb 17, 2016

Yes, by using logs.

@wildbunny

This comment has been minimized.

Show comment
Hide comment
@wildbunny

wildbunny Feb 17, 2016

@obscuren Aren't logs only good for contemporary transactions? We need to be able to query historical transactions as well for robustness sake.

@obscuren Aren't logs only good for contemporary transactions? We need to be able to query historical transactions as well for robustness sake.

@bricky

This comment has been minimized.

Show comment
Hide comment
@bricky

bricky Feb 17, 2016

@altsheets I'm using @obscuren's approach from your other issue here and dumping the data into an external db. Certainly not an elegant solution.

bricky commented Feb 17, 2016

@altsheets I'm using @obscuren's approach from your other issue here and dumping the data into an external db. Certainly not an elegant solution.

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Feb 17, 2016

Member

LOGs can be queried/searched (historical) and watched.

Member

obscuren commented Feb 17, 2016

LOGs can be queried/searched (historical) and watched.

@wildbunny

This comment has been minimized.

Show comment
Hide comment
@wildbunny

wildbunny Feb 17, 2016

@obscuren My point is, if I suffer a power outage, or some other hardware loss, which includes the logs, can I query the chain to get the data back? That is what's required for full robustness.

@obscuren My point is, if I suffer a power outage, or some other hardware loss, which includes the logs, can I query the chain to get the data back? That is what's required for full robustness.

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Feb 17, 2016

Member

Yes. LOGs can be fetched at any given time.

var filter = contract.SomeEvent({param: "one"}, function(error, event) {
    // notifications would happen in here
})

filter.get(function(error, logs) {
     // historical data
});
Member

obscuren commented Feb 17, 2016

Yes. LOGs can be fetched at any given time.

var filter = contract.SomeEvent({param: "one"}, function(error, event) {
    // notifications would happen in here
})

filter.get(function(error, logs) {
     // historical data
});
@wildbunny

This comment has been minimized.

Show comment
Hide comment
@wildbunny

wildbunny Feb 17, 2016

@obscuren Even if my HD blows up? ... and I get a replacement

@obscuren Even if my HD blows up? ... and I get a replacement

@karalabe

This comment has been minimized.

Show comment
Hide comment
@karalabe

karalabe Feb 17, 2016

Member

@wildbunny Please stop trolling.

Member

karalabe commented Feb 17, 2016

@wildbunny Please stop trolling.

@wildbunny

This comment has been minimized.

Show comment
Hide comment
@wildbunny

wildbunny Feb 17, 2016

@karalabe This is a genuine question

@karalabe This is a genuine question

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Feb 17, 2016

Member

@wildbunny yes, even if your HD blows up. They are stored in the receipts of the blockchain. They are part of the consensus rules.

Member

obscuren commented Feb 17, 2016

@wildbunny yes, even if your HD blows up. They are stored in the receipts of the blockchain. They are part of the consensus rules.

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Feb 17, 2016

Member

That's why it's costing you Gas.

Member

obscuren commented Feb 17, 2016

That's why it's costing you Gas.

@wildbunny

This comment has been minimized.

Show comment
Hide comment
@wildbunny

wildbunny Feb 17, 2016

@obscuren Perfect, thank you

@obscuren Perfect, thank you

@frozeman

This comment has been minimized.

Show comment
Hide comment
@frozeman

frozeman Feb 17, 2016

Member

@wildbunny there is a fromBlock: 0 option, which gives you passed logs you may have missed ;) Please read the RPC docs or web3.js docs

Member

frozeman commented Feb 17, 2016

@wildbunny there is a fromBlock: 0 option, which gives you passed logs you may have missed ;) Please read the RPC docs or web3.js docs

@Gustav-Simonsson

This comment has been minimized.

Show comment
Hide comment
@Gustav-Simonsson

Gustav-Simonsson Feb 17, 2016

Member

@altsheets As I also enjoy abusing Ethereum as a currency, I started on a quick & dirty impl here: https://github.com/Gustav-Simonsson/go-ethereum/tree/list_transactions

It uses sqlite to index txs on both sender and recipient. It's WIP and pretty hacky - have to rewrite it quite a bit before it'll be ready for others to use.

Preliminary benchmarks:

  • sqlite DB size after indexing all mainnet txs: 600MB
  • JSON-RPC call listTransactions on address with 60 txs: <1ms. On address with 25k txs: <2s.
  • Indexing from scratch: 12 hours on a ivy bridge i7 machine.

All of these metrics can probably be improved a lot, as the current hack is just a first stab and not optimised.

TODOs:

  • Index during normal sync instead of traversing blockchain from HEAD to genesis.
  • Handle forks & reorgs (by integrating into the code that already handles this for the leveldb instance used for blocks)
  • Include txs from tx pool in API return
  • Return more than the tx hash, perhaps configurable in the API call
  • cmd line flags to make the call & required indexing optional, and perhaps also configure from which block to begin indexing (to save time if one knows how far back the txs one is interested in go)
Member

Gustav-Simonsson commented Feb 17, 2016

@altsheets As I also enjoy abusing Ethereum as a currency, I started on a quick & dirty impl here: https://github.com/Gustav-Simonsson/go-ethereum/tree/list_transactions

It uses sqlite to index txs on both sender and recipient. It's WIP and pretty hacky - have to rewrite it quite a bit before it'll be ready for others to use.

Preliminary benchmarks:

  • sqlite DB size after indexing all mainnet txs: 600MB
  • JSON-RPC call listTransactions on address with 60 txs: <1ms. On address with 25k txs: <2s.
  • Indexing from scratch: 12 hours on a ivy bridge i7 machine.

All of these metrics can probably be improved a lot, as the current hack is just a first stab and not optimised.

TODOs:

  • Index during normal sync instead of traversing blockchain from HEAD to genesis.
  • Handle forks & reorgs (by integrating into the code that already handles this for the leveldb instance used for blocks)
  • Include txs from tx pool in API return
  • Return more than the tx hash, perhaps configurable in the API call
  • cmd line flags to make the call & required indexing optional, and perhaps also configure from which block to begin indexing (to save time if one knows how far back the txs one is interested in go)
@altsheets

This comment has been minimized.

Show comment
Hide comment
@altsheets

altsheets Feb 17, 2016

wow @Gustav-Simonsson you are a star!

I gather it is mainly in eth/addr_txs.go ?

Indexing from scratch: 12 hours

Oops. But 12 * 3600 / 1018621 = 0.04241 is not too bad, actually (there is the 42 again! :-) )

not optimised

Which are the bottlenecks?

Could multithreading help? Query 500 blocks at a time? This supersimple question is not time-ordered.

todo: Handle forks & reorgs

yes, and that is actually one reason why I had refrained from just doing it myself.

Fantastic work!

Rawr

wow @Gustav-Simonsson you are a star!

I gather it is mainly in eth/addr_txs.go ?

Indexing from scratch: 12 hours

Oops. But 12 * 3600 / 1018621 = 0.04241 is not too bad, actually (there is the 42 again! :-) )

not optimised

Which are the bottlenecks?

Could multithreading help? Query 500 blocks at a time? This supersimple question is not time-ordered.

todo: Handle forks & reorgs

yes, and that is actually one reason why I had refrained from just doing it myself.

Fantastic work!

Rawr

gmkung added a commit to gmkung/homestead-guide that referenced this issue Feb 24, 2016

Cleaned up questions and removed some vague ones
These questions need to be more specific before they can be answers:


How will Ethereum deal with all security issues plaguing Bitcoin?


    Permits all sorts of contracts, allows for faster TXs, will utilise eco-friendly validation.
    It's a trustless system. Not another IOU system.
    Solves more important issues.
    A number of solutions, primarily blockchain sharding.
    Through carefully structured incentives.
    How can anyone deal with forking?
    Through the use of POS validation and aforementioned scaling solutions.
    What security issues?
ethereum/go-ethereum#1897
@fairglu

This comment has been minimized.

Show comment
Hide comment
@fairglu

fairglu Mar 14, 2016

This kind of information is also required for 3rd party auditing, currently ethereum is hard and error-prone to audit, even for basic transfers and coin generation. There is no point in having sophisticated gas and difficulty mechanics if the monetary part is not sound and auditable (not just for isolated accounts, but for all blockchain accounts and transactions). This is also needs to be auditable efficiently and in real-time, so that fool play can be detected and acted upon quickly.

While the core ethereum team is out there to build something, and spurious transactions are more likely to be bugs than scams, this cannot be said or assumed for clones and private chains. Since the code is very complex, it is very hard to justify code and consensus can be trusted, even with multiple implementations (actually multiple implementations for a private chain will only make the 3rd party auditing more expensive).

In summary, this is not a care-bear world, without standardized auditing (of which tx listing is a key part), things lean more on the trust side than on the trust-less one.

fairglu commented Mar 14, 2016

This kind of information is also required for 3rd party auditing, currently ethereum is hard and error-prone to audit, even for basic transfers and coin generation. There is no point in having sophisticated gas and difficulty mechanics if the monetary part is not sound and auditable (not just for isolated accounts, but for all blockchain accounts and transactions). This is also needs to be auditable efficiently and in real-time, so that fool play can be detected and acted upon quickly.

While the core ethereum team is out there to build something, and spurious transactions are more likely to be bugs than scams, this cannot be said or assumed for clones and private chains. Since the code is very complex, it is very hard to justify code and consensus can be trusted, even with multiple implementations (actually multiple implementations for a private chain will only make the 3rd party auditing more expensive).

In summary, this is not a care-bear world, without standardized auditing (of which tx listing is a key part), things lean more on the trust side than on the trust-less one.

@dmxhZGp1c2hh

This comment has been minimized.

Show comment
Hide comment
@dmxhZGp1c2hh

dmxhZGp1c2hh Mar 26, 2016

+1

Does anyone thinks that not knowing when, from who and how much you received is a cool feature?
Messages like Accounts can't display incoming transactions. does not make any sense. They can track amount, you can use them to send and receive ether, but somehow you can't track this.
Think about this from average user point of view. He needs to go to 3rd party site to check who send him ether.

As I understood the only option is to create contract. So I need to pay to receive ether? Really?

+1

Does anyone thinks that not knowing when, from who and how much you received is a cool feature?
Messages like Accounts can't display incoming transactions. does not make any sense. They can track amount, you can use them to send and receive ether, but somehow you can't track this.
Think about this from average user point of view. He needs to go to 3rd party site to check who send him ether.

As I understood the only option is to create contract. So I need to pay to receive ether? Really?

@altsheets

This comment has been minimized.

Show comment
Hide comment

:-)

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Mar 27, 2016

Member

I can only reiterate about what has already been said which makes this feature questionable and difficult to implement correct.

listTransactions will not work for light-clients. light-clients and full clients need to support the same set of functionality. Even if full nodes would index all transactions and all calls it still wouldn't be feasible since there's no way to actually verify that the transactions the node is giving you, are valid and correct. The only part which can be validated are the transaction, but since we're also dealing with contracts (which don't send transactions but invoke an EVM opcodes) we can't actually verify that value transfers that occurred during the execution of the contract are valid (a full node might be malicious).

As I understood the only option is to create contract. So I need to pay to receive ether? Really?

Pay to receive ether? Of course not. The sender is paying, not the receiver. The only part that would cost you ether is the deployment of the contract, which costs you something like 0.001 cents, if not less. This can't be an argument not to use contracts.

Member

obscuren commented Mar 27, 2016

I can only reiterate about what has already been said which makes this feature questionable and difficult to implement correct.

listTransactions will not work for light-clients. light-clients and full clients need to support the same set of functionality. Even if full nodes would index all transactions and all calls it still wouldn't be feasible since there's no way to actually verify that the transactions the node is giving you, are valid and correct. The only part which can be validated are the transaction, but since we're also dealing with contracts (which don't send transactions but invoke an EVM opcodes) we can't actually verify that value transfers that occurred during the execution of the contract are valid (a full node might be malicious).

As I understood the only option is to create contract. So I need to pay to receive ether? Really?

Pay to receive ether? Of course not. The sender is paying, not the receiver. The only part that would cost you ether is the deployment of the contract, which costs you something like 0.001 cents, if not less. This can't be an argument not to use contracts.

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Mar 27, 2016

Member

Please note that I'm not arguing that this isn't a good/nice/cool/epic feature. I'm arguing that, while seemingly easy to implement at first sight, is actually quite difficult to do correct, if at all.

This isn't bitcoin where the only thing you have are value transfers. Our core is much more complicated that a feature like this becomes difficult to implement or impossible to implement correctly at all.

Member

obscuren commented Mar 27, 2016

Please note that I'm not arguing that this isn't a good/nice/cool/epic feature. I'm arguing that, while seemingly easy to implement at first sight, is actually quite difficult to do correct, if at all.

This isn't bitcoin where the only thing you have are value transfers. Our core is much more complicated that a feature like this becomes difficult to implement or impossible to implement correctly at all.

@Joter85

This comment has been minimized.

Show comment
Hide comment
@Joter85

Joter85 Mar 27, 2016

@obscuren well that explains it then. last paragraph. thanks

Joter85 commented Mar 27, 2016

@obscuren well that explains it then. last paragraph. thanks

@dmxhZGp1c2hh

This comment has been minimized.

Show comment
Hide comment

@obscuren, understood.

@altsheets

This comment has been minimized.

Show comment
Hide comment
@altsheets

altsheets Mar 27, 2016

just to clarify, this approach #1897 (comment) is futile then?


EDIT: and with futile I meant, that what Gustav-Simonsson seems to have implemented already ... would lead to incorrect results? (Because if not, then just make it optional, or ship it as a standalone code/extension - and everyone's happy?). Working, correct code is the best argument possible. But some of the more recent comments since then would only be understandable, if his approach is ... simply wrong. I am confused. And that is why I asked for clarification - not for restating arguments from before his post, or inventing new ones. If there is already a solution in code, to answer the OP's question - then why argue whether such a solution should exist at all, or not? Makes no sense to me. So: Is his approach only slow/mem-hungry - or is it leading to incorrect results?

just to clarify, this approach #1897 (comment) is futile then?


EDIT: and with futile I meant, that what Gustav-Simonsson seems to have implemented already ... would lead to incorrect results? (Because if not, then just make it optional, or ship it as a standalone code/extension - and everyone's happy?). Working, correct code is the best argument possible. But some of the more recent comments since then would only be understandable, if his approach is ... simply wrong. I am confused. And that is why I asked for clarification - not for restating arguments from before his post, or inventing new ones. If there is already a solution in code, to answer the OP's question - then why argue whether such a solution should exist at all, or not? Makes no sense to me. So: Is his approach only slow/mem-hungry - or is it leading to incorrect results?

@karalabe

This comment has been minimized.

Show comment
Hide comment
@karalabe

karalabe Mar 27, 2016

Member

@altsheets Concurrent fast sync takes 11 mins on my machine, uses 2.2GB. The proposed solution needed 12h (!!), used 0.6GB extra data. Guess the feasibility ;)

On a serious note, a pruned node does not have the data you want. Isn't stored anywhere, so to get it, you need to crunch through and replay the entire Ethereum history, so fast sync is out of the question. Pruning is out of the question. It's not part of the consensus rules, so light clients are out of the question.

The only place where this can be implemented generically is for archive nodes, but those will be very rare. Average users won't have the data needed to index such transactions.

This is analogous to the real world: if I look at 5 dollars in my wallet, I don't expect the universe to tell me all the people that ever had those banknotes. No do I want a list with all the cash I've spent and received throughout my lifetime. I know how much I have now and I don't care who gave me how much when. Note, this is for cash. Now when I open a bank account and get a credit card, I expect to see all of my transaction history, but I don't get that for free, I pay my yearly maintenance fees to access that data. It's the same with Ethereum: the Ether is a plain account is cash: you know how much you have, you see the currently pending changes, but no history; whereas a contract can act as a bank account, just you get to choose all the rules of it. The more featured you wany, the more you need to pay the bank to get it, or in this case, you need to support the network with tx fees to maintain extra data on your behalf.

Just as a closing remark, the same way as it would be awesome to have automatic past tx history for plain accounts, it would be awesome to have tx data for my personal cash expenditure... But it's not something most pepole need, and it's not sometging that bothers you in real life. Simply assume that a plain account is cold hard cash, whereas a contract is a bank account working accordong to your rules.

Member

karalabe commented Mar 27, 2016

@altsheets Concurrent fast sync takes 11 mins on my machine, uses 2.2GB. The proposed solution needed 12h (!!), used 0.6GB extra data. Guess the feasibility ;)

On a serious note, a pruned node does not have the data you want. Isn't stored anywhere, so to get it, you need to crunch through and replay the entire Ethereum history, so fast sync is out of the question. Pruning is out of the question. It's not part of the consensus rules, so light clients are out of the question.

The only place where this can be implemented generically is for archive nodes, but those will be very rare. Average users won't have the data needed to index such transactions.

This is analogous to the real world: if I look at 5 dollars in my wallet, I don't expect the universe to tell me all the people that ever had those banknotes. No do I want a list with all the cash I've spent and received throughout my lifetime. I know how much I have now and I don't care who gave me how much when. Note, this is for cash. Now when I open a bank account and get a credit card, I expect to see all of my transaction history, but I don't get that for free, I pay my yearly maintenance fees to access that data. It's the same with Ethereum: the Ether is a plain account is cash: you know how much you have, you see the currently pending changes, but no history; whereas a contract can act as a bank account, just you get to choose all the rules of it. The more featured you wany, the more you need to pay the bank to get it, or in this case, you need to support the network with tx fees to maintain extra data on your behalf.

Just as a closing remark, the same way as it would be awesome to have automatic past tx history for plain accounts, it would be awesome to have tx data for my personal cash expenditure... But it's not something most pepole need, and it's not sometging that bothers you in real life. Simply assume that a plain account is cold hard cash, whereas a contract is a bank account working accordong to your rules.

@karalabe

This comment has been minimized.

Show comment
Hide comment
@karalabe

karalabe Mar 27, 2016

Member

Typed it on my phone, sorry for the many typos.

Member

karalabe commented Mar 27, 2016

Typed it on my phone, sorry for the many typos.

@dmxhZGp1c2hh

This comment has been minimized.

Show comment
Hide comment
@dmxhZGp1c2hh

dmxhZGp1c2hh Mar 27, 2016

@altsheets, it really make sense. The problem is that if you have bitcoin background it's kinda hard to implement same features the same way. For example my users have virtual balances and to transfer some amount to balance they use their own bitcoin address. It's easy to track, because each address belongs to some user. In ethereum I don't see same ways.

So creating account for each user is not an option. OK. I can create contract that will track transactions. But this way all my users will send ether to same contract address and how to distinguish who is who? Now I need to ask users to provide some additional data. Do I ask them to provide address, so I can track them later or to provide extra data in each transaction? Well if they use some 3rd party service mby they don't know exactly from which address ether will be send or don't have permissions to add extra data in transactions.

In bitcoin (or similar altcoin) you just create new address for every event you want to track and check later for address in transactions. Super easy and flexible.

@altsheets, it really make sense. The problem is that if you have bitcoin background it's kinda hard to implement same features the same way. For example my users have virtual balances and to transfer some amount to balance they use their own bitcoin address. It's easy to track, because each address belongs to some user. In ethereum I don't see same ways.

So creating account for each user is not an option. OK. I can create contract that will track transactions. But this way all my users will send ether to same contract address and how to distinguish who is who? Now I need to ask users to provide some additional data. Do I ask them to provide address, so I can track them later or to provide extra data in each transaction? Well if they use some 3rd party service mby they don't know exactly from which address ether will be send or don't have permissions to add extra data in transactions.

In bitcoin (or similar altcoin) you just create new address for every event you want to track and check later for address in transactions. Super easy and flexible.

@dmxhZGp1c2hh

This comment has been minimized.

Show comment
Hide comment
@dmxhZGp1c2hh

dmxhZGp1c2hh Mar 27, 2016

@altsheets, about implementation. listtransactions in bitcoin only returns related transaction for your wallet. So you don't need to save data about all address <-> transaction relations.

What about this logic. We have array of our account addresses (let's call it addresses). When node gets new transactions, it checks if sender or receiver is in our addresses array. If yes, we put txid of transaction in some cache table. Now we can use already implemented API to get info about transactions.

@altsheets, about implementation. listtransactions in bitcoin only returns related transaction for your wallet. So you don't need to save data about all address <-> transaction relations.

What about this logic. We have array of our account addresses (let's call it addresses). When node gets new transactions, it checks if sender or receiver is in our addresses array. If yes, we put txid of transaction in some cache table. Now we can use already implemented API to get info about transactions.

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Mar 28, 2016

Member

@altsheets as I said, I'm not arguing the usefulness of the feature I'm arguing the solutions that have either been suggested or created. If you read my response above you'll find the answer as to why this doesn't work, i.e. The results can not be guaranteed, for anything but full nodes. Since full nodes will not be dominating our network in the future makes this feature request extremely difficult if not at all possible to implement correctly (i.e. The results can never be verified).

Member

obscuren commented Mar 28, 2016

@altsheets as I said, I'm not arguing the usefulness of the feature I'm arguing the solutions that have either been suggested or created. If you read my response above you'll find the answer as to why this doesn't work, i.e. The results can not be guaranteed, for anything but full nodes. Since full nodes will not be dominating our network in the future makes this feature request extremely difficult if not at all possible to implement correctly (i.e. The results can never be verified).

@dmxhZGp1c2hh

This comment has been minimized.

Show comment
Hide comment
@dmxhZGp1c2hh

dmxhZGp1c2hh Mar 28, 2016

The results can not be guaranteed, for anything but full nodes.

Is that a problem? Full node will have some additional API for tracking, makes sense.

The results can not be guaranteed, for anything but full nodes.

Is that a problem? Full node will have some additional API for tracking, makes sense.

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Mar 28, 2016

Member

That would mean some dapps won't work with chains that were created with the fast sync option.

Mist will be our primary application to serve dapps and mist always uses the fast sync and therefore it will be impossible to support such a feature.

This feature won't work with light clients either and I expect light client to eventually dominate the network mostly.

I'm not going to make a promise but we could make such a feature available under, say, admin. This means you are free to use the feature but isn't part of standard web3 and will never be either. but please note that no dapp, including the wallet dapp or mist, will ever be able to make use of this. This would purely be considered a administrative utility.

Member

obscuren commented Mar 28, 2016

That would mean some dapps won't work with chains that were created with the fast sync option.

Mist will be our primary application to serve dapps and mist always uses the fast sync and therefore it will be impossible to support such a feature.

This feature won't work with light clients either and I expect light client to eventually dominate the network mostly.

I'm not going to make a promise but we could make such a feature available under, say, admin. This means you are free to use the feature but isn't part of standard web3 and will never be either. but please note that no dapp, including the wallet dapp or mist, will ever be able to make use of this. This would purely be considered a administrative utility.

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Mar 28, 2016

Member

For example an exchange could use this feature if it wants to be a full/archive node indefinitely.

Member

obscuren commented Mar 28, 2016

For example an exchange could use this feature if it wants to be a full/archive node indefinitely.

@fairglu

This comment has been minimized.

Show comment
Hide comment
@fairglu

fairglu Mar 29, 2016

I do not see why nodes that do not perform full validation are considered in this discussion.

Clients that have to trust other nodes will be intrinsically unsafe for services and dapps, and thus unusable, especially if/when full archive nodes are rare and sybil attacks have become easy. Any use case with real world consequences will need accounting and auditing capability, which means full history for a reasonable time frame (at least weeks, years in many cases). This means running full validating nodes (with data that can be trusted through blockchain trustless mechanisms and consensus) or... rely on "trusted" payment processors and web services to run those nodes for you.

Just as for other blockchains, even a light client is vastly more resource-intensive than relying on a web service ala blockchain.info or coinbase. And if a light client cannot provide full, validated information, it would be both less practical and less safe. There would be zero incentive in running your own light client if it cannot match the information provided by explorers like ether camp's full transparency.

So the problem may not be an easy one to solve, but it is a necessary cost to be paid for a light client to be useful in a wild, hostile world.

fairglu commented Mar 29, 2016

I do not see why nodes that do not perform full validation are considered in this discussion.

Clients that have to trust other nodes will be intrinsically unsafe for services and dapps, and thus unusable, especially if/when full archive nodes are rare and sybil attacks have become easy. Any use case with real world consequences will need accounting and auditing capability, which means full history for a reasonable time frame (at least weeks, years in many cases). This means running full validating nodes (with data that can be trusted through blockchain trustless mechanisms and consensus) or... rely on "trusted" payment processors and web services to run those nodes for you.

Just as for other blockchains, even a light client is vastly more resource-intensive than relying on a web service ala blockchain.info or coinbase. And if a light client cannot provide full, validated information, it would be both less practical and less safe. There would be zero incentive in running your own light client if it cannot match the information provided by explorers like ether camp's full transparency.

So the problem may not be an easy one to solve, but it is a necessary cost to be paid for a light client to be useful in a wild, hostile world.

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Mar 29, 2016

Member

@fairglu all of the current functionality that is supported by web3 (blocks, receipts, logs, etc) can be verified by a light client. We must consider light clients in this discussion because:

  1. light clients shouldn't be any less than archive nodes in terms of web3 functionality;
  2. light client and nodes synced using --fast should not be considered sub-sets of web3.

Transactions that come from the call eth_listTransactions can not be verified without having the full chain at your disposal and even then it becomes incredible impractical to validate the transactions as you'd need to parse the entire chain and figure which transactions are valid and which aren't.

The problem is however solvable by using the Ethereum system as intended as I've only pointed out so many times. If you're looking for a simple value transfer network with said functionality your best option is the Bitcoin network.

In any case DApps that rely on transaction tracking are very likely doing it wrong and I'm sure there are more than enough people willing to help solve whatever issue it you're having, including myself. You're always welcome to join our gitter to discuss your issue you're having :-)

Member

obscuren commented Mar 29, 2016

@fairglu all of the current functionality that is supported by web3 (blocks, receipts, logs, etc) can be verified by a light client. We must consider light clients in this discussion because:

  1. light clients shouldn't be any less than archive nodes in terms of web3 functionality;
  2. light client and nodes synced using --fast should not be considered sub-sets of web3.

Transactions that come from the call eth_listTransactions can not be verified without having the full chain at your disposal and even then it becomes incredible impractical to validate the transactions as you'd need to parse the entire chain and figure which transactions are valid and which aren't.

The problem is however solvable by using the Ethereum system as intended as I've only pointed out so many times. If you're looking for a simple value transfer network with said functionality your best option is the Bitcoin network.

In any case DApps that rely on transaction tracking are very likely doing it wrong and I'm sure there are more than enough people willing to help solve whatever issue it you're having, including myself. You're always welcome to join our gitter to discuss your issue you're having :-)

@obscuren

This comment has been minimized.

Show comment
Hide comment
@obscuren

obscuren Mar 29, 2016

Member

Locking this issue but leaving it open.

We're considering a suitable implementation for this feature.

No ETA.

Member

obscuren commented Mar 29, 2016

Locking this issue but leaving it open.

We're considering a suitable implementation for this feature.

No ETA.

@ethereum ethereum locked and limited conversation to collaborators Mar 29, 2016

@karalabe karalabe closed this Feb 12, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.