eth_getTransactionCount does not include pending transactions when 'pending' qualifier given

[drewshaver]

As the title states, tx count is not getting incremented for pending transactions when giving second argument of 'pending'. It seems to be treated the same as 'latest'.

System information

Geth version: 1.4.10
OS & Version: OSX

> web3.eth.getTransactionCount("0xf82e...", "pending");
5
> web3.eth.sendTransaction({from: "0xf82...", to: "0xf1c...", value: 42000000000000000});
I0624 15:57:41.848826 eth/api.go:1193] Tx(0xbd094a59eb8f05653f35fa93a9254db95bc6b9b5bdd3b95aedda27bb781545f9) to: 0xf1c...
"0xbd094a59eb8f05653f35fa93a9254db95bc6b9b5bdd3b95aedda27bb781545f9"
> web3.eth.getTransactionCount("0xf82...", "pending");
5 (should be 6)
> I0624 15:57:59.315075 miner/worker.go:337] 🔨  Mined block (#4529 / 7788e873). Wait 5 blocks for confirmation
I0624 15:57:59.315639 miner/worker.go:555] commit new work on block 4530 with 1 txs & 0 uncles. Took 529.458µs
I0624 15:57:59.315664 miner/worker.go:433] 🔨 🔗  Mined 5 blocks back: block #4524
I0624 15:57:59.315989 miner/worker.go:555] commit new work on block 4530 with 1 txs & 0 uncles. Took 310.783µs
> web3.eth.getTransactionCount("0xf82...", "pending");
6

[fjl]

This is critical for the upcoming Go API as well. Maybe getTransactionCount(..., "pending") should return the transaction pool's latest known nonce instead of looking at the pending block.

[bas-vk]

Are you sure you used geth 1.4.10?

> eth.getTransactionCount(eth.accounts[0])
5
> eth.getTransactionCount(eth.accounts[0], "pending")
5
> eth.sendTransaction({from: eth.accounts[0], to: eth.accounts[1], value: 1})
"0xd3546397a3b0c1fb0a6c47364a66b1841f4ebffb15f58bb5c8d0e1f453108e59"
> eth.getTransactionCount(eth.accounts[0])
5
> eth.getTransactionCount(eth.accounts[0], "pending")
6
> miner.start(1)
true
... tx mined ...
> eth.getTransactionCount(eth.accounts[0])
6
> eth.getTransactionCount(eth.accounts[0], "pending")
6

Geth will grab the nonce from the pending state if you ask for it.

[murrekatt]

Same issue here. Looking at the code in eth/api.go here there is no mention of pending.

[bas-vk]

stateAndBlockByNumber is called which gets the pending state from the miner.

[fjl]

We will change it to look at the transaction pool because it knows about the latest nonce faster.

[murrekatt]

@fjl that sounds very good.

I'm testing out the Go bindings and in the example here there seems to be some races to be aware of due to the sleep:

address, tx, token, err := DeployToken(auth, backends.NewRPCBackend(conn), new(big.Int), "Contracts in Go!!!", 0, "Go!")
if err != nil {
    log.Fatalf("Failed to deploy new token contract: %v", err)
}
fmt.Printf("Contract pending deploy: 0x%x\n", address)
fmt.Printf("Transaction waiting to be mined: 0x%x\n\n", tx.Hash())

// Don't even wait, check its presence in the local pending state
time.Sleep(250 * time.Millisecond) // Allow it to be processed by the local node :P

name, err := token.Name(&bind.CallOpts{Pending: true})

I'm doing two transactions from the same account after each other and right now the nonce is the same for both which means once tx will be invalid once the other is mined.

How will the concurrency work after the change to use the transaction pool? Will multiple transactions from the same account increment the nonce without any tricks with sleep etc.?

Also when is this change planned to hit develop?

[fjl]

I cannot give you a timeline but it will be included in the next major release.

[murrekatt]

OK, thanks @fjl.

I think the Go bindings are really cool and was also wondering a bit how they were meant to be used in various situations e.g.:

1. single account calling multiple functions on a single contract (resulting in transactions)
2. single account calling multiple contracts (resulting in transactions)

In addition, the two above could be done concurrently from e.g. different goroutines. As the underlying mechanism from the calling go code right now calls out to the backend over RPC to get the transaction count or later the tx pool nonce there is a race. Hence, I assume concurrent transactions for the same account is not supported. Right?

However, for 1 & 2 it would be possible to keep track of the nonce in the client code, maybe in the TransactOpts by the underlying backend? Or something along those lines so that a caller using the same account would automatically get the nonce incremented without a need to ask over RPC. This opens of course up other points such as keeping the local value in sync. Has this been thought about? I'd be interested to understand what's been considered.

[Nexus7]

@fjl : Just checking if this change to look at the transaction pool for the nonce was made yet as obviously this issue is still currently open here...

[nickluijtgaarden]

I am seeing that this issue is still open. Does this mean this is not fixed?

[doppio]

@fjl any update on this? Is this change still going to happen?

[mcdee]

Any progress on this? It seems like a critical issue for those of us attempting to generate multiple transactions with Go, and still has a milestone tag of two major releases back.

[chatch]

There is a related issue in Parity (openethereum/parity-ethereum#5014) with "pending". Parity has a non-standard RPC call parity_nextNonce to get the nonce considering all pending transactions in the pool.

Maybe this could be added to the standard JSONRPC API. Something like eth_nextNonce. Or the suggestion on the issue of using "pending_mining" or "pending_inclusion" could work too.

Then this getTransactionCount with "pending" can remain as meaning "the currently mined block (including pending transactions)". ie. pending transactions in a block being mined.

[yondonfu]

At the moment I'm doing some local testing with this change to GetTransactionCount in internal/ethapi/api.go to ask the transaction pool for the nonce if the the transaction count is asked for a pending block:

// GetTransactionCount returns the number of transactions the given address has sent for the given block number
func (s *PublicTransactionPoolAPI) GetTransactionCount(ctx context.Context, address common.Address, blockNr rpc.BlockNumber) (*hexutil.Uint64, error) {
	if blockNr == rpc.PendingBlockNumber {
		nonce, err := s.b.GetPoolNonce(ctx, address)
		if err != nil {
			return nil, err
		}

		return (*hexutil.Uint64)(&nonce), nil
	} else {
		state, _, err := s.b.StateAndHeaderByNumber(ctx, blockNr)
		if state == nil || err != nil {
			return nil, err
		}

		nonce := state.GetNonce(address)

		return (*hexutil.Uint64)(&nonce), nil
	}
}

Thoughts on if this is a reasonable solution?

This issue seems to still be a problem for anyone using abigen to submit concurrent transactions since transact in https://github.com/ethereum/go-ethereum/blob/master/accounts/abi/bind/base.go#L176 uses PendingNonceAt to retrieve the pending nonce which I believe uses GetTransactionCount

[davux]

@fjl (2016-08-06)

I cannot give you a timeline but it will be included in the next major release.

When is the next major release? It's been a year and a half and the bug is serious and still present.

[dshulyak]

@yondonfu this is what @fjl suggested some time ago in this thread. In my opinion, you should submit PR with your change

[winteraz]

Any chance to have this fixed this year(2018) ?

[vernon99]

I wonder, how can it still open? How people deal with a need to send multiple transactions from the same account without storing the nonce (super fragile and can lead to transaction failures or overrides)? Do I miss something or this is a must have for any normal project?

[gavinmarkglynn]

This seems like to be too serious of bug to be left hanging? Is there any possibility of a fix? This issue will cause stability problems for any general application using Ethereum smart contracts.

[vernon99]

Btw, for now it seems we found a workaround by accessing node's pool using txpool api, fetching pending transactions and adding the number to eth_getTransactionCount. But yeah, this is a super crappy hack since you have to do extra fetch of a massive size on any transaction. Should be fixed on a node level!

[drewshaver]

@vernon99 just fyi, there's a potential race condition in that workaround if the node state changes between calls. (i think)

[fjl]

(Finally) fixed in #15794.