Generated transaction IDs are not unique

[OlegMazurov]

Description

I generate many transactions in parallel from the same account (it's a setup stage in a benchmark built on top of the Java Client SDK; the transactions are AccountCreateTransaction). Occasionally, the same ID is generated for two different transactions. If those transactions are submitted to the same node I get the DUPLICATE_TRANSACTION error for the last one. If, however, they are submitted to different nodes, it becomes possible to get a receipt for a different transaction that is accepted as valid. In particular, AccountCreateTransaction receives a wrong account ID that can not be used with the key provided with the transaction.
The problem is with the TransactionId.generate() method:

    public static TransactionId generate(AccountId accountId) {
        Instant instant = Clock.systemUTC().instant().minusNanos((long) (Math.random() * 5000000000L + 8000000000L));
        return new TransactionId(accountId, instant);
    }

The randomization works OK at a low speed but at a high TPS (up to 25K in my case) the probability of generating the same value becomes significant. A strictly monotonic clock source is required here. Unfortunately, existing Java clock sources do not provide such guarantee in multithreaded environment.

Thanks to the possibility of setting my own transaction ID, I have been able to implement the following scheme that works reliably in my benchmark:

    private final long adjustment = System.currentTimeMillis() * 1_000_000L - System.nanoTime();
    private final AtomicLong monotonicTime = new AtomicLong();

    private TransactionId generateTxnId(Client client) {
        long currentTime;
        long lastTime;
        do {
            currentTime = System.nanoTime() + adjustment;
            lastTime = monotonicTime.get();
            if (currentTime <= lastTime) currentTime = lastTime + 1;
        } while (!monotonicTime.compareAndSet(lastTime, currentTime));
        return new TransactionId(client.getOperatorAccountId(), Instant.ofEpochSecond(0, currentTime));
    }

Something along those lines should be implemented in Client SDKs.

Steps to reproduce

I used Load Generator to run CryptoTransferLoadTest against a 7-node network.

Additional context

No response

Hedera network

other

Version

v2.24.0

Operating system

Linux

[thenswan]

Hi @OlegMazurov, and thank you for raising this issue. To give you an update -- it is currently in progress :)

[thenswan]

Hi @OlegMazurov, would like to request more info from your side:

• How did you run the steps above in terms of the infrastructure (what setup did you use for both hedera nodes instances and load generator app)?
• What configuration exactly did you use (command prompt) for running Load Generator?
• Where did you see the error output re DUPLICATE_TRANSACTION exactly (logs from the hedera node or from the load generator)?

[OlegMazurov]

• Single node running Hedera app. I enable the following configuration parameters but that is probably not necessary to reproduce the bug:

records.useConsolidatedFcq=true
accounts.storeOnDisk=false
tokens.storeRelsOnDisk=false
tokens.nfts.useVirtualMerkle=false

Here is the command line:

java -XX:+UseZGC -Xms32g -Xmx150g -Xlog:gc*:gc.log -XX:ConcGCThreads=14 -XX:MaxDirectMemorySize=32g -XX:MetaspaceSize=100M -cp data/lib/* com.swirlds.platform.Browser

• To run load generator on the same node:

java -Dorg.slf4j.simpleLogger.defaultLogLevel=debug -cp network-load-generator-0.0.1.jar com.hedera.benchmark.CryptoTransferLoadTest -c 16 -a 10000000 -t 3600

• The error is reported by the client during the account creation phase:

15:50:36:347 [main] INFO com.hedera.benchmark.LoadTest - Created 16 clients, starting workload
15:50:44:717 [hedera-sdk-7] INFO com.hedera.benchmark.WorkingQueue - Transactions: 100000  TPS(EMA): 11950  TPS(current): 11950 
15:50:44:718 [hedera-sdk-7] INFO com.hedera.benchmark.WorkingQueue - Transactions: UPDATE TPS 20000 -> 18000
15:50:50:259 [hedera-sdk-28] INFO com.hedera.benchmark.WorkingQueue - Transactions: 200000  TPS(EMA): 14993  TPS(current): 18037  DUPLICATE_TRANSACTION:3
...

Note that the current main branch (commit 8a4e1df2) does not contain my fix/workaround for this issue. It is only in the network-mode branch, which I'm actively working on and which may be integrated at any moment.

[iron4548]

Unsure if this one is related but even since I installed 2.29.0 I've been running into this error from time to time:

Hedera transaction 0.0.xxxxx@xxxxxxxxxxxxxxxx.123000000 failed pre-check with the status INVALID_TRANSACTION_START with all kind of transactions (like ContractExecuteTransaction, TokenAssociateTransaction etc.). My time is sync'd within 1 second. I'm on a Windows PC.

Note that there are only 3 decimal places in the nanosecond part, and the rest is zero. This seems unusual.

I rolled back to 2.28.0 and my code worked without issues.

[OlegMazurov]

See also hashgraph/hedera-services#7959

[iron4548]

See also hashgraph/hedera-services#7959

Thanks, I had a look but it's not clear on what I need to do. It appears to be resolved in v0.42 - is that for com.hedera.hashgraph:hedera-protobuf-java-api:0.42.0 ? I'm not including that one in my build.gradle. I was only using "com.hedera.hashgraph:sdk:2.29.0"