Internal error wheh Coinbase makes 2 withdrawals in same tx

[splix]

When importing data from Coinbase REST API I have the following error:

ERROR: Fatal exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/dali/dali_main.py", line 181, in _dali_main_internal
    resolved_transactions: List[AbstractTransaction] = resolve_transactions(transactions, dali_configuration, args.read_spot_price_from_web)
  File "/usr/local/lib/python3.10/site-packages/dali/transaction_resolver.py", line 287, in resolve_transactions
    transaction = _resolve_intra_intra_transaction(transaction1, transaction2, None)
  File "/usr/local/lib/python3.10/site-packages/dali/transaction_resolver.py", line 499, in _resolve_intra_intra_transaction
    to_exchange: str = _resolve_fields(
  File "/usr/local/lib/python3.10/site-packages/dali/transaction_resolver.py", line 58, in _resolve_fields
    raise RP2RuntimeError(f"Internal error: {name1} and {name2} are both unknown: {transaction1}\n{transaction2}")
rp2.rp2_error.RP2RuntimeError: Internal error: transaction1.to_exchange and transaction2.to_exchange are both unknown: IntraTransaction:

The reason is that there are two different BTC withdrawals that were made through a single onchain transaction. I.e., both operations have the same transaction hash which is used as unique_id, but different to addresses.

[splix]

It seems that the original data (which I see as a raw_data field) has a field called id which is a UUID. It has different values for those transactions, and to my understanding this is a Unique Id assigned by Coinbase. All other fields (including to) could be equal.

[eprbell]

Can you run with debug logs enabled? Logs are stored in the log directory. To generate debug logs, prepend the command line with LOG_LEVEL=DEBUG, e.g.:

LOG_LEVEL=DEBUG dali_us -s -o output/ config/test_config.ini

Then open the log and look for occurrences of the unique_id: it should give us clues as to what is happening. If you can, post here the log entries with the problematic unique_id, however before posting here any personal data please read this and feel free to sanitize your logs to skip sensitive info.

[splix]

unique_id is a transaction hash for those withdrawals.

But both withdrawals are in the same Bitcoin transactions, therefore it's the same string value.

[eprbell]

I would need to see the logs to understand what is happening. Could you post the lines with the problematic unique id from the log?

[splix]

Here is the error details from the log:

rp2.rp2_error.RP2RuntimeError: Internal error: transaction1.to_exchange and transaction2.to_exchange are both unknown: IntraTransaction:
  plugin=Coinbase
  unique_id=10515c4a87a1abd5036a8ab946ac21f96a756f7ddb2446f194757782058f331a
  raw_data={"id": "d3edbfd0-0478-4946-ae46-8bb7aeaa15b9", 
		"type": "send", 
		"status": 
		"completed", 
		"amount": {"amount": "-0.10000123", "currency": "BTC"}, 
		"hash": "10515c4a87a1abd5036a8ab946ac21f96a756f7ddb2446f194757782058f331a", 
		...}
  timestamp=2021-05-08 02:12:47+0000
  asset=BTC
  from_exchange=Coinbase
  from_holder=ME
  to_exchange=__unknown
  to_holder=__unknown
  spot_price=61235.00367021319696278636396
  crypto_sent=0.10000123
  crypto_received=__unknown
  notes=None
IntraTransaction:
  plugin=Coinbase
  unique_id=10515c4a87a1abd5036a8ab946ac21f96a756f7ddb2446f194757782058f331a
  raw_data={"id": "8417d09a-46ed-43c3-9507-95bc05278005",
		"type": "send", 
		"status": 
		"completed", 
		"amount": {"amount": "-0.10000123", "currency": "BTC"}, 
		"hash": "10515c4a87a1abd5036a8ab946ac21f96a756f7ddb2446f194757782058f331a", 
		...}
  timestamp=2021-05-08 02:12:47+0000
  asset=BTC
  from_exchange=Coinbase
  from_holder=ME
  to_exchange=__unknown
  to_holder=__unknown
  spot_price=61235.00367021319696278636396
  crypto_sent=0.10000123
  crypto_received=__unknown
  notes=None

[eprbell]

Interesting: the two transactions have different ids (so they are treated by the Coinbase plugin as two different transactions) but the same hash/unique id (so the transaction resolver thinks they are two halves of the same transaction).

I have a few follow-up questions:

• do you remember any details about this transaction? I.e. where was it from / to? Was it a Coinbase to Coinbase transfer? Anything else that makes it stand out?
• was it by any chance a Coinbase Pro transaction? Sometimes there is duplication of information between Coinbase and Coinbase Pro.
• could you paste the full raw data for both transactions? That may contain some clues.

Thanks.

[splix]

Right, it's two parts of the same transaction.

Both are standard withdrawals to two different on-chain non-coinbase addresses. Just made with a single on-chain transaction, that's why they share the same hash.

It was made with Coinbase Pro, but I don't use this plugin. It's available with Coinbase API, so it's the only plugin configured currently.

[eprbell]

I'm not sure I understand: how do you make two withdrawals to two different on-chain addresses using one single on-chain transaction (same hash) in Coinbase/Coinbase Pro? This sounds like functionality that is not implemented in the plugin (I wasn't even aware you could do it in CB).

[splix]

You don't need to do anything special. Just make a withdrawals via a standard Coinbase UI, then make another one in a minute or less. Most likely it will be the same outgoing transaction on-chain.

Since Coinbase aggregates multiple withdrawals for different customers over a period of a few minutes into one transaction it also means that your multiple withdrawals comes to the same transaction as well.

I mean, It's a standard functionality of Bitcoin protocol. if you look at any withdrawal from Coinbase, it's always multiple outputs. And the same is possible with Ethereum with new Coinbase withdrawal which now they make via a smart contract.

[eprbell]

I'm aware that BTC transactions can have multiple inputs and outputs at the protocol level, I just haven't seen it used in Coinbase / CBPro before (i.e. I haven't seen two outgoing CB transfers with the same hash before). We may have to add the output index to the transaction hash as the unique id: is that information available in the raw data? If you can paste the raw data for the two transactions (sanitized of personal info) we can start thinking on how to best solve this.

[splix]

I have multiple of them just in my withdrawal history. I guess an average person doesn't make a few withdrawals in a row, so it doesn't happen too often.

Coinbase JSON doesn't include txout unfortunately. Most of the other fields could technically be same (like recipient address and amount). There is created_at timestamp field, which is most likely be different because you don't create two withdrawals in exactly same second.

So the only fields that are unique in that JSON are id and idem, both are UUID.

[eprbell]

If we don't have the index I'm not sure how to fix this: how can the transaction resolver know which of the two same-hash outgoing Coinbase transactions to map to each of the two destinations? The created_at field doesn't help because it doesn't carry information to tie together source and destination. I will think on this some more, but I don't see a simple solution to this issue.

[eprbell]

I thought some more about this issue and I want to summarize the problem and describe it precisely from the perspective of DaLI internals.

Let's consider the example reported in this issue:

• two outgoing transactions from Coinbase Pro to two external addresses (let's call them HWA and HWB);
• both transactions are in the amount of 0.1 BTC (the amount is rounded here for simplicity);
• both transactions share the same hash (let's call it H) because at the protocol level they are two outputs of the same transaction (this is the crux of the problem);
• let's say HWA receives 0.099 (0.001 is the fee). This transaction is also marked with the same hash H;
• let's say HWB receives 0.098 (0.002 is the fee). This transaction is also marked with the same hash H.

Now let's look at how DaLI sees the issue:

• the transaction resolver uses the hash to put put together (to "resolve") two ends of an on-chain transaction;
• individual plugins (e.g. the Coinbase Pro plugin) don't know where an external transfer goes to, so they return an incomplete transaction with the hash as unique id: the transaction resolver will try to match it, after all plugins have read their data. This assumes that each couple of incomplete transactions shares the same hash and that there is no more than one source and one destination;
• as we've seen here, the assumption is not valid and this is the issue.

How could the issue be resolved? The problem is that some information is missing: if we had the output index together with the hash, on Coinbase Pro, on HWA and on HWB, we could use (hash, out index) as the unique id, instead of just hash. This would make the transaction resolver happy and we could fix the problem this way. But without this information how can the transaction resolver match source to destination? One idea: perhaps we could try to gather the missing information directly from the blockchain.

[splix]

Right, there is not transaction index and it doesn't seem to be possible to recover it in any way. Also, not only amounts but also the target address could be the same, so there is nothing to distinguish them.

Btw, I think the fees should be considered as an equal amount in this case. Technically, it's the proportional part of the byte size for that particular output, and in this case only a type of address could affect it. I.e., different fees for P2PKH and P2SH. But economically, I think, Coinbase considers them as equal output splitting the fees proportionally.

What I'm probably missing is why you don't want to use the id provided with the Coinbase JSON? It's different for each withdrawal, even for withdrawals with the same hash.

[splix]

Ah, the JSON actual has a field for the fee: "network": { "transaction_fee": {"amount": "0.00001234", "currency": "BTC"} }. I though you mean that it needs to be calculated

[eprbell]

The problem with id is that it's Coinbase-specific and the other side of the transaction doesn't know about it, so it can't be used to define a global name space for matching. Would you be able to share the full raw_data (after sanitizing it of personal information)? Sometimes coinbase uses multiple JSON structures to define a transaction and the main structure points to another via an id (e.g. transfer), so I wonder if the same could apply here. Another thought: could you try with the Coinbase Pro plugin? It'd be interesting to see the raw_data for it as well.