Further optimize blockchain transaction inserts to DB

[peterbroadhurst]

In PR chain with #1343

• Creates a new txwriter package that depends on both operations and txcommon to do batched inserts
  • Dispatches to a pool of background workers
  • Assigns a new ID to each transaction in the batch
  • Splits to transactions with/without idempotencyKey set
  • Attempts to insert all with new InsertTransactions multi-insert function in DB layer
  • Handles partial results on the idempotencyKey set TX, by querying those idempotency keys
  • Compares ID of the transactions in the DB, with the inserts, to find the duplicates
  • Inserts the operations with a new multi-insert function in the DB layer (these always have new IDs)
• Updates contracts package to use ^^^ instead of one-by-one insertion of tx+ops
  • The inbound thread for invoke/deploy requests doesn't even have a DB transaction now (RunAsGroup call removed)
  • Creation of the core.Operation moved up the function, as need to be passed in
  • Idempotency handling to resubmit operations in the case of a clash, unchanged
• Updates blockchain connector interface to split parsing with execution
  • We were compiling the FFI schema twice per transaction in the blockchain connector
• Adds caching to contracts package
  • For requests referencing an FFI: FFI ID + methodPath -> FFIMethod + []*FFIError
  • For all requests: Hash of compiled JSON Schema, to the JSON Schema object
  • Also builds a hash-of-hashes across the combination of method+errors, which is used to cache the result from the blockchain connector validation

[codecov-commenter]

Codecov Report

Merging #1354 (e40ec89) into main (f286d82) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff            @@
##             main    #1354    +/-   ##
========================================
  Coverage   99.98%   99.98%            
========================================
  Files         314      315     +1     
  Lines       21581    21902   +321     
========================================
+ Hits        21577    21898   +321     
  Misses          2        2            
  Partials        2        2            

Impacted Files	Coverage Δ
internal/data/message_writer.go	100.00% <ø> (ø)
internal/assets/token_approval.go	100.00% <100.00%> (ø)
internal/assets/token_pool.go	100.00% <100.00%> (ø)
internal/assets/token_transfer.go	100.00% <100.00%> (ø)
internal/blockchain/ethereum/ethereum.go	100.00% <100.00%> (ø)
internal/blockchain/fabric/fabric.go	100.00% <100.00%> (ø)
internal/broadcast/manager.go	100.00% <100.00%> (ø)
internal/contracts/manager.go	100.00% <100.00%> (ø)
internal/contracts/operations.go	100.00% <100.00%> (ø)
internal/coreconfig/coreconfig.go	100.00% <100.00%> (ø)
... and 7 more

[nguyer]

@peterbroadhurst It does look like we have a coverage miss here: https://app.codecov.io/gh/hyperledger/firefly/pull/1354/blob/internal/operations/context.go

[nguyer]

Do we know for sure that this is a resubmitted operation at this point? Or is it possible that we would still hit this in return in a case where the user re-used an idempotency key and we should have rejected it earlier?

[peterbroadhurst]

Great question... digging

[peterbroadhurst]

ok - I've worked through the code, and did find (existing I think in all cases) issues in how the TXID was being propagated, and a weirdness on an interface that returned a single operation even though there might be more.

Commit added e40ec89