Skip to content

tep-1007.adoc

Latest commit

 

History

History
366 lines (249 loc) · 11.1 KB

tep-1007.adoc

File metadata and controls

366 lines (249 loc) · 11.1 KB
Raw

TEP-1007: Txn Set Checksum

Txn set checksum and report account selector checksum are verification and auditing tools to ensure that generated reports are using correct and known inputs.

Summary

For overal information, see Accounting Auditing and example reports listed below:

Transaction set checksum

Transaction set checksum (txn set checksum) is secure and cryptographically strong hash and proof of presence of transactions which are used to do accounting calculations.

Txn set checksum is based on UUID of Txn, and it is hash function’s result of those UUIDs. Hash is calculated so that it is possible to verify it easily with external tools. See Journal Format and description of Transaction metadata.

Txn set checksum is calculated based on following algorithm:

  • Txn’s UUIDs are collected as canonical 8-4-4-4-12 lower case hexadecimal strings.

  • Sorted list of UUID stings is fed to hash function so that each UUID is separated with single newline ('\n', ASCII 0x0A) character.

  • Resulting hash function hash is Txn Set Checksum

See contrib for example script (txn-set-checksum.sh) how to calculate that with standard UNIX command line tools.

Assumptions and Restrictions

Each transaction must have unique UUID. Presence of UUID and uniqueness are enforced.

Txn set checksum verifies that same set of transactions (UUIDs) is used for calculations, but it does not verify actual content of used transactions. If and when content verification is needed, then Tackler’s git storage backend could be used for that. In that case, content verification is done by combining reported git’s commit id and txn set checksum. Together commit id and txn set checksum creates tamper proof verification of used transaction data.

To re-verify previous tackler report, it is possible to use git commit id (from old report) as an input reference. See Usage Guide and Git Storage Guide for further info.

Report account selector checksum

Report account selectors core.reporting.accounts and core.reports.<report-type>.accounts are used to select which accounts are included into reports. Selected accounts affect calculated and reported totals.

Report account selector checksum is secure and cryptographically strong hash of account selector, e.g. it can be used as fingerprint of used account selector.

Account selector checksum is calculated based on following algorithm:

  • Sorted list of account selectors is fed to hash function (e.g. SHA-256) so that each selector is separated with single newline ('\n', ASCII 0x0A) character.

  • Resulting hash is Account Selector Checksum

Journal file format

No changes to journal format are needed.

Implementation

CLI changes

No changes to CLI args are needed.

CONF changes

New configuration group for core:

If txn-set-checksum is true, then all transactions must have valid :uuid:-metadata field. See Journal Format for further info.

Machinery

Following changes are needed

  • ✓ enforce uuid

    • ✓ Verify presence of UUID at parse time

    • ✓ Hard error if UUID is missing during txn set checksum calculation
      This is either internal error or logic error within usage of Server API

  • ✓ calculate txn set checksum

  • ✓ calculate account selector checksum

Tackler supports hash functions provided by Java platform, most notably SHA-256 …​ SHA-512 algorithms on Java 8, and SHA3-256 …​ SHA3-512 on Java 11+.

See MessageDigest on:

POC implementation

Txn set checksum with external tools

(
    find "$1"  -type f -name '*.txn' | \
    xargs -n100 grep -h ';:uuid:'
) | \
    sed -E 's/[[:space:]]+;:uuid:[[:space:]]+([a-fA-F0-9-]+)[[:space:]]*/\1/' | \
    tr 'A-F' 'a-f' | \
    sort | \
    sha256sum
          val txns = rawTxns.flatten.sorted(OrderByTxn)

          val txnHash = txns.map(_.header.uuid match {
              case Some(uuid) => uuid.toString
              case None => throw new TacklerException("missing uuid")
            })
            .sorted
            .foldLeft(MessageDigest.getInstance("SHA-256"))({
              case (hash, uuid) => {
                hash.update((uuid + "\n").getBytes("UTF-8"))
                hash
              }
            }).digest()

          def hex2str(hash: Array[Byte]) = {
            hash.map(b => "%02x".format(0xff & b)).mkString
          }

API changes

Api changes to server or client interfaces.

Server API changes

Changes to server API

  • ✓ Txn set checksum data and mechanism to TxnData

  • ✓ Report account selector checksum

Client API changes

Changes to client API or JSON model

  • ✓ Txn set checksum to Metadata

  • ✓ Report account selector checksum to Metadata

New dependencies

No new dependencies

Reporting

Changes to reports or reporting

Balance report

Changes to balance report

  • ✓ txn set checksum

    • ✓ text

    • ✓ json

  • ✓ account selector checksum

    • ✓ text

    • ✓ json

Balance Group report

Changes to balance group report

  • ✓ txn set checksum

    • ✓ text

    • ✓ json

  • ✓ account selector checksum

    • ✓ text

    • ✓ json

Register report

Changes to register report

  • ✓ txn set checksum

    • ✓ text

    • ✓ json

  • ✓ account selector checksum

    • ✓ text

    • ✓ json

Exporting

Changes to exports or exporting

Equity export

Changes to equity export

  • ✓ Audit / verification material to equity export?

    • ✓ General metadata (e.g. Git metadata)

    • ✓ txn set checksum

    • ✓ account selector checksum

      • ✓ Empty selector, e.g. "select all"

      • ✓ Active selector

Identity export

Changes to identity export

  • ✓ no changes

Documentation

Future plans and Postponed (PP) features

There are several possibilities to enhance txn set checksum:

  • Option to turn off uuid duplicate detection

  • Support SHA-3, this should be possible by changing JDK version: http://openjdk.java.net/jeps/287

    • ✓ Make this configurable

  • External listing which includes all used transaction UUIDs

  • There could be a separate, actual content hash which is calculated over some normalization of Txn data.

Postponed (PP) features

Anything which wasn’t implemented?

Tests

Normal, ok-case tests to validate functionality:

  • ✓ test basic txn set checksum

    • ✓ test audit staff alone, without git

  • ✓ different hash algorithms

    • ✓ test configuration settings of different hash algorithm

    • ✓ reporting with different hash algorithm

  • ✓ reports

    • ✓ { balance, balance-group, register } x { text, json }

  • ✓ Account selector checksum

    • ✓ None (All pass)

      • ✓ { balance, balance-group, register } x { text, json }

      • ✓ { equity } x { txn }

    • ✓ All have same global selector

      • ✓ { balance, balance-group, register } x { text, json }

      • ✓ { equity } x { txn }

    • ✓ Each report has own selector, global is set

      • ✓ balance

      • ✓ balane-group

      • ✓ register

      • ✓ equity

  • ✓ exports

    • ✓ test equity

      • ✓ test case with all metadata (txn-set-checksum, git-storage, filters, account-selector-hash)

      • ✓ feed generated equity back (e.g. check validity of format)

  • ✓ test that upper case UUIDs result same txn-set-checksum as lower case UUIDs

  • ✓ test that filtered Txns has correct (new) txn set checksum

  • ✓ test case with all metadata (txn-set-checksum, git-storage, filters, account-selector-hash)

  • ✓ metadata in case that there are no matching accounts to be reported

  • ✓ Filter Txns multiple times, check correct metadata / txn-set-checksum

Errors

Various error cases:

// valid
scala> java.util.UUID.fromString("69439222-4d8b-4d0e-8204-50e2a0c8b664")
res1: java.util.UUID = 69439222-4d8b-4d0e-8204-50e2a0c8b664

// invalid
scala> java.util.UUID.fromString("694aaaaa39222-4d8b-4d0e-8204-50e2a0c8b664")
res2: java.util.UUID = aaa39222-4d8b-4d0e-8204-50e2a0c8b664

  • ✓ e: invalid hash type

Perf

Add new perf test target for txn set checksum

  • ❏ perf test of txn set checksum

    • ❏ with txn set checksum

    • ❏ without txn set checksum

Feature and Test case tracking

Tackler is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.