Skip to content

Consolidate duplicated base64 primitives (six-crate-merge legacy) — prerequisite for P1.2 #56

Description

@joeldsouzax

Summary

The base64 primitives are duplicated across modules — a legacy artifact of the six-crate→one-crate merge (700f5fb, "copy CESR/KERI sources from agency"). Each original crate (cesr-utils, cesr-core, cesr-stream, keri-core) carried its own copy; post-merge they were never deduplicated. This blocks P1.2 (#29): optimizing base64 is premature while there are 3–5 copies to optimize.

This is duplication, not a correctness bug — every copy uses the identical URL-safe alphabet and passes its own keripy vectors (gate is green). The risk is maintenance drift and bug-surface, not current wrongness.

What is genuinely duplicated (keripy has ONE of each)

Integer ↔ B64 (keripy intToB64/b64ToInt) — 3 copies:

  • utils/encode.rs::encode_int + utils/decode.rs::decode_to_int (generic PrimInt)
  • stream/util.rs::int_to_b64 + b64_to_int (u64, own alphabet + own char match)
  • core/indexer/indexer.rs::int_to_b64 (u32, own alphabet)

Consumers are inconsistent: stream/unwrap.rs + indexer/builder.rs already call crate::utils::*, but stream/message.rs + indexer/indexer.rs call the local copies.

URL-safe alphabet / reverse table5 copies: utils/utils.rs (B64_URL_CHARS+B64_DECODE_INDEX), stream/util.rs, stream/binary.rs, core/indexer/indexer.rs, utils/b64.rs (added on the #29 branch — to be folded in or reverted).

What is NOT duplication (legitimate keripy domain structure — keep separate)

keripy intentionally has two domains: text (qb64) and binary (qb2), via _infil/_exfil (qb64) and _binfil/_bexfil (qb2). So:

  • stream/binary.rs (qb64_to_qb2/qb2_to_qb64) legitimately mirrors keripy codeB64ToB2/_bexfil — the binary domain. It stays a distinct codec.
  • The raw↔qb64 codec is the text domain. Distinct from qb2.

We do NOT merge the text and binary domains. Consolidation only dedupes the shared low-level primitives (one alphabet table, one int↔b64) that both domains draw from.

Proposed scope

  1. One canonical alphabet + reverse table in utils (the single source of truth).
  2. One generic int↔b64 in utils; delete stream/util.rs + indexer copies, repoint callers.
  3. Keep stream/binary.rs qb2 codec (rebuilt on the canonical alphabet table, logic unchanged).
  4. Decide the fate of utils/b64.rs (the P1.2 · Base64 fast path (CESR inner loop) #29 scalar codec) — fold into the canonical module or revert.

Constraints / acceptance

  • No behavior change. All existing keripy test vectors + keripy_diff differential vectors stay green.
  • nix flake check green, incl. cesr-nostd / cesr-wasm; unsafe-free; no lint relaxation.
  • Each moved/merged primitive keeps (or consolidates) its keripy vectors in one canonical location.

Relationship to #29

Prerequisite for #29 (P1.2 base64 fast path). #29 is paused pending this; its baseline bench + spec + plan live on branch perf/p1.2-base64-baseline-bench. Once consolidated, #29's fast-path optimizes the single canonical module.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions