Pymarc API compatibility: comprehensive gap analysis and implementation plan

[dchud]

Take a look at https://github.com/matthewdeanmartin/rmarc/blob/main/bench/bench_deathmatch.py. It has several spots where slightly different python api code is used for mrrc processing and that suggests api differences from the pymarc we are aiming for compatibility with.

Please check each api call. Are these true gaps in our pymarc compatibility? Because we aim for as close to 100% compatibility as reasonable possible, review each carefully to confirm whether or not they represent a gap we could close for better pymarc compatibility in our python wrapper.

Identify each gap and come up with a plan to close the gap for improved compatibility.

Update mrrc's python api, tests, example code, introductory and reference documentation, and any other public facing docs as appropriate to close these gaps.

[dchud]

Pymarc API compatibility gaps (Bead: bd-qjjs)

Analysis of rmarc bench_deathmatch.py against pymarc's installed API reveals 4 gaps where mrrc diverges from pymarc:

1. Record accessors are methods, pymarc uses @property

title, author, isbn, issn, subjects, notes, location are all @property in pymarc (record.title), but methods in mrrc (record.title()). The benchmark has a special mrrc code path for this.

2. Missing Field.value() method

pymarc Field.value() returns space-joined subfield values (or control field data). mrrc has no equivalent — the benchmark uses str(field) as a workaround.

3. Missing Record.as_json() method

pymarc uses record.as_json(**kwargs) (wraps json.dumps(record.as_dict())). mrrc only has record.to_json().

4. Missing parse_xml_to_array() top-level function

pymarc exports parse_xml_to_array(xml_file, strict=False, normalize_form=None). mrrc uses xml_to_records() with different calling conventions (string input vs file path/handle).

---

Each gap has a straightforward fix in mrrc/__init__.py — the Rust core doesn't need changes. Gap 1 is the most impactful (breaking change for record.title() callers) and may warrant a deprecation shim.

Bead: bd-qjjs

[dchud]

Expanded gap analysis (Bead: bd-qjjs)

Deep API surface comparison found 12 additional gaps beyond the original 4. Full details in the bead, but highlights:

Critical for drop-in compatibility

• as_marc() / as_marc21() — no way to serialize Record to ISO 2709 bytes (round-trip blocker)
• as_dict() — pymarc's as_json() is built on this
• add_field(*fields) / remove_field(*fields) — pymarc accepts multiple; mrrc accepts one
• remove_fields(*tags) — bulk removal by tag, missing entirely

Naming mismatches

• physicaldescription vs physical_description
• uniformtitle vs uniform_title
• addedentries — missing entirely
• pubyear returns int in mrrc, str in pymarc

Field-level gaps

• Field.format_field() — human-readable string, used by Record property helpers
• Field.add_subfield(code, value, pos=None) — positional insertion missing
• Field.as_marc() / as_marc21() — field-level binary serialization

GIL considerations

• as_json() — no GIL release needed (single record, in-memory)
• parse_xml_to_array() — should follow MARCReader's input dispatch pattern: native Rust I/O for file paths, GIL-managed read for Python file objects, then GIL-free Rust XML parsing. Not just a thin alias over xml_to_records().

Bead: bd-qjjs

[dchud]

Consolidated gap inventory (supersedes earlier comments) — Bead: bd-14jh

Review found errors, duplicates, and 4 new gaps in earlier analysis. This is the authoritative reference. 24 gaps total across 4 tiers.

Tier 1: Structural / semantic divergences (highest impact)

ID	Gap	Impact
A	Separate ControlField class — pymarc has NO ControlField; both types are pymarc.Field. mrrc's separate class breaks isinstance(f, Field) and means get_fields() returns mixed types.	Breaks type checks
B	ControlField .value vs pymarc .data — pymarc stores control field content in .data. mrrc uses .value.	Silent AttributeError
C	Record accessors are methods, pymarc uses @property — 17 accessors (title, author, isbn, issn, issn_title, issnl, subjects, notes, location, series, sudoc, publisher, pubyear, physicaldescription, uniformtitle, addedentries)	Breaks every attribute access
D	Record['999'] returns None vs KeyError — pymarc raises KeyError for missing tags. mrrc returns None.	Silently changes control flow
E	Field.__str__ broken — returns <mrrc.Field object at 0x...>. pymarc returns =245 10$aThe Great Gatsby$c.... __getattr__ doesn't intercept dunders.	Breaks display/logging
F	Record.as_marc()/as_marc21() missing — no single-record ISO 2709 serialization. MARCWriter only writes to files.	Blocks round-trip workflows

Tier 2: Missing methods and signature mismatches

ID	Gap
G	Field.value() missing. Note: ControlField .value attribute conflicts — resolve via Gap B (rename to .data).
H	Field.format_field() missing — human-readable text, used by Record property helpers.
I	Record.as_json(**kwargs) missing — pymarc wraps json.dumps(as_dict()). Keep mrrc's to_json() too.
J	Record.as_dict() missing — pymarc builds as_json() on this.
K	parse_xml_to_array() missing — accepts paths/handles/StringIO. GIL policy: paths and strings → zero GIL (entirely Rust). Python file objects → GIL only for .read(). Document preferred path-based usage.
L	add_field(*fields) — pymarc accepts multiple; mrrc accepts one.
M	remove_field(*fields) + remove_fields(*tags) — pymarc accepts multiple + has bulk tag removal.
N	add_ordered_field() / add_grouped_field() missing.
O	Field.add_subfield(code, value, pos=None) — positional insertion missing.
P	Field.as_marc() / as_marc21() — field-level binary serialization missing.

Tier 3: Naming and return type mismatches

ID	Gap
Q	physicaldescription vs physical_description, uniformtitle vs uniform_title, addedentries missing entirely
R	pubyear returns int in mrrc, str in pymarc (confirmed)

Tier 4: Lower priority

ID	Gap
S	decode_marc() — semi-internal, populates Record from raw bytes. Uncommon in user code.
T	Reader/writer variants — JSONReader, JSONWriter, MARCMakerReader, XMLWriter, TextWriter.
U	Exception hierarchy — pymarc has ~15 specific exception classes. mrrc exposes generic errors.
V	MARC8 Python API — Rust core has MARC8 support but nothing is exported at the Python level. pymarc exports MARC8ToUnicode class, marc8_to_unicode(), map_marc8_field(), map_marc8_record(). Fix: expose via PyO3 + wrapper.
W	Convenience functions — map_records(), map_xml(), parse_json_to_array(), record_to_xml_node().
X	Constants — LEADER_LEN, DIRECTORY_ENTRY_LEN, END_OF_FIELD, END_OF_RECORD, SUBFIELD_INDICATOR, etc.

GIL policy

Avoid holding the GIL whenever possible. as_json() and as_marc() have no GIL concern (in-memory). parse_xml_to_array() should be zero-GIL for file paths and string content; GIL only needed for Python file objects' .read().

Bead: bd-14jh

[dchud]

Replaced bead bd-qjjs with clean bd-14jh — single comment with the full consolidated gap inventory (24 gaps, 4 tiers), fix proposals, GIL policy, and documentation requirements. Earlier comments on this issue are superseded.

Bead: bd-14jh

[dchud]

Final verified gap analysis — Bead: bd-wifu

Replaces bd-qjjs and bd-14jh. Every gap verified via live introspection of both pymarc and mrrc. Found 28 gaps across 4 tiers.

Key finding not in earlier versions: the JSON serialization formats are structurally different (Gap G). pymarc follows the MARC-in-JSON / code4lib schema ({leader, fields} with subfields as array of single-key objects). mrrc uses a flat array with subfields as a single object (loses duplicate subfield codes). as_json() must produce pymarc-compatible output.

Tier 1: Structural (7 gaps)

ID	Gap
A	Separate ControlField class — pymarc has no ControlField, everything is Field
B	Control field .value attr vs pymarc .data attr
C	17 Record accessors are methods, pymarc uses @Property
D	Record['999'] returns None; pymarc raises KeyError
E	Field.__str__ returns <mrrc.Field object> not MARC display format
F	Record.as_marc() / as_marc21() missing — no ISO 2709 bytes
G	NEW: JSON schema mismatch — different top-level structure AND subfield representation

Tier 2: Missing methods (10 gaps)

ID	Gap
H	Field.value() missing
I	Field.format_field() missing
J	Record.as_json(**kwargs) and Record.as_dict() missing (must use pymarc JSON schema)
K	parse_xml_to_array() missing — zero-GIL for paths/strings, GIL only for Python file .read()
L	add_field(*fields) — pymarc accepts multiple; mrrc accepts one
M	remove_field(*fields) + remove_fields(*tags) signature mismatches
N	add_ordered_field() / add_grouped_field() missing
O	Field.add_subfield(pos=) — positional insertion missing
P	Field.as_marc() / as_marc21() missing
Q	Field.linkage_occurrence_num() missing

Tier 3: Naming / types (3 gaps)

ID	Gap
R	physicaldescription, uniformtitle (no underscore), addedentries — missing or wrong names
S	pubyear returns int; pymarc returns str
T	Field.convert_legacy_subfields() classmethod missing

Tier 4: Lower priority (8 gaps)

ID	Gap
V	decode_marc() — semi-internal
W	Reader/writer variants — JSONReader, JSONWriter, MARCMakerReader, XMLWriter, TextWriter
X	Exception hierarchy — 10 pymarc classes, mrrc has zero
Y	MARC8 Python API — Rust has it, Python exports nothing
Z	Convenience functions — map_records(), map_xml(), parse_json_to_array(), etc.
AA	Constants — LEADER_LEN, END_OF_RECORD, SUBFIELD_INDICATOR, etc.
BB	Record instance attributes — force_utf8, pos, to_unicode

Full details with fix proposals in the bead.

Bead: bd-wifu

[dchud]

Superseded by #71 — clean, dependency-ordered implementation plan replacing the accumulated analysis across multiple comments here.