#870 Add support for in-place processing of VRL files with RDWs

[yruslan]

Summary by CodeRabbit

• New Features

  • Added support for reading variable-length, RDW-prefixed records, including configurations where headers are included in record lengths.
  • Improved handling of RDW headers, including endianness, length validation, and invalid or zero-length records.
  • Added support for VRL+RDW input and output workflows.

• Bug Fixes

  • Variable-length records now load correctly when record sequence mode is enabled without a custom parser.

• Documentation

  • Expanded documentation for fixed- and variable-length record extraction.

• Tests

  • Added coverage for RDW length variations and round-trip COBOL data conversion.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6b7c48fd-f790-41cb-aa01-6a9a3684f1bd

📥 Commits

Reviewing files that changed from the base of the PR and between 525c5ae and 35fbe7d.

📒 Files selected for processing (15)

• README.md
• cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/parser/headerparsers/RecordHeaderParserRDW.scala
• cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/reader/VarLenNestedReader.scala
• cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/reader/extractors/raw/FixedRecordLengthRawRecordExtractor.scala
• cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/reader/extractors/raw/FixedWithRecordLengthExprRawRecordExtractor.scala
• cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/reader/extractors/raw/RawRecordContext.scala
• cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/reader/extractors/raw/VariableBlockVariableRecordExtractor.scala
• cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/reader/extractors/raw/VariableRecordLengthRecordExtractor.scala
• cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/reader/recordheader/RecordHeaderDecoderRdw.scala
• cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/reader/recordheader/RecordHeaderParameters.scala
• cobol-parser/src/test/scala/za/co/absa/cobrix/cobol/processor/impl/CobolProcessorBaseSuite.scala
• cobol-parser/src/test/scala/za/co/absa/cobrix/cobol/reader/recordheader/RecordHeaderParametersFactory.scala
• pom.xml
• project/Dependencies.scala
• spark-cobol/src/test/scala/za/co/absa/cobrix/spark/cobol/SparkCobolProcessorSuite.scala

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.

---

Walkthrough

The change adds RDW-based variable-length record extraction, supports record lengths that include or exclude headers, updates extractor selection, adds VRL+RDW tests, refreshes ScalaDoc, updates the GPG example, and bumps ScalaTest.

Changes

RDW variable-length records

Layer / File(s)	Summary
RDW length contract and parsing cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/reader/recordheader/..., cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/parser/headerparsers/RecordHeaderParserRDW.scala, cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/reader/extractors/raw/RawRecordContext.scala	RecordHeaderParameters now records whether headers are part of the record length. RDW decoder parameters are exposed. RDW parsing is delegated to the companion object with endianness and adjustment values.
VRL extractor selection and reading cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/reader/VarLenNestedReader.scala, cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/reader/extractors/raw/...	Variable-length sequence input now creates VariableRecordLengthRecordExtractor. The extractor reads RDW headers, skips invalid records, buffers payloads, and advances offsets. Related raw extractors receive ScalaDoc comments.
RDW extraction and conversion tests cobol-parser/src/test/scala/za/co/absa/cobrix/cobol/..., spark-cobol/src/test/scala/za/co/absa/cobrix/spark/cobol/SparkCobolProcessorSuite.scala	Tests cover extractor selection, payload reading, header-inclusive and header-exclusive RDW lengths, transformed binary output, and JSON round trips.

Documentation and test tooling

Layer / File(s)	Summary
Example and ScalaTest updates README.md, pom.xml, project/Dependencies.scala	The GPG example uses gpg_private_key_passphrase and a wildcard .gpg path. ScalaTest is updated from 3.2.19 to 3.2.20.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: 🟡 Moderate · up to 35fbe

This PR enables in-place RDW processing but can emit a partial payload or silently drop a record when a file ends before the declared length, potentially corrupting or suppressing records from malformed inputs. Merge should wait for fail-closed handling or explicit owner acceptance of this behavior.

Sequence Diagram(s)

sequenceDiagram
  participant VarLenNestedReader
  participant VariableRecordLengthRecordExtractor
  participant RecordHeaderParserRDW
  participant InputStream
  VarLenNestedReader->>VariableRecordLengthRecordExtractor: create extractor for variable-length sequence input
  VariableRecordLengthRecordExtractor->>InputStream: read RDW header
  VariableRecordLengthRecordExtractor->>RecordHeaderParserRDW: processRdwHeader with byte order and adjustment
  RecordHeaderParserRDW-->>VariableRecordLengthRecordExtractor: return record metadata
  VariableRecordLengthRecordExtractor->>InputStream: read record payload
  VariableRecordLengthRecordExtractor-->>VarLenNestedReader: return buffered payload

Loading

Poem

A rabbit reads records in a neat little row
RDW marks each length as the bytes flow
Headers may count, or stay outside
Payloads wait buffered inside
Tests hop after, with outputs verified low!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding in-place processing support for variable-record-length files with RDW headers.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (15 skipped: 15 unsupported.)

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/870-add-support-in-place-vrl

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

JaCoCo code coverage report - 'cobol-parser'

Overall Project	90.06% -0.6%	🍏
Files changed	42.39%	🍏

File	Coverage
RecordHeaderParameters.scala	100% -71.43%	🍏
VariableBlockVariableRecordExtractor.scala	97.17%	🍏
FixedRecordLengthRawRecordExtractor.scala	95.19% -0.96%	🍏
VariableRecordLengthRecordExtractor.scala	93.9%	🍏
RecordHeaderDecoderRdw.scala	91.84%	🍏
FixedWithRecordLengthExprRawRecordExtractor.scala	71.12%	🍏
VarLenNestedReader.scala	67.71% -0.42%	🍏
RecordHeaderParserRDW.scala	41.58% -0.69%	🍏

[github-actions]

JaCoCo code coverage report - 'spark-cobol'

Overall Project	83.57%	🍏

There is no coverage information present for the Files changed