Record types defined in layout

[Zaba505]

We are using cpybkc for work where the files are licensed via a vendor which means we're ideally avoiding checking in the copybook definitions into our code repositories. Could cpybkc be updated to fully hold the record definitions instead of referencing them from a copybook file? This would greatly help us since the cpybkc file format wouldn't count under our vendors license and could be checked into our repos with no issues.

[Zaba505]

Thanks for raising this. It's a good point, and we're taking it on. A layout will be able to hold its record definitions itself, and cpybkc init will write them for you from your copybooks. I'll plan the work into stories and link them here.

Why

Today a layout isn't enough to regenerate code by itself. cpybkc also needs every copybook the layout names, at the paths it names them. If those copybooks can't be committed, everyone who regenerates has to obtain them and lay them out exactly as before, and no checked-in file records whether they did. That's a poor experience. It's also a correctness risk: a different copybook version produces different code with no error.

cpybkc already refuses a copybook search path for exactly this reason (cli/SPEC.md):

A search path makes the same layout resolve to different copybooks on two machines with no difference between them that any file records, which is the failure a generated file library cannot detect and cannot recover from — the code compiles, and the offsets are wrong.

Uncommitted copybooks lead to the same failure. A layout that carries its record definitions closes that gap: what's checked in is everything a regeneration reads.

What it will look like

• init does the conversion. cpybkc init --copybook … already reads your copybooks to scaffold a layout. It will also be able to write each record's definition into the layout, so nobody transcribes PICTUREs by hand. It reads COBOL through the same library cpybkc uses today.
• The conversion covers everything. Every copybook cpybkc can read today will convert, not just a simple subset.
• The details come first in the spec. The notation, and how it sits beside today's (copybook "path" NAME) reference, will be settled in the layout spec before anything is built.

On the licence

I can't tell you what your licence covers. That's for your vendor or counsel. An inlined definition will carry the same item names, nesting, PICTUREs, USAGEs, OCCURS and REDEFINES as the copybook, because that's what makes it enough to generate from. The generated Go code already carries each item's COBOL name and shape too. Please confirm with your vendor that committing this form is fine.

In the meantime

cpybkc never looks at git, so a copybook doesn't have to be committed. It only has to be on disk at the path the layout names, relative to the layout, while cpybkc runs. A git-ignored cpy/ beside the layout, filled by a setup or CI step, works today. The cost is the one described above: every machine that regenerates has to fill it the same way.

[Zaba505]

The work is planned and filed. One correction to my reply above: cpybkc init won't write definitions into the layout. It writes one definitions file per copybook beside the layout, and each record names its file where it names a copybook today:

(record DEBIT-POSTING
  (definitions "posting.cpy.sexpr" POSTING-RECORD)
  (alternative (item DEBIT-POSTING PST-DEBIT)))

A definitions file is the copybook's entries written as S-expressions, one entry per data description entry, with every clause cobol-go reads:

(entry 01 LEDGER-TRAILER)
  (entry 05 TRL-TYPE (pic "X(2)"))
  (entry 05 TRL-NET (pic "S9(13)V99") (usage COMP-3))

If you'd rather keep everything in one file, the same entries can sit in the layout in a (definitions-block NAME …), and a record names the block instead. (copybook "path" NAME) keeps working, so one layout can mix all three, and all three give byte-identical generated code. The syntax above is the proposal; the spec stories settle it before anything is built.

Spec first:

• story(layout): decide how a copybook's entries and clauses are written as S-expressions, in a definitions file or in a block inside the layout #395: how entries and clauses are written
• story(layout): decide how a record names a definitions file or a block beside (copybook "path" NAME), and what a run reads #399: how a record names a definitions file or a block
• story(cli): decide how cpybkc init writes a definitions file per copybook beside the layout it scaffolds #403: how init writes one definitions file per copybook

Then:

• story(layout): a definitions file or a definitions block is read and checked against the published layout schema before any COBOL in it is judged #400, story(layout): every copybook cpybkc can read converts to a definitions file that cobol-go reads back as the same entries #401, story(layout): a record can name a definitions file or a definitions block beside (copybook …), and a run says plainly it resolves neither yet #402, story(layout): a COBOL name the S-expression grammar cannot read as a symbol is written (word "…") wherever a layout names an item #404: reading and checking definitions, and converting copybooks, with a check that every copybook cpybkc can read today converts and reads back the same
• story(resolve): a record whose definitions file the layout names resolves to the descriptor its copybook gives, with every fault pointing into that file #405, story(resolve): a record whose definitions are written in a block in the layout resolves to the descriptor its copybook gives, with every fault pointing into the layout #409: definitions files and blocks resolve to the same output their copybook gives
• story(cli): cpybkc init writes each copybook as a definitions file beside the layout, so a project regenerates with no copybook #408: init writes the definitions files, so a project regenerates with no copybook present
• story(layout): a worked example whose project ships definitions files and no copybook #410: a worked example with no copybook in the tree

Supporting stories: #396 and #397 (schema checking), #406 (init notes records it can't derive instead of refusing the whole copybook), #398 (release policy) and #407 (copybooks that use COPY, which this doesn't need).

[Zaba505]

The spec for how a record names its definitions is written: #399, in pull request #415. Here is where it landed.

Definitions can live in the layout or in files the layout names. Each record names its own source, where (copybook "path" NAME) goes today:

;; a definitions file, by its path relative to the layout
(record DEBIT-POSTING
  (definitions "posting.cpy.sexpr" POSTING-RECORD)
  (alternative (item DEBIT-POSTING PST-DEBIT)))

;; a definitions-block held in this layout, by its name
(definitions-block TRAILER
  (entry 01 LEDGER-TRAILER)
    (entry 05 LT-TYPE (pic "X")))

(record LEDGER-TRAILER (definitions TRAILER LEDGER-TRAILER))

(copybook "path" NAME) is unchanged. One layout can mix all three sources, record by record. The spec requires all three to give byte-identical output: a record bound to a copybook, to the definitions file converted from it, or to a block holding that file's entries.

cpybkc init writes one definitions file per copybook, beside the layout. It does not write definitions into the layout, which my first reply above said it would. (#403 settles the details.)

Moving a file's entries into the layout is a documented edit (#403). Paste the entries into a definitions-block, point the record at the block, and check the result with cpybkc --emit-ir. The two runs' descriptors should compare equal byte for byte.

A few rules keep what a run reads predictable:

• A run reads the layout, plus exactly the copybooks and definitions files its records name.
• A block opens nothing.
• A definitions file can't name another file, so there's no chain of includes to follow.
• A block is visible only in its own layout. Definitions shared between layouts go in a file.

[Zaba505]

All 17 stories have landed on main. #395 through #410 and #423 are merged, and the feature ships in the next release after v0.0.7.

What you can do now

• Convert once with init. cpybkc init --copybook … writes one definitions file per copybook beside the layout, and each record names its file with (definitions "path" NAME). Commit the layout and the definitions files, and the project regenerates with no copybook present.
• Or keep it all in the layout. A record can name a (definitions-block NAME …) held in the layout instead of a file. (copybook "path" NAME) still works, and one layout can mix all three.
• The output doesn't change. A record bound to a copybook, to the definitions file converted from it, or to a block holding those entries resolves to a byte-identical descriptor. CI checks this, and checks that every copybook cpybkc can read converts and reads back the same.
• A worked example with definitions files and no copybook in the tree is at example/receivables.

Two things since the last update

Copybooks that use COPY now work (#423). A layout says where each member is with (copy-member STDHDR "cpy/stdhdr.cpy"), or (copy-member (of STDHDR PAYLIB) …) for COPY … OF. There is still no search path. For init, pass --copy-member STDHDR=cpy/stdhdr.cpy. The definitions file init writes holds the copybook's expansion, so the members aren't needed after conversion either.

The definitions notation covers more of COBOL (#432). cobol-go now reads constructs that conversion used to refuse, so they convert:

• A prefixed literal is written as its text, prefix in upper case: "X'00'", "N'AB'".
• An ALL literal is a form: (value (all "'*'")), (value (all SPACES)).
• A word that begins with a digit is written (word "1ST-FIELD").
• VALUES ARE needs no notation; ARE is a noise word, like IS.

The details are in docs/layout/SPEC.md.

What I said about the licence in my first reply still stands. A definitions file carries the same names and shapes as the copybook it came from, so please check with your vendor before you commit one.

Closing this as resolved. If something doesn't convert the way you expect, please open a new discussion with the copybook shape.