cobolwork 0.5.0

cobolwork 0.5.0: precision per rule, measured by labels a machine made.

Install: npm install -g @portll/cobolwork or pip install cobolwork; either gives the command cobolwork. The PyPI package runs the same files with the Node.js on your PATH, which must be Node 22 or later. The npm package is attached here as cobolwork-0.5.0.tgz to pin, and as cobolwork.tgz.

What 0.5.0 adds over 0.4.0:

• Precision per rule, from machine labels, with the label source on every number. Each path finding's verification plan was run in ironwork with a marker at its source. A marker that reaches the sink's operand at the finding's line confirms the finding, and the run's sealed journal is the evidence. Flaws, and near-misses that differ from them only in the guard, were also planted into real programs, so those labels are known by construction. An unknown is counted as neither right nor wrong, so a rule's precision is a range until nothing about it is unknown. docs/spec/reach.md §9.6 says how it is measured; the table is below.
• Over 500 held-out repositories, 1,896 path findings in 48 repositories were labelled by execution, in ironwork as released in 0.2.0. 34 are confirmed, all cics-terminal-to-log, which puts that rule at 12.6% to 100%. None was refuted, so every other rule labelled by execution is 0% to 100%. Most stay unknown for reasons that are measured: 709 sinks ironwork has no trace for yet (689 of them a file assigned from a data item), 551 sources the labeller cannot feed (303 of them GnuCOBOL command-line and environment reads, which ironwork refuses as not Enterprise COBOL), and 326 findings in programs ironwork does not compile.
• Seven rules were measured on planted flaws and near-misses, 25 real host programs for each: OS commands, dynamic program loads, CICS COMMAREA length checks, subscripts, reference modification and dynamic SQL from the command line, and a CICS transfer to a program the terminal names. Each reported every flaw planted for it and no near-miss: 275 flaws of 275 and 0 near-misses of 325, 100% precision and 100% seeded recall. Seeded recall is measured against planted flaws, not against flaws as they occur.
• --all-routes gives each path finding from scan and flow every source and every statement on any route from one of them to the sink, not only the shortest route. Without the flag nothing changes. docs/spec/reach.md §9.8 states when coverage from fuzzed runs may refute a finding; until a run records the statements it executes and which operand bytes came from the input, no finding is refuted by coverage.
• HLASM is read. The hlasm set reports a MODESET to key zero or supervisor state, EX and EXRL, the cross-memory instructions, a direct call to the security product, and the CSECT or ENTRY a COBOL CALL reaches. A job step that runs an assembler module is a defined program. Each operation is looked up in rules/hlasm-operations.json, which cites the IBM manual that defines it. HLASM is read, not assembled: macros are not expanded and conditional assembly is not evaluated. Every file z390 assembles in the 3,185-repository corpus reads with no card error.
• CICS web clients: web-client-opens-cleartext reports a WEB OPEN that asks for HTTP by its option, by SCHEME, or through a client URIMAP the CSD defines with SCHEME(HTTP) and no ATTLS(AWARE). web-receive-length-exceeds-area reports a WEB RECEIVE or WEB CONVERSE whose MAXLENGTH is longer than its INTO area.
• Weak cryptography asked of ICSF. The crypto set reads calls to ICSF callable services by IBM's parameter lists, and reports a single-length key (SINGLE, KEYLN8 or SINGLE-R) from Key Generate, MD5 or SHA-1 for One-Way Hash, and an initialization vector that only ever holds a constant. SC-13 joins the NIST mapping.
• Credential rules report less that is not a credential, and more that is. cobol-value-credential reads data levels only (01 to 49 and 77), reads a VALUE across lines, and passes over values of several words. The TSO, RACF and CICS rules pass over placeholders and stop at comments. New: jcl-racf-new-password for the new password in PASSWORD=(old,new), a JCL PARM that ends in a quote, a CONNECT with IDENTIFIED BY, and a CICS password change. The mainframe credential rules are offered to Betterleaks in pull request 379.
• A check made before an EXEC CICS LINK, XCTL, START, RETURN, LOAD or RELEASE holds after it: their PROGRAM, TRANSID and SYSID options are only read, so a loop back through the command no longer removes the check.
• An abend from ironwork's fuzzing is placed in the file it happened in when that file is a member of a COPY library or a program CALLed from a load library, through the roots the fuzz manifest lists. A file that cannot be placed is reported as a problem, not as a finding.
• Control analysis compares a write only with the fields it can overlap. On 26 programs of 17 MB each, the flow set takes 106 to 130 s against 198 to 244 s, with identical findings.

The HLASM reading is from the 0.8.0 roadmap and the CICS web client rules from 0.7.0, shipped early.

The package's lib/revision.json names the commit it was built from, 2144d56.

Precision, 500 held-out repositories

Rule	Labels	Right	Wrong	Unknown	Precision	Recall
argv-or-env-to-arithmetic	execution 84	0	0	84	0% to 100%	-
argv-or-env-to-connection-target	execution 5	0	0	5	0% to 100%	-
argv-or-env-to-dynamic-file-path	execution 588	0	0	588	0% to 100%	-
argv-or-env-to-dynamic-program-load	planted 25	25	0	0	100%	100%
argv-or-env-to-dynamic-sql	execution 4	0	0	4	0% to 100%	-
argv-or-env-to-dynamic-sql	planted 50	50	0	0	100%	100%
argv-or-env-to-loop-bound	execution 23	0	0	23	0% to 100%	-
argv-or-env-to-occurs-depending-count	execution 76	0	0	76	0% to 100%	-
argv-or-env-to-os-command	execution 30	0	0	30	0% to 100%	-
argv-or-env-to-os-command	planted 25	25	0	0	100%	100%
argv-or-env-to-reference-modification	execution 50	0	0	50	0% to 100%	-
argv-or-env-to-reference-modification	planted 50	50	0	0	100%	100%
argv-or-env-to-subscript	execution 40	0	0	40	0% to 100%	-
argv-or-env-to-subscript	planted 50	50	0	0	100%	100%
cics-commarea-without-length-check	planted 50	50	0	0	100%	100%
cics-protected-field-to-record-key	execution 6	0	0	6	0% to 100%	-
cics-protected-field-to-record-update	execution 6	0	0	6	0% to 100%	-
cics-terminal-to-arithmetic	execution 126	0	0	126	0% to 100%	-
cics-terminal-to-cics-dynamic-transfer	planted 25	25	0	0	100%	100%
cics-terminal-to-log	execution 270	34	0	236	12.6% to 100%	-
cics-terminal-to-outbound-host	execution 2	0	0	2	0% to 100%	-
cics-terminal-to-queue-name	execution 5	0	0	5	0% to 100%	-
cics-terminal-to-reference-modification	execution 11	0	0	11	0% to 100%	-
cics-terminal-to-subscript	execution 11	0	0	11	0% to 100%	-
cics-web-to-outbound-host	execution 2	0	0	2	0% to 100%	-
cics-web-to-reference-modification	execution 2	0	0	2	0% to 100%	-
database-to-queue-name	execution 5	0	0	5	0% to 100%	-
file-record-to-dynamic-file-path	execution 101	0	0	101	0% to 100%	-
file-record-to-dynamic-program-load	execution 140	0	0	140	0% to 100%	-
file-record-to-message-queue	execution 11	0	0	11	0% to 100%	-
file-record-to-os-command	execution 69	0	0	69	0% to 100%	-
file-record-to-outbound-http	execution 2	0	0	2	0% to 100%	-
jcl-instream-to-arithmetic	execution 221	0	0	221	0% to 100%	-
jcl-instream-to-subscript	execution 1	0	0	1	0% to 100%	-
jcl-parm-to-arithmetic	execution 1	0	0	1	0% to 100%	-
jcl-parm-to-reference-modification	execution 3	0	0	3	0% to 100%	-
jcl-parm-to-subscript	execution 1	0	0	1	0% to 100%	-

Verdict	Labels	Right	Wrong	Unknown	Precision	Recall
attacker-driven	execution 1507	34	0	1473	2.3% to 100%	-
mitigated	execution 55	0	0	55	0% to 100%	-
refuted	execution 6	0	0	6	0% to 100%	-
upstream	execution 328	0	0	328	0% to 100%	-

Summary

Precision per rule, measured by labels a machine made.

• Running each finding’s verification plan in ironwork confirmed 34 findings over 500 held-out repositories and refuted none, so cics-terminal-to-log is 12.6% to 100% and every other rule labelled by execution is 0% to 100%.
• Seven rules reported all 275 flaws planted into real programs and none of the 325 near-misses.
• --all-routes lists every route of a path finding, not only the shortest.
• HLASM is read, and its privileged operations are reported.
• Cleartext CICS web clients, oversized web receives and weak ICSF cryptography are reported.
• The credential rules report fewer things that are not credentials, and four more forms that are.
• Control analysis takes about half the time on large programs.