v6.0.0-crc64

First official release of the CRC64-compatible line of the SCANOSS engine.

⚠️ LDB compatibility

This release requires LDB 5.0.0-crc64 or later, from the crc64 branch of scanoss/ldb.

Engine 5.x and earlier go with the traditional LDB 4.x line. The two combinations are not interchangeable in either direction. The constraint comes from the LDB on-disk table layout and the library API surface, not from the choice of hash, so it applies in both modes — CRC64 and MD5 alike.

If you need to stay on LDB 4.x, use the traditional engine line: v5.4.27.

Release lines

The engine is now maintained as two parallel lines, tagged in the same repository:

Line	Branch	Versions	Tag format	Requires LDB
Traditional	main	5.x	v5.4.27	4.x (main branch of scanoss/ldb)
CRC64-compatible	crc64	6.x and later	v6.0.0-crc64	5.x-crc64 (crc64 branch of scanoss/ldb)

Releases of this line carry a mandatory -crc64 suffix, so a binary always identifies where it came from:

$ scanoss -v
scanoss-6.0.0-crc64

RPM packages spell it 6.0.0_crc64, since RPM does not allow - in the Version field. Debian packages keep the hyphen.

What defines this line

• Key length is taken from the table, never assumed. Each LDB table reports its key length (key_ln) from its own configuration, and the engine derives everything from it: 8-byte keys are CRC64, 16-byte keys are MD5. Every table reader was converted — file records, WFP records, MZ source and notices archives, binary scanning, and the -P/-C lookups — so none of them still assumes a 16-byte key.
• MD5 remains fully supported. An MD5 knowledge base keeps working, as long as it was built with LDB 5.x-crc64.
• Bare hash scan targets may be CRC64 or MD5. A hash passed as TARGET is accepted at either length.
• Optional path table. When the knowledge base carries a path table, file paths are resolved through it in every file-table reader.

New: project reconstruction

-p, --project URL_HASH lists the file hash and path of every file in a project, reconstructing its file structure:

$ scanoss -p 1b64ffc16c4cf58a
0018a8688c0c53b3,V2rayNG/app/src/main/java/com/v2ray/ang/util/HttpUtil.kt
011191f1ec12d856,V2rayNG/app/src/main/res/drawable-mdpi/ic_stat_direct.png
...

It requires the pivot table, and the url hash may be MD5 or CRC64.

LDB version is now enforced, at build time and at run time

The engine links libldb dynamically, so the LDB present when it is compiled is not necessarily the one it loads later. Both are validated, and neither check can drift from the other: the requirement is declared once, in inc/ldb_compat.h, and read by both.

• Build time — make aborts before compiling anything if the installed LDB is too old or from the wrong line. Two sources are checked: the ldb.h the compiler resolves, and the libldb.so the linker resolves. The second matters because a stale library earlier in the loader search path yields a binary that compiles cleanly and then fails at run time on the very machine that built it.
• Run time — the loaded library is validated before any table is opened.

"Too old" and "wrong release line" are reported as distinct, actionable errors, because they call for different fixes. An MD5-only LDB whose version number is high enough is still rejected.

The previous checks did not survive this line. The build-time one truncated the version to three characters (5.0.0-crc64 became 5.0, and a two-digit major made bc fail with a syntax error), compared it as a decimal so that 4.10 ranked below 4.2, and never looked at the suffix. The run-time one used strcmp, which is lexicographic: 5.10.0 sorts below 5.9.0, so the engine would have refused to start against a newer LDB. bc is no longer a build dependency.

Notable fixes

• Snippet matching on CRC64 knowledge bases. The WFP record length was hardcoded to 18 bytes (16-byte file id + 2-byte line number). It is now taken from the table definition, which is 10 bytes on a CRC64 knowledge base.
• MZ source and notices archives. The record id length is derived from the table key: the first two key bytes name the MZ file and the rest is the id stored in each record, so 14 bytes for MD5 and 6 for CRC64. The notices path is also built from the table definition instead of a hardcoded oss/notices.
• source_hash in scan results is built using the file table's key length, so it is correct on CRC64.
• -P/--purl and -C/--url-hash validate their input hash against the table key length instead of rejecting anything that is not a 32-character MD5.
• Experimental binary matching is key-length and database aware.
• Project reconstruction memory. -p allocated a fixed 500 MB output buffer and grew the listing with repeated strcat, which is O(n²). It now uses a growable buffer.
• Dangling pointer in fetch_related_purls(): component->crclist pointed at a local stack array after the function returned.
• Duplicate components in the match list are collapsed by purl.
• --force-snippet swallowed its TARGET. It was declared as taking a required argument while the code never reads one, so scanoss --force-snippet TARGET silently consumed TARGET as the option's argument and scanned nothing. It now correctly takes no argument.

Documentation

The README, docs/source/index.rst and the help text were audited against the code and corrected. Among the fixes: --min-match-hits and --min-match-lines were documented but do not exist and are rejected by the parser — the real names are --min-snippet-hits and --min-snippet-lines; --force-snippet was described as "same as -b"; --max-file-content-size, -P/--purl, -C/--url-hash, -p/--project and -S/--snippet-scan were missing from the README; and the install instructions pointed at an archive of a branch that does not exist. CONTRIBUTING.md now documents the branch and tag scheme.

Build prerequisites

zlib1g-dev and libgcrypt-dev, plus LDB 5.0.0-crc64 or later:

git clone -b crc64 https://github.com/scanoss/ldb
cd ldb && make all && sudo make install && cd ..

git clone -b crc64 https://github.com/scanoss/engine
cd engine && make all && sudo make install