OPENNLP-1909: General verified installer for user-supplied third-party resources - #1211
Draft
krickert wants to merge 16 commits into
Draft
OPENNLP-1909: General verified installer for user-supplied third-party resources#1211krickert wants to merge 16 commits into
krickert wants to merge 16 commits into
Conversation
ResourceInstaller fetches a user-chosen location, verifies an optional SHA-256 before anything is unpacked, detects the format from bytes, and extracts tar.gz and zip archives with path-escape rejection; plain gzip and raw files store under their source names. Nothing is bundled and no location is built in, so externally licensed data stays entirely outside the distribution. The tar plumbing moves to an internal reader shared with the dictionary installer.
…d tar reader, corrected javadoc
…xample Add a resource installer section to the model-loading manual citing ResourceInstallerTest.
- Move the 512-byte tar header probe out of ResourceInstaller into TarStream.startsWithHeader, so the header layout is known in exactly one class: the size field offsets and the ustar magic check now sit next to the reader that already owns them. - Split the tar unpacking out of unpackGzip into its own unpackTar helper, leaving unpackGzip with the single decision between an archive and a plain file. - Replace the hand rolled hex loops with HexFormat.of().formatHex in the installer and in its test, and derive the uppercase digest helper from the lowercase one instead of a second digit table. - Replace the manual fill and skip loops in TarStream with readNBytes and skipNBytes, mapping EOFException onto the existing "truncated tar archive" failure. - Extract named constants for values that were spelled inline: the SHA-256 algorithm name, the gzip suffix, the default resource name, the copy buffer size, the gzip and zip magic bytes, and the tar block size, name length, size and type offsets and regular-file type flags, in main and test code alike. - Validate source and targetDirectory separately so the message names the parameter that was null, and reject a null or non markable stream in startsWithHeader. - Document every private helper with @PARAM, @return and @throws, and drop the commentary that explained motivation rather than behavior from the class javadoc and from the empty private constructor. - Pin the argument validation messages in ResourceInstallerTest and assert the installed file list rather than a bare directory count. - Cover startsWithHeader: the stream position survives detection on a real archive, non tar content is rejected through a parameterized case per reason, and null or non markable streams are rejected. - Narrow the test digest helpers from throws Exception to NoSuchAlgorithmException. - Rewrite the docbook section so it matches the class javadoc, reindent it with spaces like the rest of the file, and fix the programlisting so the example and its trailing comment render.
An OpenNLP model file is itself a zip archive, so byte-based format detection unpacked a downloaded model into its manifest and *.model innards where the operator asked for the model. A source named *.bin is now always stored verbatim: every OpenNLP model consumer loads the packed file.
The model-loading section now names ResourceInstallerTest#testInstallEndToEndUsageExample as the pin for its programlisting.
Red evidence: mvn test-compile fails with "cannot find symbol: class Limits" and the missing install(URI, Path, String, Limits) and resolveRedirect seams; the SHA-512, staged-atomicity, and redirect policy tests pin behavior the current installer does not have. The new tests cover: SHA-512 digests selected by hex length next to SHA-256, malformed digests rejected as argument errors, staged installation that leaves the target untouched when a tar, zip, or truncated archive fails partway, download and expansion ceilings against oversized sources and small archives that expand into bombs, and a scripted loopback HTTP server exercising redirects (absolute, relative, capped chains, missing Location, non-http targets, https downgrade), error statuses, stalled responses against the read timeout, and bodies that exceed the download ceiling with or without a declared length.
Downloads and unpacking now run under Limits: http and https fetches get connection and read timeouts, follow at most a capped number of redirects, resolve relative Locations, and refuse redirect targets that leave the http and https schemes or downgrade https to http. A declared content length beyond the download ceiling fails before the body is read, and both the transferred and the expanded bytes are charged against their ceilings so lying servers and archive bombs abort within one buffer. Checksums accept SHA-512 next to SHA-256, selected by hex digest length, and malformed digests fail fast as argument errors. Installation is staged: content unpacks into a hidden staging directory on the target filesystem and is promoted by renames only after the download verified and every entry unpacked cleanly, so a failed installation leaves the target as it was. The red suite from the previous commit passes: 27 ResourceInstaller tests, 12 scripted local-server HTTP tests, 14 TarStream tests, and the full runtime module at 1700 tests.
The manual's resource-installer section now explains digest-length algorithm selection (64 hex characters SHA-256, 128 SHA-512), the staged installation guarantee that a failed install leaves the target untouched, and the bounded network behavior: timeouts, the capped redirect policy with its scheme and downgrade rules, and the download and expansion ceilings with their defaults. A limits listing is mirrored by ResourceInstallerTest#testInstallWithinCustomCeilingsSucceeds.
Both sides of each ceiling are now asserted: a download and an expansion exactly at the ceiling install, one byte of cumulative overrun across entries rejects, proving the expansion budget is shared rather than per entry. Limits.DEFAULT values are pinned, all five redirect statuses the code claims (301, 302, 303, 307, 308) are followed under a parameterized test, a zero redirect allowance refuses the first redirect, a malformed Location fails loud with the offending value, SHA-512 comparison ignores hex letter case like SHA-256, and reinstalling over the same target replaces the delivered files. The manual now cites the staged-installation and default-limit tests next to the guarantees they assert. 39 + 19 installer tests green.
Does not compile on its own: the tests call Limits.builder() and
createDownloadFile, both added in the following commit. Against the
current implementation, with those two added as stubs, the run is:
ResourceInstallerHttpTest
testSubMillisecondReadTimeoutStillTimesOut
timed out after 15 seconds
testTimeoutBeyondTheMillisecondRangeIsCapped
Arithmetic long overflow
ResourceInstallerTest
testPromotionRefusesToFollowASymlinkedDirectory
Expected java.io.IOException to be thrown, but nothing was thrown
testUnsupportedSourceSchemeIsRejected
expected IllegalArgumentException but was java.net.UnknownHostException
expected IllegalArgumentException but was java.net.UnknownServiceException
expected IllegalArgumentException but was java.nio.file.NoSuchFileException
expected "source scheme must be ..." but was "URI is not absolute"
testUnsupportedSourceSchemeIsRejectedBeforeCreatingTheTarget
expected IllegalArgumentException but was java.net.UnknownHostException
TarStreamTest
testHeaderWithWrongChecksumIsRejected
testRejectedPaxGlobalHeader (8 cases)
testSparseEntriesAreRejected (2 cases)
testBase256SizeFieldIsRead
testBase256SizeFieldCarriesLengthsBeyondTheOctalRange
testBase256SizeFieldAcceptsTheLargestRepresentableLength
testBase256SizeFieldBeyondTheLongRangeIsRejected
testNegativeBase256SizeFieldIsRejected
the base-256 size encoding is not read at all
testStartsWithHeaderRejectsUstarMagicWithoutAChecksum
expected false but was true
testPaxExtendedHeaderSuppliesTheEntryName
testGnuLongNameHeaderSuppliesTheEntryName
expected the full path, but was its first 100 bytes
testPaxExtendedHeaderSuppliesTheEntrySize
expected 10 but was 0
testGnuHeaderDoesNotReadItsAtimeAsANamePrefix
expected "./short.txt" but was "15237132225/./short.txt"
testUstarPrefixIsJoinedToTheName
testHeaderWithAnEmptyNameIsRejected
testEntryStreamRejectsInvalidReadRanges
testZeroLengthReadReturnsZero
expected 0 but was -1
The sub-millisecond read timeout hangs rather than failing an assertion:
zero milliseconds means no timeout to HttpURLConnection, so the tightest
setting a caller can express becomes the loosest. That is why the test
carries an explicit @timeout.
The tar fixtures follow archives written by GNU tar 1.35 rather than a
minimal shape, because the shape is the point. A pax archive carries an
extended header ahead of every entry, including entries needing no
override, so the metadata-only case is the common one. The entry header
after an extension header holds a truncated name, so the extension header
is the only place the real one appears. Verified against tar --format=pax,
--format=posix, --format=gnu, and --format=gnu --incremental.
The base-256 size fixture is synthetic, since producing a real one needs an
entry of 8 GiB or more. It was cross-checked the other way instead: GNU tar
1.35 lists a header written by TarArchives.base256Header as 8589934592
bytes, so the encoding the tests assert against is the one tar writes.
Two timeout tests replace an earlier pair that computed the expected
milliseconds with their own copy of the conversion. Those asserted the
test's arithmetic, not the installer's, and would stay green through any
regression. These drive HttpURLConnection instead.
testClassicHeaderWithoutUstarMagicIsRead passes before the change as well.
It guards the new checksum-based detection against dropping classic v7
archives, which the previous ustar-magic shortcut happened to accept.
Also consolidates the tar fixture that existed once per test package into
TarArchives, which now writes real ustar magic and header checksums, and
builds classic, GNU, and prefix headers plus pax records with correct
length prefixes.
ResourceInstaller Timeouts: a positive duration shorter than a millisecond rounded to zero, which HttpURLConnection reads as no timeout at all, and Duration.toMillis raises ArithmeticException on a duration too large for the long range, before the old Math.min could cap it. Conversion now clamps into [1, Integer.MAX_VALUE] and catches the overflow. Schemes: only http, https, and file are accepted, checked at the public boundary before the target directory is created. Anything else went to whichever URL handler the runtime had installed; those carry no connection or read timeout, so an unresponsive server blocked the caller forever. The byte ceilings did apply on that path, since they are charged as bytes arrive. file locations now read through Files.newInputStream. Download file: created on the target filesystem rather than in the system temporary directory, where the 1 GiB default ceiling could exhaust a small /tmp while the target had room. It is hidden, so a leaked one shows up as staging residue rather than as an installed file. Promotion: refuses to descend through a symbolic link that already exists below the target. Every entry name can be inside the staging directory and the content still land outside the target, because createDirectories followed such a link. This covers links present when the installation runs, not a tree modified while it runs. Limits: adds a builder seeded from DEFAULT, so a caller states only what differs instead of five positional arguments, two of them Duration and two of them long. The record stays immutable and the builder validates through the canonical constructor. TarStream Detection now verifies the header checksum instead of accepting the ustar magic outright, so arbitrary content carrying that magic at offset 257 is no longer read as an archive with a trusted size field. Both the unsigned and the signed sum are accepted, as historical writers differ. Long names are read from the extension header that carries them. A pax extended header supplies path and size for the entry after it; a GNU long-name header supplies its name. Both are everyday formats: bsdtar and tar --format=posix write an extended header ahead of every entry, and the entry header that follows holds only a truncated name, so neither ignoring nor refusing the extension header is workable. Keywords other than path and size are ignored, because this reader exposes only an entry's name, size, type, and content, and no other pax keyword changes those. Sparse entries are the exception and are refused, by type flag S and by the GNU.sparse.* records, because their archived bytes encode holes rather than the content. A global header is refused if it carries path or size, which would change every entry after it. The POSIX ustar name prefix is honored, but only for POSIX ustar. GNU writes "ustar" and two blanks where ustar writes "ustar" and a NUL, and puts atime at the offset ustar gives to the prefix, so reading the two alike delivered every entry of a GNU incremental archive under a directory named after an octal timestamp. A size field in the base-256 encoding, which GNU writes for entries beyond the eleven octal digits the field holds, is read, so an entry of 8 GiB or more states its length correctly. The sign sits in the bit below the encoding marker, not in the marker itself, so a negative value is refused rather than wrapped into an enormous positive length, and a value past the long range is refused rather than truncated into a short one, which would stop the reader inside the entry and leave it reading content as headers. A header with a valid checksum but no name is refused, matching what detection already required of the first byte. entryStream's read(byte[], int, int) validates its range before answering, so an invalid range is reported even when the entry is exhausted or the length is zero, and a zero-length read returns 0 rather than -1. This override raises the exceptions InputStream specifies rather than the IllegalArgumentException used elsewhere in the package.
States the accepted source schemes and why the others are refused: those handlers carry no connection or read timeout. The byte ceilings applied on that path either way, so the manual does not claim otherwise. Records which tar formats unpack under their real paths, namely bsdtar, tar --format=posix, and tar --format=gnu, and which entries are refused and why. Scopes the staged-installation guarantee to symbolic links that already exist below the target, which is what the check covers. Switches the Limits example to Limits.builder(), so it states only the limits that differ from the defaults instead of five positional arguments. The listing is mirrored by ResourceInstallerTest#testInstallWithinCustomCeilingsSucceeds.
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Aug 14, 2026
krickert
added a commit
that referenced
this pull request
Aug 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a general installer for user-supplied third-party resources: training corpora, dictionary archives, and lexicons that the project cannot bundle. The caller supplies the location and thereby accepts that resource's license; no locations are built in and no data ships with OpenNLP.
ResourceInstalleris a single hardened download-and-unpack path:http,https, andfilelocations are accepted, checked at the public boundary. Remote fetches carry connection and read timeouts, follow a bounded number of redirects, and refuse redirects that leave the http/https schemes or downgrade https to http.Limitsvalue (download bytes, expanded bytes, entry counts);Limits.DEFAULTapplies when none is given andLimits.builder()starts from it.*.binsources are stored packed, because OpenNLP model files are zip archives their consumers load packed.The tar reading is a small forward-only reader,
TarStream, rather than a dependency. It reads classic v7, POSIX ustar, GNU, and pax formats with a valid header checksum required in every case, honors the ustar name prefix field, reads paxx(path, size) and GNULlong-name extension headers, and reads GNU base-256 sizes for entries of 8 GiB or more. Sparse entries are refused because their archived bytes are not the file content, and a pax global header carryingpathorsizeis refused because it would silently rewrite every following entry. Metadata expansion (extension header sizes) is bounded like everything else.Every behavior above is pinned by tests, in failing-test-then-fix commit pairs.
Context: #1190 and #1191 each grew a per-feature download path during review, and the discussion on #1191 raised the right question about how much download surface the project should carry. This class is the proposed answer: one shared, fail-closed installer that those PRs (and the existing
DownloadUtilmodel fetch) can converge on, so hardening lands once instead of once per feature.