Repository navigation
Running Imports
ImportResult result = tie.importFiles(definition, context);ImportContext |
Meaning |
|---|---|
consumer |
Required. Receives every valid record |
variables |
Values of the {variable} markers of the name patterns |
watermarks |
Last processed timestamp per layout, as returned by the previous run |
alreadyProcessed |
A predicate on RemoteFile (name, size, last modified): true skips the file |
listener |
RunListener: onFileStarted, onLineRejected, onFileFinished, onWarning
|
cancellation |
A CancellationToken; cancel() from any thread |
providedInputs |
Streams for a ProvidedStreamChannel
|
workingDirectory |
Working copies; defaults to the Tie's, else java.io.tmpdir
|
fileHook |
FileHook: onFileReceived, onFileCompleted
|
-
Validation. The definition, the SPIs and the context (variables, provided inputs, working directory) are validated as a whole. Any problem throws
ConfigurationExceptionwith every problem; no file is touched. -
Listing. Every input channel is listed before any file is processed. If one cannot be listed, the run fails with no file processed (
ImportResult.error, typeCHANNEL,AUTHENTICATIONorHOST_KEY). -
Selection. A listed file is a candidate when:
- its name is safe (no
/,\or..; otherwise it is ignored withUNSAFE_FILE_NAME); - its name matches the pattern of exactly one root layout (two layouts →
AMBIGUOUS_LAYOUT, the run fails); - with
watermark, its timestamp is strictly after the watermark of its layout; - the
alreadyProcessedpredicate, if any, returnsfalse.
The extension does not filter: a matching file with an extension the layout does not allow fails when its turn comes (
EXTENSION_NOT_ALLOWED). Two candidates of a layout with the same timestamp fail the run (DUPLICATE_TIMESTAMP). - its name is safe (no
-
Ordering. By layout (declaration order), then by timestamp (or by name without timestamp), then the provided inputs in list order.
-
Each file, in order:
-
RunListener.onFileStarted. - The extension and
maxFileSizeare checked; the file is downloaded or opened. -
FileHook.onFileReceivedwith the local copy. - With
VALIDATE_THEN_DELIVER: the file is copied to the working directory and fully read through the engine without delivering (the validation pass). - The delivery pass: every record goes through the engine and, if valid, to the consumer; children right after their parent.
-
FileHook.onFileCompletedwith the provisional result. - The post-success action (
MOVEorDELETE), then the result reports. - The watermark of the layout advances to the file's timestamp.
-
RunListener.onFileFinishedwith the final result.
-
-
A failed file stops the run: the following files are
NOT_PROCESSED.
STREAMING (default) |
VALIDATE_THEN_DELIVER |
|
|---|---|---|
| Reading | One pass | A validation pass, then a delivery pass |
| Structural error in the middle | Earlier records stay delivered; deliveredUpToLine says up to where |
Nothing is delivered |
maxRejectedPercent |
Not allowed | Checked after the validation pass, on engine rejections only |
| Transformers and resolvers | Called once per line | Called once per line and pass |
lineNumber is the ordinal of the record in its file, from 1: children are numbered right after their
parent. rowNumber is the physical row of CSV/XLSX files, used to find a problem in the file.
RecordConsumer consumer = record -> {
if (repository.exists(record.get("taxId", String.class))) {
return Outcome.rejected("duplicate"); // CONSUMER_REJECTED {reason: duplicate}
}
repository.save(record);
return Outcome.APPLIED;
};- It runs on the thread of the run, in file and line order.
-
TypedRecordgivesget(field),get(field, Class),has(field),rawValues(),layout(),sourceFileName(),lineNumber(),parentLineNumber()andchildrenDeclared(). - An exception: with
REJECT_LINE(default) the line is rejected (CONSUMER_FAILED) and the file goes on; withABORT_FILEthe file fails (CONSUMER) and the run stops. - The transaction is yours: TIE never undoes what was applied.
A line is rejected by the engine (field errors), by the consumer, or by its rejected parent
(PARENT_REJECTED, with onParentRejected = CASCADE). Rejections never stop a file unless a threshold
is exceeded:
-
maxRejected: the file fails when rejections exceed it (with2, the third rejection fails it). Every rejection counts, children and cascades included. -
maxRejectedPercent: withVALIDATE_THEN_DELIVER, the file fails before any delivery when the engine rejections of the validation pass exceed the percentage of all records read.
With JSON or XML, a child layout's records are delivered right after their parent, each with
parentLineNumber. When a parent is rejected, its children are rejected too (CASCADE, the default)
or processed anyway (PROCESS).
FileHook is only called for files of an input channel, never for provided inputs:
-
onFileReceived(ReceivedFile)— once per file, before any record, with the complete local copy. The copy is only valid during the call: copy it to keep it. -
onFileCompleted(FileResult)— after the last record and before the move, the report and the watermark. A good place to commit what your consumer did for this file.
An exception in either fails the file (HOOK): it is not moved, has no report and does not advance the
watermark. Records delivered before onFileCompleted stay delivered.
TIE keeps no state; combine any of:
-
Watermark: the timestamp of the last successful file per layout. Pass the stored map in, store
result.watermarks()out. It never advances past a failed or cancelled file. - Already-processed predicate: your own registry.
-
Post-success action:
MOVE(folder)orDELETEthe source after success. Nothing is moved or deleted unless the file succeeded. A failed move or delete is aPOST_ACTION_FAILEDwarning; the file still counts as applied.
CancellationToken.cancel() is checked before each root record: the record in progress finishes, the
next one is not delivered, the file is CANCELLED (not moved, watermark unchanged) and the following
files are NOT_PROCESSED.
ImportResult |
|
|---|---|
status |
SUCCESS, FAILED or CANCELLED
|
files |
One FileResult per candidate |
watermarks |
The new watermarks to store |
warnings |
Every warning of the run |
reports |
One ReportRef per report written |
error |
Set when the run failed before any file |
FileResult |
|
|---|---|
status |
SUCCESS, FAILED, CANCELLED, NOT_PROCESSED
|
linesRead, applied, rejected
|
Counters, children included |
deliveredUpToLine |
Last line processed through the consumer |
error |
FileError{type, message, rowNumber, lineNumber} |
warnings |
Warnings of this file |
timestamp |
The timestamp of the name (null for provided inputs and patterns without timestamp) |
| Situation | Outcome |
|---|---|
| Invalid configuration |
ConfigurationException, no file touched |
| No candidate files |
SUCCESS, no files, watermarks unchanged |
| A channel cannot be listed |
FAILED before any file |
| A line with errors | Rejected; the file goes on |
| Threshold exceeded | The file fails; the run stops |
| Structural or encoding error | The file fails; see the delivery strategies |
| The move, delete or report fails after success | Warning; the file is applied and the watermark advances |
onFileFinished throws |
LISTENER_FAILED warning; nothing else changes |
| A file with 0 records |
SUCCESS, moved, watermark advanced |
TIE is MIT-licensed · tie · tie-core · tie-spring-boot-starter · Changelog
Using TIE
How it works
Reference
Contributing