Skip to content

Running Imports

themarioga edited this page Nov 8, 2026 · 1 revision

Running Imports

ImportResult result = tie.importFiles(definition, context);

The context

ImportContext Meaning
consumer Required. Receives every valid record
variables Values of the {variable} markers of the name patterns
watermarks Last processed timestamp per layout, as returned by the previous run
alreadyProcessed A predicate on RemoteFile (name, size, last modified): true skips the file
listener RunListener: onFileStarted, onLineRejected, onFileFinished, onWarning
cancellation A CancellationToken; cancel() from any thread
providedInputs Streams for a ProvidedStreamChannel
workingDirectory Working copies; defaults to the Tie's, else java.io.tmpdir
fileHook FileHook: onFileReceived, onFileCompleted

Step by step

  1. Validation. The definition, the SPIs and the context (variables, provided inputs, working directory) are validated as a whole. Any problem throws ConfigurationException with every problem; no file is touched.

  2. Listing. Every input channel is listed before any file is processed. If one cannot be listed, the run fails with no file processed (ImportResult.error, type CHANNEL, AUTHENTICATION or HOST_KEY).

  3. Selection. A listed file is a candidate when:

    • its name is safe (no /, \ or ..; otherwise it is ignored with UNSAFE_FILE_NAME);
    • its name matches the pattern of exactly one root layout (two layouts → AMBIGUOUS_LAYOUT, the run fails);
    • with watermark, its timestamp is strictly after the watermark of its layout;
    • the alreadyProcessed predicate, if any, returns false.

    The extension does not filter: a matching file with an extension the layout does not allow fails when its turn comes (EXTENSION_NOT_ALLOWED). Two candidates of a layout with the same timestamp fail the run (DUPLICATE_TIMESTAMP).

  4. Ordering. By layout (declaration order), then by timestamp (or by name without timestamp), then the provided inputs in list order.

  5. Each file, in order:

    1. RunListener.onFileStarted.
    2. The extension and maxFileSize are checked; the file is downloaded or opened.
    3. FileHook.onFileReceived with the local copy.
    4. With VALIDATE_THEN_DELIVER: the file is copied to the working directory and fully read through the engine without delivering (the validation pass).
    5. The delivery pass: every record goes through the engine and, if valid, to the consumer; children right after their parent.
    6. FileHook.onFileCompleted with the provisional result.
    7. The post-success action (MOVE or DELETE), then the result reports.
    8. The watermark of the layout advances to the file's timestamp.
    9. RunListener.onFileFinished with the final result.
  6. A failed file stops the run: the following files are NOT_PROCESSED.

Delivery strategies

STREAMING (default) VALIDATE_THEN_DELIVER
Reading One pass A validation pass, then a delivery pass
Structural error in the middle Earlier records stay delivered; deliveredUpToLine says up to where Nothing is delivered
maxRejectedPercent Not allowed Checked after the validation pass, on engine rejections only
Transformers and resolvers Called once per line Called once per line and pass

Line numbers

lineNumber is the ordinal of the record in its file, from 1: children are numbered right after their parent. rowNumber is the physical row of CSV/XLSX files, used to find a problem in the file.

The consumer

RecordConsumer consumer = record -> {
    if (repository.exists(record.get("taxId", String.class))) {
        return Outcome.rejected("duplicate");           // CONSUMER_REJECTED {reason: duplicate}
    }
    repository.save(record);
    return Outcome.APPLIED;
};
  • It runs on the thread of the run, in file and line order.
  • TypedRecord gives get(field), get(field, Class), has(field), rawValues(), layout(), sourceFileName(), lineNumber(), parentLineNumber() and childrenDeclared().
  • An exception: with REJECT_LINE (default) the line is rejected (CONSUMER_FAILED) and the file goes on; with ABORT_FILE the file fails (CONSUMER) and the run stops.
  • The transaction is yours: TIE never undoes what was applied.

Rejections and thresholds

A line is rejected by the engine (field errors), by the consumer, or by its rejected parent (PARENT_REJECTED, with onParentRejected = CASCADE). Rejections never stop a file unless a threshold is exceeded:

  • maxRejected: the file fails when rejections exceed it (with 2, the third rejection fails it). Every rejection counts, children and cascades included.
  • maxRejectedPercent: with VALIDATE_THEN_DELIVER, the file fails before any delivery when the engine rejections of the validation pass exceed the percentage of all records read.

Nested records

With JSON or XML, a child layout's records are delivered right after their parent, each with parentLineNumber. When a parent is rejected, its children are rejected too (CASCADE, the default) or processed anyway (PROCESS).

Hooks

FileHook is only called for files of an input channel, never for provided inputs:

  • onFileReceived(ReceivedFile) — once per file, before any record, with the complete local copy. The copy is only valid during the call: copy it to keep it.
  • onFileCompleted(FileResult) — after the last record and before the move, the report and the watermark. A good place to commit what your consumer did for this file.

An exception in either fails the file (HOOK): it is not moved, has no report and does not advance the watermark. Records delivered before onFileCompleted stay delivered.

Duplicate protection

TIE keeps no state; combine any of:

  • Watermark: the timestamp of the last successful file per layout. Pass the stored map in, store result.watermarks() out. It never advances past a failed or cancelled file.
  • Already-processed predicate: your own registry.
  • Post-success action: MOVE(folder) or DELETE the source after success. Nothing is moved or deleted unless the file succeeded. A failed move or delete is a POST_ACTION_FAILED warning; the file still counts as applied.

Cancellation

CancellationToken.cancel() is checked before each root record: the record in progress finishes, the next one is not delivered, the file is CANCELLED (not moved, watermark unchanged) and the following files are NOT_PROCESSED.

The result

ImportResult
status SUCCESS, FAILED or CANCELLED
files One FileResult per candidate
watermarks The new watermarks to store
warnings Every warning of the run
reports One ReportRef per report written
error Set when the run failed before any file
FileResult
status SUCCESS, FAILED, CANCELLED, NOT_PROCESSED
linesRead, applied, rejected Counters, children included
deliveredUpToLine Last line processed through the consumer
error FileError{type, message, rowNumber, lineNumber}
warnings Warnings of this file
timestamp The timestamp of the name (null for provided inputs and patterns without timestamp)

What happens when…

Situation Outcome
Invalid configuration ConfigurationException, no file touched
No candidate files SUCCESS, no files, watermarks unchanged
A channel cannot be listed FAILED before any file
A line with errors Rejected; the file goes on
Threshold exceeded The file fails; the run stops
Structural or encoding error The file fails; see the delivery strategies
The move, delete or report fails after success Warning; the file is applied and the watermark advances
onFileFinished throws LISTENER_FAILED warning; nothing else changes
A file with 0 records SUCCESS, moved, watermark advanced

Clone this wiki locally