Skip to content

Design Notes and FAQ

themarioga edited this page Nov 8, 2026 · 2 revisions

Design Notes and FAQ

Why stateless?

Applications already have a database. TIE takes the state it needs (watermarks, an "already processed" predicate) as input and returns the new state, so you keep it next to your data, in the same transaction if you like. TIE never writes to a database and keeps nothing between runs.

How much memory does it use?

Memory is bounded by one record, not by the file: every format is read and written in streaming. A JSON or XML record is materialized with its children; the file never is. XLSX is read from a file with random access and streamed sheet by sheet. The test suite imports 1 GB CSV and JSON files and a 200,000-row XLSX with a 256 MB heap.

Is Tie thread-safe?

Yes. A Tie has no state between executions: share one instance and run different integrations concurrently. Each run is synchronous: one file after another, one record after another, on the calling thread.

Can two runs process the same directory at the same time?

TIE does not prevent it. Coordinate runs yourself (a lock, a single scheduler instance).

What if my process crashes in the middle of a file?

There is no resume. Records delivered before the crash stay applied, the file is not moved, and the watermark you stored is still the previous one, so the next run processes the file again. Make your consumer idempotent, or commit per file in FileHook.onFileCompleted and use the alreadyProcessed predicate.

When should I use VALIDATE_THEN_DELIVER?

When a file must be applied entirely or not at all with respect to structural errors and rejection percentages: nothing is delivered unless the whole file reads correctly and the percentage of engine rejections stays within the threshold. It reads the file twice.

Why does a name pattern have no wildcards?

So that the timestamp, the variables and the layout of a file are unambiguous, and two layouts can never claim the same file silently: a name that matches two layouts fails the run.

Which versions does it use?

Java 25, Jackson 3, Apache POI 5.5, Commons CSV 1.14, JSch 2 (com.github.mwiede), SLF4J 2 and, for the starter, Spring Boot 4.1.

Why are the modules in separate repositories?

Each module has its own repository, issues and history, while the aggregator keeps the parent pom, the examples and one version for everything. They are linked as git submodules; see Repositories and Building for cloning, changing a module and releasing.

Is there a module-info?

No. POI depends on a library without a stable module name, so tie-core is an automatic module named org.tie (and the starter org.tie.spring). Implementation classes live under org.tie.internal and are not part of the API.

Clone this wiki locally