🚀 Version 0.8.0

Version 0.8.0

Version 0.8.0 brings a considerable set of improvements making it the best version to use for production applications. Several bug fixes, logging improvements and debugging measures have been improved as well as some of the issues surrounding fast-fuzzy.

What's new

• Local docs removed, We no longer serve the local openapi copy of the docs on the /docs endpoint. This became quite a burden to maintain and keep up to date in two places rather than just redirecting to https://docs.lnx.rs which now supports previous version via the ?version=<major>.<minor> flag e.g. https://docs.lnx.rs?version=0.8
• Queries are now followed the type: { <context> } pattern for payloads rather than having a base value field, and then having each query kind inconsistently require additional context. See below for an example.
• Delete by query mode, this will delete the matched documents based on your search query. This respects the limits and offsets you give the query so to delete all you may need to send several requests.
• Delete specific document endpoint, this allows you to delete a document via DELETE /indexes/:index/documents/:document_id.
• Indexes are now backed by sled for index metadata storage. This is incredibly useful going forward ensuring atomic behaviour when writing to and from disk with the fast-fuzzy spell correction system. This is a breaking change however, the data sub-folder of this new directory is a fully Tantivy compatible index. Theoretically, you can mount a custom program directly to this folder in order to perform actions.
• Fast-fuzzy garbage collection, this means that the frequency dictionaries are adjusted again when deleting documents which should prevent potential relevancy loss when working with a system that has a high update rate. The adjustments are only made when calling commit or auto-commit runs the operation.
• Tracing information, we have moved from log to tracing providing significantly more information for debugging and profiling in future releases, most notably this gives us the ability to add OpenTelemetry tracing later on.
• JSON log files (--json-logs) lnx can now produce line-by-line json logs where each object is a new log event. This is an amazing addition for anyone using an ingestion system or wanting to parse the logs.
• Pretty logs (--pretty-logs) for the extra flamboyant users. This makes reading the logs much easier / prettier but at the cost of taking up quite a few more lines per event. Looks nice though.
• Verbose logs (--verbose-logs) adds additional metadata to each log including thread-name and thread-id.
• Disable ASNI colours (--disable-asni-logs) This is mostly required for logging to files or ingestion systems. Doesn't look as nice though without the colours :(
• Logging Directory (--log-directory <dir>) This replaces the --log-file attribute and instead produces hourly log files in the directory, formatted using the aforementioned flags.
• RUST_LOG env var support. For more control over what to log and what not to log you can directly adjust the RUST_LOG env var which will override the log-level flag or env var. By default lnx will set this to be <log-level-flag>,compress=off,tantivy=info
• Snapshot support added snapshot support is now supported, this is essentially a wrapper around zipping the index files up and unzipping them again. You can set an automatic snapshot with the --snapshot-interval <interval> flag, take a single snapshot with the --snapshot subcommand, adjust the output directory with the --snapshot-directory <dir> flag and finally load snapshots with the --load-snapshot <file> flag. Each snapshot is generated with the name format of snapshot-<utc-timestamp>-lnx-v<lnx-version> it's important to note that older snapshots may not be compatible with future lnx releases. Although this should be avoided between most versions.

What's been fixed

• lnx now handles interrupt signals better and correctly cleans up indexes more reliably, this should prevent dangling locks in future.

What's been removed

• --log-file has been removed in favour of --log-directory powered by tracing.
• The storage type memory has been depreciated/downgraded, the system will now treat this the same as tempdir until it's removed in future versions. This came out of reasoning that realistically theirs not much difference between the two other than tempdir is more reliable on bigger indexes allowing the OS to page in and out of disk.
• The original --pretty-logs flag has been changed from disabling ASNI colours to Enabling pretty logging, this used to be pretty unintuitive before but now this actually does what the name suggests.

New Query Pattern

Before your query style would be

{
  "query": {
    "value": "foo",
    "kind": "fuzzy"
  }
}

now it's

{
  "query": {
    "fuzzy": { "ctx": "foo" }
  }
}

For more info see https://docs.lnx.rs/?version=0.8#tag/Run-searches