You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Analyze > Audit report: header checks (page count in the header vs. file, change counter, reserved bytes, encoding, auto-vacuum), use of every page (b-tree, overflow, freelist, pointer map, unreferenced), freelist consistency, structural anomalies (loops, invalid pointers), traces of secure_delete (freeblocks / freed pages wiped with zeros), rollback journal (hot / persisted header, page images, checksums); CSV export
WAL frames are verified like SQLite does on recovery (header checksum, salts, cumulative frame checksums): state per frame current / superseded / uncommitted / stale (earlier generation) / invalid, comparison with mxFrame and nBackfill of the -shm file; the "Transactions" tab of the WAL panel uses the verified states
WAL time travel: the database state after every committed transaction is written to a new file (with manifest) and opened as a separate database; pages that may be newer because of an earlier checkpoint are reported
Forensic analysis > Recovered records: class (older version, re-inserted, deleted, unreferenced page, index entry), where it was found (freeblock, unallocated area, freelist page, ...) and confidence high / medium / low with the reasons for every recovered record; also in the table context menu ("Classification and confidence...")
Fix: NodeQL / SQL analyzer and export to SQLite failed with "near "REFERENCESconversations": syntax error" for columns without a declared type but with a column constraint (e.g. conversation_id REFERENCES conversations(id)); such columns now get BLOB affinity, column names are quoted in the generated CREATE TABLE / INSERT statements (keywords such as "order" work as column names)
Fix: in UTF-16 databases (UTF-16LE/BE), tables and indexes whose CREATE statement has at most 28 characters (e.g. CREATE TABLE u(a,b)) were not recognised; their rows were not shown under the table (found while checking the NIST SFT-01/02 test cases)
Lighthouse: large databases (several GB) can be opened directly and read-only (File > "Open in Lighthouse (read-only, large databases)..."; offered automatically for Lighthouse databases over 512 MB, also after an ETSI import). The map is streamed from the file instead of recovering every record into memory and copying it into an in-memory database; complete rows are read on demand, presets and query assistant run on the file and use the importer's indexes (2.5 GB / 5.7 million records: about 2.2 GB heap). The file is opened with mode=ro (and immutable=1 if there is no -wal content) and is not modified; the helper views of the query assistant are TEMP views. Deleted records are only available in the normal FQLite analysis.
Lighthouse presets: the self-join analyses no longer compare every record with every other record (same results, verified against the previous queries): recurring pairs use 30-minute buckets per cell site, twins compare cell-set signatures, identifier analyses (companions, partners, nearby identifiers) only join the identifier's sessions / cell sites, communication in the same cell only the sessions with two numbers; on databases with the importer's indexes, grouping by number/IMSI/IMEI sorts instead of walking the index, and the assistant views call_pairs/sms_pairs no longer join through idx_source_file (5.7 million records: every preset under 45 s, before up to more than 30 minutes)
memory: recovered rows are stored compactly (one exact-size array per row instead of an ObservableList wrapper with change builder and ArrayList), hexdump rows without raw bytes share one instance, equal cell values share one String (pooled per table column, columns with mostly unique values are left alone), line numbers are shared; the post-processing passes (stale copies, older versions) only index the live records the deleted records actually ask for. Results are byte-identical (all rows and hexdumps compared). ETSI database with 400,000 records: 1.4 GB instead of 3.7 GB in the GUI, import 38 s instead of 54 s
large databases (from 256 MiB, Settings > Display & limits, 0 = off): the live entries of index b-trees are loaded when the index table is opened (or for an export, search or timeline) instead of while loading. Deleted and freelist index entries are found and shown right away, exactly as before; the post-processing gets the live entries of one index at a time. Not used if a WAL archive or rollback journal exists. ETSI database with 400,000 records and 8 indexes: 0.6 GB instead of 1.4 GB in the GUI, same loading time
faster loading: the search for schema entries reads the file block by block instead of byte by byte (same hits); indexes without deleted entries are not read a second time for the post-processing
Lighthouse (ETSI) databases written by FQLite's importer are recognised (response_records, request_metadata, import_log): the search for deleted records is skipped - the evidence is the imported XML file - and the records are checked against the import log instead (rows per source file); a note shows the result, a warning if the counts differ (database modified after the import). Settings > Display & limits: "Search deleted records in Lighthouse (ETSI) databases" = 1 restores the full search. 1.7 million records: 39 s instead of 112 s
Lighthouse, read-only direct mode: records of the same cell share their coordinate and cell objects, equal timestamps one object, display-only fields are kept in a separate object only when set; IMSI/MSISDN/IMEI values are shared again (the check for unique columns now decides after 100,000 values at a hit rate below 5 %); large tables are read in parallel in rowid ranges (up to 8 connections), "yyyy-MM-dd HH:mm:ss" timestamps get a fast parser, ULI decoding is cached, end_time and the raw coordinates are only read when they are needed as a fallback, sorting by time uses a numeric key. All fields of all points are identical (compared on 5.7 million records). 5.7 million records: 0.7 GB instead of 2.2 GB heap; 1.7 million records: 8 s instead of 18 s on 2 cores
Lighthouse overview for very large databases (from 2 million records, Settings > Display & limits, 0 = always): instead of loading every record, one pass over the file builds aggregates - one marker per location with the number of records, records per location and hour, records per minute - and single records are read from the file when shown. The map draws the locations weighted (clusters add up the numbers; tooltip; click shows the first record, double-click loads the records of a location, on several locations zooms in); "Map: visible period" in the timeline shows the numbers of the visible time span. The timeline shows every record in time order but reads only the visible page through the index on start_time. Timeline filters, picked identifiers, query templates and location loads run as SQL on the file: results up to 500,000 records (setting) are loaded completely with all functions, larger ones as an overview of the result; "Reset" returns to the overview. IMSI path and IMEI storyboard read the records of the chosen identifier through its index; the dataset cell-site layer and the MOC/MTC map use the aggregates (call list up to 500,000 calls). 5.7 million records: overview in about 15 s with under 100 MB for the aggregates (GUI after opening 0.17 GB heap instead of 0.72 GB for the points alone); aggregates, timeline pages and window counts checked against the full load. Query results on directly opened databases are read as compact points from the file; free-form queries are limited to 500,000 rows in overview mode.
Lighthouse, large databases: queries of the assistant with subqueries ("Wechsler", "Wanderer") no longer take many minutes. The condition is evaluated once and its rowids are kept in a shared in-memory database (the file stays untouched); counting, loading, timeline pages and identifier lists use them. start_time ranges covering more than 5 % of the records are checked during a sequential scan instead of reading every row through the index in random order, the templates group with GROUP BY +column (sort instead of walking the index) and the Wanderer template checks the identifier while scanning (same results). 5.7 million records: Wechsler over four weeks 9 s instead of 33-73 s, Wanderer over four weeks 16 s for the selection.
Lighthouse, Wanderer query on a database opened directly: the table below the map is computed first and the records are selected by membership in its IMSIs, so the whole table is no longer grouped twice (same records, checked); an overview of a selection checks the selected rowids in memory (bit set) instead of in SQL. 5.7 million records, Wanderer over four weeks (5.3 million records): 35 s instead of 56 s until the result is shown.
Lighthouse, Statistics: "Restrict time range" now has a calendar and hour/minute/second fields instead of free text; the range is preset with the first and last start_time of the records (days outside are greyed out in the calendar), the date can still be typed.
Lighthouse, MOC/MTC tab: "Restrict time range" with the same date/time fields (preset with the time span of the data) and "Apply"; map and call list show only the calls of the range. Single records are filtered in memory; on the overview of a large database the calls of the range are read from the file (completely up to the detail limit, else as an overview).
Lighthouse overview cache: the overview of a large database is stored in ~/.fqlite/lighthouse-cache (not next to the database, evidence folders stay untouched) and read from there when the same, unchanged file is opened again (path, size and modification time of the file and of a -wal file; number of records and largest rowid are checked again; the 20 most recently used overviews are kept). 5.7 million records: overview in about 2 s instead of 16 s; checked against a freshly built overview.
File > Open Recent: the last 10 opened databases (stored in ~/.fqlite/recent.txt); an entry opens the file again the way it was opened - FQLite analysis or Lighthouse (read-only, marked "(Lighthouse)"); files that no longer exist are shown disabled; "Clear list" empties it.
Fix (Lighthouse query assistant): a time range named with one date only was ignored when the model returned only one end or a range of another day, so e.g. "Zeige mir alle Wanderer vom 10.04. an." returned the Wanderer of the whole period. "vom/am DATE" now means that day, "ab/seit DATE" from that day to the end of the data, "bis (zum) DATE" from the start of the data; times of day the model extracted are kept.
Lighthouse query assistant: "Wanderer in allen Funkzellen" (also "in jeder Funkzelle", "in sämtlichen Zellen") shows the IMSIs seen at every cell site of the time range; if none was, those with the most cell sites, and the status line says so (before: all Wanderer).
Fix (Lighthouse, database opened directly): the window could hang for the duration of a long query (spinning cursor, no reaction to input). Tooltips and the detail panel read the complete row of a record through the shared database connection on the UI thread; while a preset or a query of the assistant was running on that connection, the lookup waited until it had finished (e.g. 32 s). Rows are now read through a separate connection; the connections are released when the window is closed.
Lighthouse (database opened directly): before a query of the assistant or an analysis of the Statistics tab, its duration is estimated - records of its start_time range counted through the index, passes (every SELECT, also in subqueries), read rate measured on this computer while building the overview (~/.fqlite/lighthouse-cache/scan-rate.txt); an upper bound, since the number of results is only known afterwards. From 30 s (Settings > Display & limits, 0 = never) a question shows the affected records and the estimated duration: "Run anyway", "Restrict time range..." (date/time fields preset with the current range or the data range; the query then runs with it) or "Cancel".
Lighthouse: running queries can be cancelled - "■ Cancel" next to "Ask" (query assistant, timeline filters, location loads, MOC/MTC time range) and the run button of the Statistics tab while an analysis runs. The statements are interrupted (sqlite3_interrupt), no further step starts, the data shown before stays; no error dialog for a cancelled query.
Fix (Lighthouse): the IMSI and IMEI selectors (path / storyboard, main map and MOC/MTC tab) blocked the application for seconds when opened on databases with many identifiers - the list measured every entry (200,000 IMSIs: about 4 s). The selectors now list at most 1,000 identifiers; the field is editable and typing part of an identifier filters all of them, Enter takes an exact or unique match. The pick list of the timeline filter also lists at most 1,000 identifiers (the filter field searches all). 200,000 IMSIs: the list opens in 0.1 s.
Lighthouse query assistant: filter queries ("Teilnehmer an der Funkzelle LAC=…", time range, calls/SMS, country, free SQL of the form SELECT * FROM response_records WHERE …) now show a participants list below the map - one row per IMSI with phone number(s), device(s) (IMEI), number of records, first and last record, sorted by number of records; a click draws the path of the IMSI, cells can be copied (Ctrl/Cmd+A, Ctrl/Cmd+C). Before, the result was only on map and timeline. The list is built from the loaded records (no second pass over the file); only for a result shown as an overview (above the detail limit) it is computed by a query, which can be cancelled. Checked against the equivalent GROUP BY query (3,367 and 122,794 IMSIs identical).
Fix (Lighthouse, Statistics): an analysis with a very large result (e.g. "Recurring phone-number pairs" on a dense cell dump: tens of millions of pairs) kept every row as a table row, and the memory use rose to more than 30 GB. The table now keeps the first 100,000 rows in sort order (the analyses sort by relevance, e.g. number of shared cell sites); the remaining rows are only counted and the status line shows the total. 300,000 records at 20 cell sites, one day (4.4 million pairs): peak 0.33 GB instead of 3.5 GB.
Lighthouse, Statistics > "Recurring phone-number pairs" counts meetings instead of record pairs. Before, every pair of records within 30 minutes at the same cell counted, so two numbers with several records each during a single stay already had a count > 1 and almost every pair that met once was listed (a dense cell dump gave tens of millions of rows). Now a meeting is a period in which both numbers had records within 30 minutes of each other at the same cell site; a gap of more than 30 minutes starts a new meeting (moving together through several cells is one meeting). Listed are pairs with at least two meetings, with the number of meetings, the number of shared cell sites and the first and last meeting, sorted by meetings. Checked against an independent implementation (identical results). 300,000 records at 20 cell sites, one day: 51,826 instead of 4.4 million pairs, about as fast as before (99 s instead of 87 s).
Lighthouse, Statistics > "Most frequent companion of an identifier" counts meetings as well (same definition as the recurring pairs: records at the same cell site within 30 minutes, a gap of more than 30 minutes starts a new meeting); columns: companion number/IMSI/IMEI, number of meetings, shared cell sites, first and last meeting, sorted by meetings. Before, a single meeting with several records on both sides gave counts of up to 100 in the column "shared cell sites". Only the needed columns of the records at the identifier's cell sites are kept (before: every column of every record at those cells), and records are only compared within neighbouring 30-minute buckets. Checked against an independent implementation (three cases identical).
Lighthouse, Statistics > "Other identifiers near an identifier" is much faster (same result rows, compared on four test databases with and without time range): for the part "same cell site (±30 min.)" only the needed columns of the records at the identifier's cell sites within 30 minutes of its first and last record are kept (before: every column of every record at those cells - on a cell dump with few cell sites nearly the whole table), and each record of the identifier is compared only with the records of the same and the neighbouring 30-minute buckets of its cell instead of with every record of the cell. 5.7 million records: 3.7 s instead of 31 s; 2 million records at 10 cell sites, identifier with 1,000 records: 19 s instead of 179 s.
Lighthouse, Statistics: "Show result in Map & Timeline" (on by default) - after an analysis whose result names identifiers (Wanderer, Wechsler, cross-hits, recurring pairs, twins, number blocks, country filter, the analyses of an entered identifier, ...), map and timeline show the records of these phone numbers/IMSIs/IMEIs within the analysis' time range (Wanderer: the IMSIs only); the entered identifier is included. Overview analyses (all numbers, party role, participants per cell, connection type, cell sites, all identifiers, country overview) leave the map unchanged. On a database opened directly the records are selected in one pass over the file with the identifiers held in memory (also those beyond the 100,000 rows of the table, up to 2 million identifiers); large results are shown as an overview; the load can be cancelled. Checked: Wanderer of one day 116,467 records of 49,158 IMSIs, companions of a number 2,566 records - equal to the counts by SQL.
Lighthouse: "Export to CSV…" in the context menu of the result lists (Statistics, participants, Wanderer, Wechsler, same cell site, MOC/MTC calls, movement history of an IMSI): all rows with the column headers, separator from the settings (CSV_SEPARATOR), UTF-8 with byte order mark (umlauts in spreadsheet programs). The Statistics export always contains the complete result, also when the table shows only the first 100,000 rows (the rows are written to a temporary file while the result is read; 200,000 Wanderer: 200,001 lines).
Fix (Lighthouse query assistant): a duration in the request was ignored by the fixed templates, so e.g. "Zeige mir alle Anrufe vom 02.03. die länger als 60 Minuten gedauert haben" returned every call of that day. The duration is now read from the text ("länger als / mehr als / über / mindestens …", "kürzer als / weniger als / unter / höchstens / bis zu …", "zwischen 5 und 10 Minuten", "Dauer > 2 min", English equivalents; seconds, minutes, hours, "eine halbe Stunde", "1 Stunde und 30 Minuten") and added to the calls/SMS, time-range, country and cell-site templates as a condition on duration_seconds (the durationTime of the record; end minus start if it is missing); the status line shows it, e.g. "Dauer > 1 h". Checked: calls of one day longer than 6 minutes, 4,429 records, equal to the count by SQL.
Fix (ETSI import): voice records with the communication time once per record (telephonyServiceUsage/communicationTime next to partyInformation, e.g. 1&1) were imported without start time, end time and duration - in a 1&1 cell dump all 1,200 call legs; they now take the record's time. The user location information of these records (GTPv1 header with TAI and ECGI, flags 0x18) was decoded as MCC 812 / MNC 356; the cell is now read from the ECGI (262/23 with eNB and cell). ETSI databases imported before need to be imported again.
Lighthouse, Statistics: two heuristics for shock calls (Schockanruf, Enkeltrick). "Shock-call suspicion: continuous calls (caller)" joins calls of one number to one line that follow each other with at most 2 minutes pause (duplicate records of the same call, e.g. IMS and switch, are merged) and lists chains of at least 20 minutes with points for caller abroad, withheld number, short/official number (110), landline called, redials, over 60 minutes and a foreign transit (incomingTrunkCountry). "Shock-call suspicion: possible collectors (cell dump)" rates every subscriber of a cell dump: foreign SIM or number, contacts abroad, open line of 15 minutes or more abroad, signal call of up to 5 s abroad, arrival 30 minutes or more after the start of the time range, short stay; IoT/M2M cards are left out. Both list the indications per row; results can be shown in Map & Timeline. Test case (cell dumps of four providers, 33,000 records, time range = the shock call): the caller is the only chain with 6 points, the two collector candidates found by hand are on top (8 and 6 points, 6 rows in total).
Shock-call heuristic (caller) extended: the chains are formed per called line, also across changing caller numbers (spoofed numbers change between calls), with +2 points for changing caller numbers and +2 for a German mobile number delivered by a carrier that is no mobile operator (A_DOMAIN_SPV_APN, e.g. ims.colt.net: number probably spoofed); new column "Delivered by"; list from 4 points. Second test case (victim called 78 minutes by a German mobile number and then a Stuttgart number, both via ims.colt.net): found with 7 points, before not at all; first test case unchanged, no other hits in either.
Fix (ETSI import): a call with both parties that additionally carries a nationalTelephonyServiceUsage (Telekom VoLTE: internetAccess with IMEI/IMSI/APN only) was imported as one data row without party role and without the other party; such records are now imported as calls with one row per party (Telekom cell dumps of the test cases: 51 and 78 calls).
Shock-call heuristics, third test case (victim called 7 minutes by a Greek number, cell dumps at the victim's address 1 hour later): the caller analysis also lists shorter calls (from 5 minutes) to a landline from a foreign, withheld, short or probably spoofed number; changing caller numbers only count for a landline. The collector analysis looks for such shock callers in the whole database (e.g. the request for the victim's line imported together with the cell dumps) and gives +3 points to subscribers with the same country and network prefix (e.g. 306… Greek mobile) and +4 for contact with the caller: five Greek SIMs that appeared at the address within 36 minutes come out on top. Call legs are paired by record number and source file (two files of one request had the same record numbers and produced wrong partners). All three test cases: only the shock caller in the caller analysis.
ETSI import: if a record gives start time and duration but no end time (e.g. provider 49124), the end time is computed.
Fix (shock-call heuristic, collectors): sender names were rated as subscribers - e.g. "Yettel" (an operator SMS to a Serbian SIM) appeared with IMSI/IMEI "null", cell "0.00000, 0.00000" and 6 points. In databases analysed by FQLite (copied into an in-memory database) missing values are '' or the text 'null' instead of NULL; they now count as missing, coordinates must be numbers other than 0/0, and only numeric phone numbers are rated. A signal call is a voice call only (an SMS has duration 0, too). The "short stay" indication is now named "at most 3 records (possibly briefly in the area)"; new column "Time span (minutes)" from first appearance to the end of the last record. Same results on the test cases with NULL and with ''/'null' values.
Fix (Lighthouse/ETSI databases): the import log check reported "Records differ from the import log" when an imported file had no records (e.g. an empty provider response: 0 in the database, 0 in the import log) - such a file has no rows in response_records and was compared as missing. Files without records now count as 0; a real difference (e.g. one deleted record) is still reported.
LLM settings: all fields of the LLM dialog take effect. GPU: Automatic (Metal on Apple Silicon, CUDA with an NVIDIA GPU, else CPU) / CPU only / all layers / number of layers; temperature, top P, top K, max. tokens (limited to what the 4096-token context leaves after the prompt), frequency and presence penalty apply to every SQL generation (free-form, ETSI-CDR and the retry with the SQLite error); the classification of the question always runs with temperature 0. Changing the model file or the GPU setting reloads the model before the next question (generations and reloads are serialised, so a model is never closed while generating). Defaults are the decoding the models were evaluated with (temperature 0, top P 0.9, top K 40, max. 384 tokens, no penalties; results unchanged for ETSI databases, the free-form generator no longer uses presence penalty 1.1). Configuration files of the old dialog keep only their model path, since their other values were never used (e.g. gpu_layers=0 would have switched Apple Silicon to CPU). -Dfqlite.llm.gpuLayers still takes precedence.
LLM settings are also available in File > Settings... (tab "LLM", same controls as the LLM Configuration dialog; both write ~/.fqlite/fqlitellm-config.properties). After "Download LLM..." the downloaded model file is entered and saved automatically.