Microsoft Access data recovery support. - #116
Conversation
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Our vendored libmdb sources (libmdb/*.c) use a newer MdbHandle struct that includes the 'locale' field. The system mdbtools-dev on Ubuntu Jammy ships an older version without this field, causing 'MdbHandle has no member named locale' errors. Since all libmdb .c files are compiled directly into BinDocument, there is no need to link against the system libmdb (-lmdb). Remove the linkage to eliminate the struct mismatch. Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
iconv.c has two code paths: - #ifdef HAVE_ICONV: uses iconv_t (standard POSIX, no locale field needed) - #else: uses mdb->locale (requires newer MdbHandle with locale field) The system mdbtools-dev on Ubuntu Jammy has an older MdbHandle without the 'locale' field. Defining HAVE_ICONV selects the iconv_t code path which is the correct choice on Linux (iconv is always available via glibc) and avoids the missing struct member error entirely. Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Two fixes for iconv.c build errors with HAVE_ICONV defined: 1. Add 'ICONV_CONST=' compile definition - on Linux glibc, iconv() takes non-const char**, so ICONV_CONST must be empty 2. Add libmdb/mdbtools.h shim that redirects to our vendored mdbtools/mdbtools.h - ensures libmdb sources always use our version of MdbHandle (with iconv_in/iconv_out fields) regardless of what system mdbtools-dev provides Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
BinDocument's CMakeLists.txt has always listed these 20 libmdb/*.c files as sources, and MdbEngine was switched to link against them instead of the system mdbtools-dev package, but the .c files themselves were never committed -- only the accompanying headers. Vendored verbatim from the official mdbtools v1.0.1 release, matching the version string already present in the committed mdbver.h. Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Two separate build-hygiene/licensing fixes for the database engines: - DuckDB was vendored as raw prebuilt binaries (libduckdb.so 51MB, libduckdb_static.a 63MB) committed directly to the repo. Switch to vcpkg's duckdb port instead, matching how sqlite3 is already pulled in. Removes ~114MB from history; DuckDbEngine.cpp only ever used the public C API (duckdb.h), so no vendored C++ header was needed either. - mdbtools (LGPLv2+) was compiled directly into BinDocument's static library. Static-linking an LGPL library into an AGPL application without satisfying LGPL section 6 (relinking) is a licensing risk. Build it as its own shared library (mdbtools.so) and link BinDocument against it dynamically instead -- the standard way to consume an LGPL dependency from a differently-licensed application. Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
The duckdb port exports find_package(DuckDB CONFIG REQUIRED) (capitalized, per its own usage file) with a plain, unnamespaced 'duckdb'/'duckdb_static' target -- not 'duckdb::duckdb'. Verified against the port's usage file on the build server rather than guessed. Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
…b move x2tlib links BinDocument (a static library) and therefore needs its own direct links to duckdb and mdbtools -- PRIVATE dependencies of a static library don't propagate to its consumers. It still had an IMPORTED duckdb pointing at the now-deleted vendored .so, plus a reference to a target named mdb that was never actually defined anywhere. Point both at the real targets and drop the now-invalid manual libduckdb.so copy step. Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
The project sets -fvisibility=hidden globally via set_default_options(), which silently hid every mdb_* C API symbol from mdbtools.so's exported symbol table -- the functions were compiled in but not linkable from outside the .so, causing 'undefined reference to mdb_open' etc. at the final executable link step despite mdbtools being correctly listed as a dependency. Override back to default visibility for this target, since its entire purpose is external linkage. Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Hendrik Leidinger <hendrik.leidinger@nextcloud.com> Assisted-by: Claude Code:Opus 4.8
Add x64-linux-v2 (default) and x64-linux-baseline overlay triplets and thread ARCH_TRIPLET/ARCH_MARCH_FLAGS build args through the core cmake invocation, so downstream builders can compile against plain x86-64 without the team maintaining a second officially-supported toolchain. Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
…d-migration Signed-off-by: Peter P. Lupo <pplupo@gmail.com> # Conflicts: # .docker/core.bake.Dockerfile
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
…arch-baseline Signed-off-by: Peter P. Lupo <pplupo@gmail.com> # Conflicts: # .docker/core.bake.Dockerfile
…d-migration Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
|
Thanks for putting this together, @pplupo and for the care that clearly went into it. The mdbtools LGPL handling and the engine abstraction are solid pieces of work. But my personal read is that this isn't a fit for core. Pulling database recovery into core means a bigger shipped build, a new parser of untrusted binary input in our default attack surface, and ongoing maintenance of the DB engines for something we don't currently have a use case for. That's a real cost, and I don't think it's one we should take on right now. I'll raise this PR at our next team meeting and put the scope question to everyone. I'll follow up here once we've talked it through. |
|
@chrip Thanks for taking the time to review this. Regarding the attack surface, the risk is heavily mitigated by how the abstraction works. The parser strictly extracts raw data from these files; it does not execute any embedded logic or binaries. The extracted information is purely treated as a data source and fed directly into the spreadsheet engine. On the question of the use case, this feature directly addresses the core philosophy of Euro-Office: true data sovereignty. For users with decades of critical records locked in MS Office Access mdb files, we need to provide an independent recovery path to break that vendor lock-in. There is a demonstrated market need for this, which is why alternatives like LibreOffice already support it. It is a powerful enabler for users migrating to our ecosystem. IMHO, this is a must-have. I appreciate you bringing this to the team meeting, and I look forward to hearing the team's thoughts. |
|
@pplupo This PR prompted a discussion about what Euro-Office stands for and the direction we want to take it in. As the outcome, I was convinced that we shouldn't reject a feature this valuable. Freeing data from formats like Access is exactly what it is about, and we build with the community, so a useful feature that someone is willing to implement belongs here. The one issue is timing. We're mid-stabilization on the project right now and don't want to land a change this size on an unstable trunk, so we're putting it on hold rather than merging now. We'll keep the PR open and pick it back up once things are stable. |
Adds a BerkeleyDbEngine implementing the existing IDatabaseEngine
interface, alongside SqliteEngine/DuckDbEngine/MdbEngine. Berkeley DB
is a schema-less key/value store, so each table is exposed with a
synthetic {Key, Value} schema and no primary/foreign keys.
- New extension .bdb, plus a content sniff (SQLite's 16-byte magic
header) to disambiguate the already-ambiguous .db extension between
SQLite and Berkeley DB.
- Registers AVS_OFFICESTUDIO_FILE_SPREADSHEET_BDB and wires it through
OfficeFileFormatChecker2's detection/extension tables.
- Links unofficial::berkeleydb::libdb (vcpkg, pinned to 18.1.40, which
is licensed AGPLv3 same as this application) into BinDocument and
x2tlib, mirroring the existing sqlite3/duckdb integration.
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
vcpkg's berkeleydb port only carries the old 4.8.30 (Sleepycat License) release; there is no port revision for the 6.0.20+ line that Oracle relicensed under AGPLv3 (the same license as this application), which is what the BerkeleyDB support was built against. Vendor the upstream source verbatim under DatabaseEngines/berkeleydb/ (matching the existing mdbtools precedent for vendoring a dependency vcpkg can't supply) and drive its own autotools build via ExternalProject_Add, producing an imported 'berkeleydb' target that BinDocument and x2tlib link against in place of the vcpkg package. Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
…I utils, GCC strictness) Three issues surfaced by an actual build attempt: - The repo-wide '**/Makefile.*' gitignore rule silently dropped dist/Makefile.in from the previous commit despite 'git add' reporting success, breaking configure's config.status. Force-add it, and vendor the one other file config.status needs that lives outside src/dist (test/tcl/include.tcl), rather than the whole test/ tree. - BDB's default 'all'/'install' targets also build/install the CLI utilities (db_archive, db_dump, ...), whose sources under util/ were deliberately not vendored (unneeded for a read-only import engine). Build/install the library-only targets instead. - BDB 18.1.40's K&R-style C trips -Wincompatible-pointer-types, which this GCC treats as a hard error by default; pass CFLAGS=-Wno-error=incompatible-pointer-types, the standard workaround for building this codebase with a modern compiler. Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Qt 6.10.1 (fetched prebuilt via aqtinstall) requires libicuuc.so.73; core's own vendored ICU was pinned to 74, so any process descending from the Qt-linked desktop app (which loads both versions into the same process tree) ends up with both ICU 73 and 74 loaded side by side. This corrupts state in a way that's hard to pin down exactly, but is 100% reproducible: x2t, invoked via fork/exec as a direct child of DesktopEditors, silently fails to parse its own task XML despite reading the correct bytes (confirmed via strace) -- while the identical operation succeeds when x2t is invoked from any other parent, or when an extra process generation is inserted between DesktopEditors and x2t (e.g. via gdbserver). Pinning core's vendored ICU to 73 instead of 74 keeps a single consistent ICU version across the whole process tree. Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
ICU only gained a top-level LICENSE file starting around release 74; release-73-1's LICENSE lives at icu4c/LICENSE (which exists in both old and new releases), so the version bump to 73 broke the vendoring script's license copy step. Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
This reverts commit b8feda1.
…pendency" This reverts commit 36a82a1.
nc-build.py cloned depot_tools then ran 'git pull origin main', tracking HEAD. gclient_paths.patch (drops four @functools.lru_cache decorators from gclient_paths.py) was authored against an older depot_tools; upstream f065bb3b0 (2026-07-13, 'Add gclient getconfig subcommand') reworked gclient_paths.py, so the patch stopped applying and core failed to build on all arches (blocking #143 and Euro-Office#116). Pin depot_tools to 6e5a13d2598ee48c9c7afc750401533f30dde16e (newest revision the patch applies against; verified with git apply --check) and set DEPOT_TOOLS_UPDATE=0 so it can't self-update back to HEAD during gclient sync. Verified: core builds and x2t links on linux/arm64. Assisted-by: ClaudeCode:claude-opus-4-8 Signed-off-by: Christoph Schaefer <christoph.schaefer@nextcloud.com>
Description
This PR introduces native support for importing and reading database and data-warehouse files directly into the spreadsheet editor. It adds unified data-extraction pipelines for:
.mdb).sqlite,.db).duckdb).parquet)Motivation
For a sovereign and independent office suite, true data sovereignty means ensuring users are never locked out of their historical data. Legacy Microsoft Access (
.mdb) databases hold decades of critical public, enterprise, and personal records. Relying on proprietary or Windows-exclusive software to open them poses a severe risk to long-term data accessibility. By natively supporting.mdbfile reading, Euro-Office guarantees that users can independently recover and migrate their historical data at any time.Furthermore, once the unified database-to-spreadsheet conversion pipeline was built to support
.mdbrecovery, it was a natural and low-overhead extension to include support for modern embedded databases like SQLite, as well as analytics-heavy formats like DuckDB and Parquet. This empowers data analysts and researchers to quickly inspect massive datasets directly from the comfort of their office suite, bridging the gap between legacy data recovery and modern data science.Technical Details
libmdb(from mdbtools) directly into the source tree to ensure consistent cross-platform builds without relying on fragmented system packages (resolvesiconvandlocalestruct mismatches).libduckdbfor reading DuckDB files and efficiently querying.parquetfiles via DuckDB's view functionality.