GraphImporter fails to load data properly when ArcadeDB server is alive

[warframesam]

When ArcadeDB server(dockerized) is running and GraphImporter(separate Java process) is run to load data, exceptions appear in server logs and data is not shown properly on UI.
Workaround: Closing the server and then running GraphImporter makes it stable

ArcadeDB Docker container logs -> ArcadeDB_Server_alive_GraphImporter_dataload_issue.log

GraphImporter logs -> ArcadeDB_GraphImporter_server_alive.log

ArcadeDB docker image version: 26.10.1-SNAPSHOT

arcadedb-integration jar version: 26.10.1-SNAPSHOT

Docker compose

services:
  arcadedb:
    image: arcadedata/arcadedb:latest
    container_name: arcadedb
    ports:
      - "2480:2480"
      - "2424:2424"
    volumes:
      - C:/Users/warfr/Documents/arcadedb/databases:/home/arcadedb/databases
    environment:
      - arcadedb.server.rootPassword=playwithdata
      - ARCADEDB_OPTS_MEMORY=-Xmx16g -Xms4g

Schema

CREATE VERTEX TYPE WORK;

-- Define Properties on WORK Type
CREATE PROPERTY WORK.id IF NOT EXISTS LONG;
CREATE PROPERTY WORK.title IF NOT EXISTS STRING;
CREATE PROPERTY WORK.abstract IF NOT EXISTS STRING;
CREATE PROPERTY WORK.authors IF NOT EXISTS STRING;
CREATE PROPERTY WORK.publication_year IF NOT EXISTS SHORT;

CREATE PROPERTY WORK.embedding ARRAY_OF_FLOATS;

-- Create Unique Index on id
CREATE INDEX WORK_id ON WORK (id) UNIQUE;

-- Define Edge Type
CREATE EDGE TYPE CITE;

Java code

public static void main(String[] args) {

        ArcadeDBGraphImporterConfig arcadeDBGraphImporterConfig = new ArcadeDBGraphImporterConfig();

        try (DatabaseFactory arcade = new DatabaseFactory(arcadeDBGraphImporterConfig.getDatabasePath())) {
            try (Database database = arcade.open()) {

                try (GraphImporter importer = GraphImporter.builder(database)
                        .vertex("WORK", new JsonlRowSource(arcadeDBGraphImporterConfig.getVertexSourcePath()), v -> {
                            v.id("id");
                            v.longProperty("id", "id");
                            v.property("title", "title");
                            v.property("abstract", "abstract");
                            v.property("authors", "authors");
                            v.intProperty("publication_year", "publication_year");
                            v.floatArrayProperty("embedding", "embedding");
                        })
                        .edgeSource("CITE", new CsvRowSource(arcadeDBGraphImporterConfig.getEdgeSourcePath()), e -> {
                            e.from("from_id", "WORK");
                            e.to("to_id", "WORK");
                        })
                        .build()) {

                    importer.run();
                    System.out.printf("\nVertices: %,d, Edges: %,d%n",
                            importer.getVertexCount(), importer.getEdgeCount());
                } catch (Exception e) {
                    throw new RuntimeException(e);
                }
            }
        }
    }

Steps to reproduce

• Start the ArcadeDB docker container
• Log into server from UI and run the schema script from UI
• Run the GraphImporter to load some data (NOTE: Keep the ArcadeDB container running, you will see WAL exceptions in container logs)
• Go back to UI and run SELECT query on WORK and CITE. (NOTE: You will see no data) [Check screenshots]
• Logout from UI and restart the ArcadeDB docker container
• Log into server from UI and run the SELECT query on WORK and CITE. (NOTE: You will see data, However, the UI tags for WORK and CITE show 0 indicating no data, which is incorrect) [Check screenshots]

Screenshots

• Server running (GraphImporter data load completed) [No data appears on UI]

• Server restarted [Data appears on UI, However, the UI tags for WORK and CITE show 0]

[lvca]

Thanks for the detailed report and logs - this is fixed now.

What was happening: running GraphImporter as a separate process while the ArcadeDB Server already had the same database open could corrupt it. The importer's own clean shutdown was deleting WAL files the still-running server had open, which is why you saw repeated errors in the server log, no data via SELECT while the server was up, and zeroed record counts after restart.

Fix: ArcadeDB now protects each WAL file individually, so a second process sharing the same database directory can no longer delete or interfere with files another instance still has open. Merged in #7504 (tracked in #7502).

Recommendation either way: don't open the same database from two processes at the same time (an embedded tool like GraphImporter and a running server). If a server already has the database open, connect to it remotely (HTTP/Postgres/etc.) instead. This is now called out explicitly in the docs.

Two smaller, unrelated things found along the way were filed separately: #7505 and #7506.

---

❤️ Thanks for helping make ArcadeDB better! ArcadeDB is free and open source, sustained by its community. If it's useful to you or your company, please consider becoming a sponsor to keep development going.

[warframesam]

@lvca Thank you for the fix.
From what I understand, If GraphImporter is used as a separate process, the server should ideally be shut down before loading any data.
Suggestion: If not already updated in the docs, this info should be added to the GraphImporter docs

[lvca]

Good call - that's exactly what went in alongside the fix. The GraphImporter page now opens with a warning: never run the importer against a database directory that a server (or any other process) already has open. To load into a database a server is serving, use the remote protocol; otherwise stop the server for the duration of the import.

https://docs.arcadedb.com/arcadedb/how-to/migration/graph-importer.html

Note that with #7504 merged the WAL files are protected individually, so the corruption path you hit is closed either way - the guidance is about avoiding two writers on the same directory in general, not a workaround for that bug.