GraphImporter fails to load LIGHTWEIGHT edges

[warframesam]

Data(~4.2 million vertices and ~125million edges) was loaded via GraphImporter.
GraphImporter logs(attached below) indicate that the data was loaded successfully, However, no data appears on UI(see screenshot attached below) for CITE edge(NOTE: edge is declared LIGHTWEIGHT in schema).

Possible cause: This has something to do with GraphBatch's lightEdges property(though it is deprecated) or the way GraphImporter handles edges with no properties.

Suggestion: GraphImporter should expose GraphBatch properties or GraphImporter builder()/constructor should take an optional GraphBatch object as parameter to allow users to have more control while loading data via GraphImporter.

Schema

CREATE VERTEX TYPE WORK;

-- Define Properties on WORK Type
CREATE PROPERTY WORK.id IF NOT EXISTS LONG;
CREATE PROPERTY WORK.title IF NOT EXISTS STRING;
CREATE PROPERTY WORK.abstract IF NOT EXISTS STRING;
CREATE PROPERTY WORK.authors IF NOT EXISTS STRING;
CREATE PROPERTY WORK.publication_year IF NOT EXISTS SHORT;

CREATE PROPERTY WORK.embedding ARRAY_OF_FLOATS;

-- Create Unique Index on id
CREATE INDEX WORK_id ON WORK (id) UNIQUE;

-- Define Edge Type
CREATE EDGE TYPE CITE LIGHTWEIGHT;

Java code

public static void main(String[] args) {

        ArcadeDBGraphImporterConfig arcadeDBGraphImporterConfig = new ArcadeDBGraphImporterConfig();

        try (DatabaseFactory arcade = new DatabaseFactory(arcadeDBGraphImporterConfig.getDatabasePath())) {
            try (Database database = arcade.open()) {

                try (GraphImporter importer = GraphImporter.builder(database)
                        .vertex("WORK", new JsonlRowSource(arcadeDBGraphImporterConfig.getVertexSourcePath()), v -> {
                            v.id("id");
                            v.longProperty("id", "id");
                            v.property("title", "title");
                            v.property("abstract", "abstract");
                            v.property("authors", "authors");
                            v.intProperty("publication_year", "publication_year");
                            v.floatArrayProperty("embedding", "embedding");
                        })
                        .edgeSource("CITE", new CsvRowSource(arcadeDBGraphImporterConfig.getEdgeSourcePath(), '\t',0), e -> {
                            e.from("from_id", "WORK");
                            e.to("to_id", "WORK");
                        })
                        .build()) {

                    importer.run();
                    System.out.printf("\nVertices: %,d, Edges: %,d%n",
                            importer.getVertexCount(), importer.getEdgeCount());
                } catch (Exception e) {
                    throw new RuntimeException(e);
                }
            }
        }
    }

GraphImporter logs -> ArcadeDB_GraphImporter_LIGHTWEIGHT_EDGE.log

[lvca]

Thanks for the very complete report - the log and the screenshot together made this quick to pin down.

Good news first: your import worked. All 75,539,070 CITE edges are in the database and traversable right now. You can confirm it without reloading anything:

SELECT out("CITE").size() FROM WORK LIMIT 10
SELECT expand(outE("CITE")) FROM WORK LIMIT 10

What went wrong is on the read side, not in GraphImporter. A LIGHTWEIGHT edge is a pair of pointers stored inside the two vertices and allocates no record of its own - that is the whole point of the declaration, and it is why the type is so much cheaper. But the edge type still owns a bucket, and SELECT FROM CITE scanned that bucket, which by construction can never hold anything. So it answered 0, SELECT count(*) FROM CITE answered 0, and Studio printed that 0 next to your type. Nothing anywhere said "this type keeps no records by design", so the only reasonable conclusion from the screen was the one you reached.

Tracked as #7477 and fixed in #7478 (milestone 26.10.1): a scan of a LIGHTWEIGHT edge type now walks the vertices that hold the edges instead of the empty bucket, count(*) agrees with it, DELETE FROM <type> reaches them too, and Studio labels the type LIGHTWEIGHT rather than showing a record count it can never have.

On your suggestion - exposing GraphBatch's properties through GraphImporter, or having the builder accept a GraphBatch - I would rather not, and I think you will agree once the cause is clear. withLightEdges is deprecated exactly because the storage shape belongs to the schema: GraphBatch already lets a type's LIGHTWEIGHT declaration win over that flag, which is why your load was correct. A knob the importer exposed could only be used to contradict your own schema. The gap was never in what the loader could be told; it was that nothing downstream could see what it had loaded.

One thing in your log worth a separate look on your side:

CITE 49,904,703 rows name an identity no WORK vertex carries: those edges were skipped

That is 40% of your edge file. If your citation edges legitimately point at works outside the 4.2M you loaded, that is expected and you can ignore it. If not, check that from_id/to_id in the TSV are spelled the same way as id in the JSONL (a W-prefix on one side and a bare number on the other is the usual cause).

[warframesam]

Thank you for the quick response and fix.
From your explanation, it is clear that this was an issue on the read side, not the data load side.

CITE 49,904,703 rows name an identity no WORK vertex carries: those edges were skipped

This ⬆️ warning can be safely ignored for now, since the cites file was generated from a dataset containing complete set of refrences and will contain edges pointing to data outside the vertices data scope. No data filteration has yet been done from my end since I am in testing phase.
I plan to load around ~300 million vertices and more than ~1 billion edges eventually, so I am doing testing on required ArcadeDB functionality

[lvca]

Fixed in 26.10.1 (PR #7478, issue #7477).

To confirm what I said earlier: your import was fine. All 75,539,070 CITE edges were in the database the whole time. The bug was that querying them by type name looked in the place where edge records live, and a LIGHTWEIGHT edge has no record there - so it answered zero.

SELECT FROM CITE, SELECT count(*) FROM CITE and DELETE FROM CITE now all work on a LIGHTWEIGHT type, and Studio labels it instead of showing 0.

Two notes: count(*) on such a type is no longer instant (there is no stored count that can be right, so it reads the graph), and TRAVERSE / TRUNCATE TYPE now return a clear error rather than quietly doing nothing - #7480 and #7481.

On your suggestion of exposing GraphBatch's settings through GraphImporter: not needed, and worth saying why. withLightEdges is deprecated because the storage shape belongs to the schema, and GraphBatch already lets your type's LIGHTWEIGHT declaration win over it - which is why your load was correct. The gap was never in what the loader could be told.

Still worth a look on your side: the log said 49,904,703 of your rows named an identity no WORK vertex carries. If those citations point at works outside your dataset, that is expected; otherwise check that from_id/to_id in the TSV are spelled the same way as id in the JSONL.

---

❤️ Thanks for helping make ArcadeDB better! ArcadeDB is free and open source, sustained by its community. If it's useful to you or your company, please consider becoming a sponsor to keep development going.

[warframesam]

@lvca Thank you for the fix.
I will test this and share the results

[warframesam]

@lvca I tested this on a small dataset (50k vertices and ~150k edges)
Data load was successful. On the UI side, the select query also works.
However, the UI tag for edge CITE still shows 0. The back-end API is returning a 0 record count for edge CITE

[lvca]

Thanks for testing this so thoroughly. I checked the network response in your screenshot, and the backend is actually already correct: "lightweight": true and "records": 0 are both there.

records: 0 for a LIGHTWEIGHT edge type is intentional (it's what #7477/#7478 deliberately chose not to make exact, to avoid paying an O(V+E) walk on every Studio page load). The sidebar badge is supposed to key off lightweight and show the word "lightweight" instead of that 0 - and it does, in the current studio-database.js shipped with #7478.

So the badge showing plain "0" in your screenshot is almost certainly your browser serving a cached copy of the pre-fix studio-database.js (Studio's static JS isn't cache-busted across a version upgrade, which is its own separate, known gap). Could you do a hard refresh (Ctrl+Shift+R / Cmd+Shift+R, or clear the site's cache) and check again?

Separately, while double-checking this I did find one real, smaller gap: the sidebar section header total (e.g. "Edges (N)") sums every type's record count, so a section containing a LIGHTWEIGHT type undercounts - "Edges (0)" even with 150,000 real edges in your CITE type. Fixed that: the total now shows a trailing "+" (e.g. "Edges (0+)") whenever the section holds a type it can't fully count, so it reads as a lower bound instead of a wrong exact number. That'll ship in the next release. Tracked as #7566.

[warframesam]

@lvca Thank you for the fix
After clearing the browser's cache(hard reload), I can see the lightweight indication on the CITE edge tag [see screenshot]
Question: Since you mentioned the tradeoff O(V+E), when using a lightweight edge, I would like to understand how does one benefit from using a lightweight edge as opposed to a normal edge with no properties?

[lvca]

Check the docs: https://docs.arcadedb.com/arcadedb/reference/lightweight-edges