Provide remote support for ArcadeGraph

[dijef]

I am missing important feature in ArcadeDB that allows creating ArcadeGraph for remote connection. Right now this supports only databases opened locally. It is important for us and is supported by some other graph vendors. It was also performing faster and was easier than using only traversal remote. On the top, it allows access through server by other clients and view the data.

[lvca]

You can use the standard Gremlin client for remote operations.

[lvca]

Check the docs under Gremlin API:

var cluster = Cluster.build()
            .port(8182)
            .addContactPoint("localhost")
            .credentials("root", "password")
            .create();
var connection = DriverRemoteConnection.using(cluster);
var g = new GraphTraversalSource(connection);

[dijef]

I am aware of that, but that forces us to re-write entire application logic, while other vendors like Arango or Orient support this.

[lvca]

What features are you using in ArcadeGraph that are not Gremlin (and therefore available through the Gremlin Client)?

[dijef]

Difference is in behaviour and usage.
The code syntax is completely different, much easier to use, as it favours object oriented programming instead of managing complex traversal API every single time. You can pass Vertices and Edges through methods as objects, and push all at end of transaction.

I believe by looking at other vendors, there are optimisation in place, therefore it executes everything much faster. I have performance tests available that prove that.

We are using same code to connect to embedded DB or deployed locally. Deployed locally allow our internal users to access same DB independently of our API. This is also useful for devs as it gives access to studio tools. We can also support different environments. Local DB puts lock and cannot be opened at same time by ArcadeDB.

[lvca]

Thanks for confirming. We do believe using ArcadeGraph is way more powerful than just Gremlin :-)
Going back to your issue, what about using RemoteDatabase? It covers most of EmbeddedDatabase API (under the hood it uses the HTTP protocol). You can run gremlin too as a command. WDYT?

[dijef]

I am also evaluating RemoteDatabase API right now. It is slightly different topic as I am facing some problems with transactions timing out randomly, or results returned limited to 20 000 instead of being 100 000. It also requires manual steps like creating Vertices and Edges up-front. And this proves my point about simplicity, as when I was using ArcadeGraph, (Orient and Arango offers similar approach for remote connection) I didn't have to worry about number of coding issues. Performance of Traversal API and RemoteDatabase is very similar, and for different operations favours one or another, however I think (sorry if I am mistaken) there is high cost for using HTTP instead of binary connection as my tests results show.

[lvca]

What kind of remote operation is slower? Are you executing remote transactions with multiple commands?

[dijef]

So I was comparing single vertex inserts, vertices and edges inserts, reading single vertices and reading vertices with edges plus traversal query. In remote connection (on same machine) I was using OrientGraph instance for TinkerPop object API. With Arcade, I could use only Traversal API as I mentioned in other posts. In all tests Orient was faster, while when using Embedded database mode, it was opposite (ArcadeDB being twice faster e.g. in single inserts).

Simplest case:

var v = graph.addVertex("inputstructure");
v.property("json", json);

gts.clone().addV("inputstructure").property("json", json).id().next()

Results in ms for 10k, 100k, 500k favouring Orient:
1 160 | 5 920
5 404 | 33 006
21 500 | 171 502

[dijef]

To add more information: OrientDB, ArangoDB and JanusGraph all suport Graph object in remote connection

[lvca]

So the second block code is the Gremlin Client code you're using with ArcadeDB only, right? Do you need to execute .clone()? Why?

With 500k, do you mean 500k add vertex operations? Are you using a single TX with OrientDB (it optimizes and batches operations in one single call)?

[dijef]

I am using .clone() as replacement of new GraphTraversalSource(drc); where drc is instance of DriverRemoteConnection.
I am testing both inserting with one single transaction and separate transaction per insert, both show same trend.

I am using Gremlin code for search operations, in tests identical code is used by Orient and ArcadeDB. In all tests Orient is faster on remote connection.

An example:

var inputStructuresV = gts.clone()
	.with("evaluationTimeout", 0)
	.V()
	.hasLabel("inputstructure")
	.toList();
for (var v : inputStructuresV) {
	String json = v.value("json");
	if (json == null || json.isBlank()) {
		throw new RuntimeException("Json not available");
	}
}
return inputStructuresV.size();

And results favouring Orient:
405 | 629
2 304 | 5 567
12 868 | 31 534

[dijef]

[lvca]

I assume the issue is with Gremlin: by using the Gremlin client, that poor guy is fetching all the vertices on the client to filter them, while with OrientDB the command is sent to the server and executed there with much lower latency.

Have you already tried to use ArcadeDB RemoteDatabase class and do something like this?

RemoteDatabase database = new RemoteDatabase("127.0.0.1", 2480, "mydb", "root", "playwithdata");

ResultSet resultSet = database.query("gremlin", "g.V().hasLabel(\"inputstructure\").toList();");
while( resultSet.hasNext() ) {
  Result result = resultSet.next();
  // DO SOMETHING WITH THE RESULT
}

[lvca]

I was able to adapt ArcadeGraph class to use a RemoteDatabase instance. It's still on my local branch, but this is the code:

 @Test
  public void insert() throws Exception {
    testEachServer((serverIndex) -> {
      final RemoteDatabase database = new RemoteDatabase("127.0.0.1", 2480 + serverIndex, DATABASE_NAME, "root",
          BaseGraphServerTest.DEFAULT_PASSWORD_FOR_TESTS);

      try (final ArcadeGraph graph = ArcadeGraph.open(database)) {
        graph.getDatabase().getSchema().createVertexType("inputstructure");

        final long beginTime = System.currentTimeMillis();

        for (int i = 0; i < 500_000; i++) {
          var v = graph.addVertex(org.apache.tinkerpop.gremlin.structure.T.label, "inputstructure", "json", "{\"name\": \"Elon\"}");
        }

        System.out.println("TOTAL " + (System.currentTimeMillis() - beginTime));
      }
    });
  }

[lvca]

This test took 137 secs on my Mac (both client and server on my PC). I changed the 2 lines addVertex() + .property() in 1 to cut latency. While now you can use the nice ArcadeGraph interface, performances with remote can't be comparable with OrientDB, because with OrientDB the requests are cached in a transaction (implicit or not). So if you insert 500K vertices instead of don't 500K calls, you could do 100 call, each with 5K insert each.

This is not supported by ArcadeDB. It's not really TCP/IP vs HTTP, but it's in the way OrientDB was optimized for such use case.

[lvca]

I like the idea to provide the same abstraction we had in OrientDB for ArcadeDB Gremlin instance. I'll create an issue from this discussion.

[lvca]

#1329

[lvca]

Ok, it's all in the main branch (23.11.1-SNAPSHOT). You can now use ArcadeGraph with a remote database.

[dijef]

Great, I am really glad this will be supported, as for me and my team this is simplest API to work with. Thanks for explaining about performance.

[lvca]

This issue has been completed: #1329.