v5.5.0 - Feedback wanted on ExecutableQuery 📣

[RichardIrons-neo4j]

In 5.5.0 we have introduced a new concept into the .net driver: ExecutableQuery. This is a simplified API for running queries against a database without needing to get into the detail of sessions and transactions. The new API is in the Neo4j.Driver.Experimental namespace, and as such is subject to future changes.

An example of very basic use is as follows:

var driver = GraphDatabase.Driver("bolt://localhost:7687", AuthTokens.Basic("neo4j", "password"));

var people = await driver
    .ExecutableQuery("MATCH (p:Person) RETURN p AS person")
    .ExecuteAsync();

foreach(var record in people.Result)
{
    Console.WriteLine(record["person"].As<INode>()["name"].As<string>());
}

This runs the cypher query and return an EagerResult object. This object contains the result summary, the list of keys present in the result, and the result itself (in the Result property). Specifically, this will return an EagerResult<IReadOnlyList<IRecord>>, as in, the Result property will contain a read-only list of IRecord objects.

To use a parameterised query is also easy:

var birthYears = await driver
    .ExecutableQuery("MATCH (p:Person WHERE p.name = $name) RETURN p.born AS born")
    .WithParameters(new Dictionary<string, object> { ["name"] = "Tom Hanks" })
    .ExecuteAsync();

var year = birthYears.Result[0]["born"].As<int>();
Console.WriteLine($"Tom Hanks born {year}");

An alternative syntax for setting the parameters would be:

    .WithParameters(new { name = "Tom Hanks" })

• dictionaries and anonymous objects are both supported as a way of setting parameters, as in the rest of the API.

It is possible to have the EagerResult object contain anything you like by using the .WithStreamProcessor method. This method allows you to specify a lambda function that will be called, receiving an IAsyncEnumerable<IRecord> as its parameter, which represents the stream of records as they are received from the database. The value you return from this method will be in the Result property of the EagerResult object returned from executing the query.

For example, in this code, the lambda function uses methods in the System.Linq.Async package to only take 5 of the records, to convert each one to an anonymously-typed object, and then to turn them into a list of these objects. An EagerResult<List<{string name, int born}>> is returned from this method, and so it is possible to iterate over the list in the Result property and access the objects.

var nameAndBirthYear = await driver
    .ExecutableQuery("MATCH (p:Person) RETURN p AS person")
    .WithStreamProcessor(
        async stream => await stream
            .Take(5)
            .Select(record => new
            {
                name = record["person"].As<INode>()["name"].As<string>(),
                born = record["person"].As<INode>()["born"].As<int>(),
            })
            .ToListAsync())
    .ExecuteAsync();

foreach (var obj in nameAndBirthYear.Result)
{
    Console.WriteLine($"{obj.name} born {obj.born}");
}

It is also possible to use await foreach in this lambda function, for example like this:

    .WithStreamProcessor(
        async stream =>
        {
            var list = new List<string>();
            await foreach (var record in stream)
            {
                list.Add(record["person"].As<INode>()["name"].As<string>());
            }

            return list;
        })

In this instance a list of strings is returned, so the result of execution the query would be of type EagerResult<List<string>>.

It is possible to return a scalar value from this lambda, for example:

var birthYearAverageCalc = await driver
    .ExecutableQuery("MATCH (p:Person) RETURN p AS person")
    .WithStreamProcessor(
        async stream =>
        {
            double sum = 0;
            int count = 0;
            await foreach (var record in stream)
            {
                sum += record["person"].As<INode>()["born"].As<int>();
                count++;
            }

            return sum/count;
        })
    .ExecuteAsync();

double averageBirthYear = birthYearAverageCalc.Result;
Console.WriteLine($"Average birth year: {averageBirthYear}");

In this case an EagerResult<double> is returned. Using this system it is possible to process all the records using the simplified API, without having to load them all into memory simultaneously.

To configure the query execution further, you can use the .WithConfig method to pass a QueryConfig object. For example to run the query while impersonating another user, you could do this:

var birthYears = await driver
    .ExecutableQuery("MATCH (p:Person WHERE p.name = $name) RETURN p.born AS born")
    .WithConfig(new (impersonatedUser: "someone-else"))
    .WithParameters(new { name = "Tom Hanks" })
    .ExecuteAsync();

As you can see, this cuts down on the amount of boilerplate and complex coding needed to query the database.

A lot happens under the hood:

• the driver will create a session (and dispose of it when done)
• run and retry a transaction function as often as it needs
• get the result and transform it via the provided transformer.

Moreover, it tracks bookmarks through a default BookmarkManager implementation so that subsequent ExecuteQuery reads can read the results of former ExecuteQuery writes.

Feedback

We're definitely looking for feedback on this feature. Any thoughts you have are gratefully received. Specific questions we would like to ask:

• Is the Keys property needed in the EagerResult?
• Would a version of ExecuteAsync that just returns the result of the query, rather than wrapping it in an EagerResult, be valuable?
• • naming - everything here is experimental and as such the naming is not set in stone. If you have better ideas for some of the identifier names, please do let us know.

[WhatzGames]

The concept of the streamprocessor seems nice, but doing something more like the following would be preferred IMHO.

var birthYears = await driver
    .ExecutableQuery("MATCH (p:Person WHERE p.name = $name) RETURN p.born AS born")
    .WithConfig(new (impersonatedUser: "someone-else"))
    .WithParameters(new { name = "Tom Hanks" })
    .ExecuteAsAsyncEnumerable(); //maybe just AsAsyncEnumerable()

await foreach (var birthYear in birthYears)
{
    // some processing
   yield return processingResult;
}

Using this approach, the results still gets eagerly loaded, but without needing to allocate a delegate.
System.Linq.Async can then also be joined with results from other queries and therefore provide more flexibility in the upper methods.

Regarding the approach with parameters, this is way more just a personal preference than anything else, but I personally do prefer a string using the composite-syntax more concerning autocompletions, since it is supported in many IDEs already.
Since there is an attribute specifically for syntax highlighting in post .NET 6 version, it might be something to consider

[thelonelyvulpes]

Thanks for taking the time to provide feedback.

In regards to returning the IAsyncEnumerable, there is a few technicalities which made us rule out a similar design.
I'll point out some constraints which guided our design:

• The proposed API is built on top of our internal retry mechanism that is there to support clusters. If you have worked with a cluster you'll likely found that the retry mechanism stops you needing to worry about transient errors that can occur such as a leader election.
• In order to ensure correctness when retrying transactions, we execute queries in an explicit transaction this ensures no data is partially committed. On the other hand IAsyncSession.RunAsync uses implicit transactions which can be useful for periodically committing data to the database but at the same time comes with a risk to integrity.
• In order to access results from a transaction in neo4j the transaction must still be open on the server.
• Like our other APIs that follow the previous constraints (IAsyncSession.WriteTransactionAsync, IAsyncSession.ExecuteWriteAsync) we need a way for users to rollback a transaction.

Given the above constraints we landed on 2 options:

• Eagerly read all the results to a collection, commit the transaction then return results, Which is the default behaviour for the proposed API.
  • The eager result will return an array of records, so you will be able to run code with those results in a for loop.
• Process results in the scope of the transaction like IAsyncSession.WriteTransactionAsync, IAsyncSession.ExecuteWriteAsync this require users to pass us code to execute.
  • Note: throwing an exception in side the delegate of WithStreamProcessor will rollback the transaction.

💡 How I imagined IAsyncEnumerable might work
What we could do is return an object that implements IAsyncEnumerable and IAsyncDisposable, when the enumeration is complete or the object is disposed we commit the transaction.
That said we ruled it exposes users to an excessive amount of complexity with regards to dealing with retries and cancelling transactions.

I hope this helps, if you need any further clarifications then let me know.

[Luigi-Pan]

Trying to figure out how access neo4j data formatted as JSON (.Net driver). The following cypher returns JSON data. Need to capture and parse this JSON-

MATCH p=(user:User {uid:"uniqueid2" })<-[r*]-(v)
with user, collect(p) as data
CALL apoc.convert.toTree(data, true) YIELD value
return value as user

Trying to access data with-
IResultCursor cursor = await _session.RunAsync(neo4jQuery);

None of my next steps work-
string name = cursor.As();
List people = await cursor.ToListAsync(record => record["name"].As());

Any help would be appreciated.