querying arangodb view inside transaction

[konnerthg]

I have an Arango collection called Documents and an arangosearch view called Documents_view. I am trying to query the view inside a transactional database:

collection = "Documents"
document = {
    "name": "foo",
}

txn_db = arango.db.begin_transaction(read=[collection], write=[collection])
txn_db.insert_document(
    collection=collection,
    document=document,
)

cursor = txn_db.aql.execute(
    query="""
        FOR o IN Documents_view
        RETURN {
           name: o.name,
        }
    """
)

print("documents in view:")
print(list(cursor))

cursor = txn_db.aql.execute(
    query="""
        FOR o IN Documents
        RETURN {
           name: o.name,
        }
    """
)

print("documents in collection:")
print(list(cursor))

txn_db.abort_transaction()

The output reads:

documents in view:
[]
documents in collection:
[{'name': 'foo'}]

It seems like the view does not function inside the transaction? I tried adding "Documents_view" to the list of the transaction's read collections as described here. But this results in an error: arango.exceptions.TransactionInitError: [HTTP 404][ERR 1203] collection or view not found

When printing the list of views: print(txn_db.views()), the Documents_view shows. So I am sure it exists. Even more oddly, when printing the full view: print(txn_db.view(name="Documents_view")), it correctly shows the inserted document: [{'name': 'foo'}]

Am I missing something? Is this behavior intended?

[apetenchea]

Hi @konnerthg,

As far as I know, what you're trying to achieve is unfortunately not possible. The reason is that for data retrieval operations, arangosearch views are by design "eventually-consistent". This means that eventually all the data in ArangoDB is matched by corresponding query expressions. There are two things to take into account:

1. Only documents that are committed are visible in the view. Once a “commit” operation is complete, all documents added/removed prior to the start of the “commit” operation are reflected by queries invoked in subsequent ArangoDB transactions. In other words, while you can definitely utilize the view inside your transaction, is is not going to reflect any changes made inside the transaction.
2. Even when changes are committed, the view is not updated instantly. This process happens in the background and takes time - not much, but enough such that if you query the view immediately after committing a transaction, the results might not fully correspond with the collection. The upper bound of this interval is configurable, see the commitIntervalMsec property. Please note that modifying this property might have additional performance implications.

Here is a moderately complex example illustrating the above (note how I messed around with the commitIntervalMsec property):

import time
from arango import ArangoClient

client = ArangoClient(hosts="http://localhost:8529")
sys_db = client.db("_system", username="root", password="passwd")

sys_db.delete_database("test", ignore_missing=True)
sys_db.create_database("test")
db = client.db("test", username="root", password="passwd")
col = db.create_collection("Documents")
documents = [{"name": "foo"}, {"name": "bar"}, {"name": "baz"}]

# synchronize the view every 1ms
db.create_view("Documents_view", "arangosearch",
               properties={"commitIntervalMsec": 1, "links": {"Documents": {"fields": {"name": {}}}}})

col.insert({"name": "initial_doc"})
# wait for the view to sync
time.sleep(0.1)

txn_db = db.begin_transaction(read=col.name, write=col.name)
for doc in documents:
    txn_db.insert_document(collection=col.name, document=doc)

# wait for the view to sync
# documents that are not committed will not be in the view
time.sleep(0.1)
cursor = txn_db.aql.execute(
    query="""
        FOR o IN Documents_view
        RETURN {
           name: o.name,
        }
    """
)

print("documents in view:")
print(list(cursor))

cursor = txn_db.aql.execute(
    query="""
        FOR o IN Documents
        RETURN {
           name: o.name,
        }
    """
)
print("documents in collection:")
print(list(cursor))

txn_db.commit_transaction()

# must wait for the view to sync
time.sleep(0.1)

txn_db = db.begin_transaction(read=col.name, write=col.name)
cursor = db.aql.execute(
    query="""
        FOR o IN Documents_view
        RETURN {
           name: o.name,
        }
    """
)
print("documents in view:")
print(list(cursor))
txn_db.abort_transaction()

The output is:

documents in view:
[{'name': 'initial_doc'}]
documents in collection:
[{'name': 'initial_doc'}, {'name': 'foo'}, {'name': 'bar'}, {'name': 'baz'}]
documents in view:
[{'name': 'initial_doc'}, {'name': 'foo'}, {'name': 'bar'}, {'name': 'baz'}]

By design, arangosearch views were more intended to simplify and optimize long-running queries, sometimes involving complex text searches. Due to this fact, some design decisions had to be made in order to optimize specifically for fast searching. With distributed systems it's often the case that one has to choose between optimizing for fast lookups vs optimizing for "immediate" data availability. There's lots of things running under the hood (replication, disk syncing, etc). Sometimes optimizing in one place introduces limitation in another (frustrating, I know) 😬

Please let me know if this sheds some light onto the subject.

[konnerthg]

So the behavior was indeed intended. Thank you for the clarification