Computation failed on loading nodes using nx-arango

[bm777]

Current Behavior

in python when trying to load a large nodes, it throw an error

Nx python code

import nx_arangodb as nxadb
from arango import ArangoClient

db = ArangoClient(hosts=host, request_timeout=300).db(username="root", password="****", verify=True)
G_adb = nxadb.DiGraph(name="graph_name", db=db)

nodes = G_adb.nodes()

Expected Behavior

It should load nodes easily like if my collection was containing only 5000 nodes.

Steps to Reproduce

Create a collection with more than 1 million nodes with some metadata.

My metadata are : title, authors, submitter, versions, created, categories and abstract.

This is one sample

{'_key': '0704.0001',
  '_id': 'lex_node/0704.0001',
  'abstract': '  A fully differential calculation in perturbative quantum chromodynamics is\npresented for the production of massive photon pairs at hadron colliders. All\nnext-to-leading order perturbative contributions from quark-antiquark,\ngluon-(anti)quark, and gluon-gluon subprocesses are included, as well as\nall-orders resummation of initial-state gluon radiation valid at\nnext-to-next-to-leading logarithmic accuracy. The region of phase space is\nspecified in which the calculation is most reliable. Good agreement is\ndemonstrated with data from the Fermilab Tevatron, and predictions are made for\nmore detailed tests with CDF and DO data. Predictions are shown for\ndistributions of diphoton pairs produced at the energy of the Large Hadron\nCollider (LHC). Distributions of the diphoton pairs from the decay of a Higgs\nboson are contrasted with those produced from QCD processes at the LHC, showing\nthat enhanced sensitivity to the signal can be obtained with judicious\nselection of events.\n',
  'authors': "C. Bal\\'azs, E. L. Berger, P. M. Nadolsky, C.-P. Yuan",
  'categories': 'hep-ph',
  'created': 'Mon, 2 Apr 2007 19:18:42 GMT',
  'submitter': 'Pavel Nadolsky',
  'title': 'Calculation of prompt diphoton production cross sections at Tevatron and\n  LHC energies',
  'versions': [{'created': 'Mon, 2 Apr 2007 19:18:42 GMT', 'version': 'v1'},
   {'created': 'Tue, 24 Jul 2007 20:10:27 GMT', 'version': 'v2'}]},

And it throws this error

it throws this error:

thread 'tokio-runtime-worker' panicked at [/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26](https://vscode-remote+ssh-002dremote-002b34-002e136-002e179-002e83.vscode-resource.vscode-cdn.net/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26):
Could not send to channel!: SendError { .. }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at [/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26](https://vscode-remote+ssh-002dremote-002b34-002e136-002e179-002e83.vscode-resource.vscode-cdn.net/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26):
Could not send to channel!: SendError { .. }
thread 'tokio-runtime-worker' panicked at [/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26](https://vscode-remote+ssh-002dremote-002b34-002e136-002e179-002e83.vscode-resource.vscode-cdn.net/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26):
Could not send to channel!: SendError { .. }
thread 'tokio-runtime-worker' panicked at [/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26](https://vscode-remote+ssh-002dremote-002b34-002e136-002e179-002e83.vscode-resource.vscode-cdn.net/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26):
Could not send to channel!: SendError { .. }
thread 'tokio-runtime-worker' panicked at [/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26](https://vscode-remote+ssh-002dremote-002b34-002e136-002e179-002e83.vscode-resource.vscode-cdn.net/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26):
Could not send to channel!: SendError { .. }
thread 'tokio-runtime-worker' panicked at [/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26](https://vscode-remote+ssh-002dremote-002b34-002e136-002e179-002e83.vscode-resource.vscode-cdn.net/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26):
Could not send to channel!: SendError { .. }
thread '<unnamed>' panicked at [/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:285:21](https://vscode-remote+ssh-002dremote-002b34-002e136-002e179-002e83.vscode-resource.vscode-cdn.net/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:285:21):
called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(529), ...)
NetworkX code error: Computation failed

And even G_adb.degree() throws this error

thread 'tokio-runtime-worker' panicked at [/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26](https://vscode-remote+ssh-002dremote-002b34-002e136-002e179-002e83.vscode-resource.vscode-cdn.net/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26):
Could not send to channel!: SendError { .. }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at [/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26](https://vscode-remote+ssh-002dremote-002b34-002e136-002e179-002e83.vscode-resource.vscode-cdn.net/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26):
Could not send to channel!: SendError { .. }
thread 'tokio-runtime-worker' panicked at [/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26](https://vscode-remote+ssh-002dremote-002b34-002e136-002e179-002e83.vscode-resource.vscode-cdn.net/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:279:26):
Could not send to channel!: SendError { .. }
thread '<unnamed>' panicked at [/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:285:21](https://vscode-remote+ssh-002dremote-002b34-002e136-002e179-002e83.vscode-resource.vscode-cdn.net/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arangors-graph-exporter-0.0.9/src/sharding.rs:285:21):
called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(20263), ...)

Environment

OS: Ubuntu 22.04
Python version: 3.12
NetworkX version: 3.4
NetworkX-ArangoDB version: 1.3.0
NetworkX-cuGraph version: 25.02.00
ArangoDB version: 3.12.4

[aMahanna]

hey @bm777

If possible can you provide more information on the data?

I've done the following:

from arango import ArangoClient
import nx_arangodb as nxadb

db = ArangoClient("...").db('_system', password="...", verify=True)

G = nxadb.Graph(name='lexnet', db=db)

docs = [{ 
  'abstract': '  A fully differential calculation in perturbative quantum chromodynamics is\npresented for the production of massive photon pairs at hadron colliders. All\nnext-to-leading order perturbative contributions from quark-antiquark,\ngluon-(anti)quark, and gluon-gluon subprocesses are included, as well as\nall-orders resummation of initial-state gluon radiation valid at\nnext-to-next-to-leading logarithmic accuracy. The region of phase space is\nspecified in which the calculation is most reliable. Good agreement is\ndemonstrated with data from the Fermilab Tevatron, and predictions are made for\nmore detailed tests with CDF and DO data. Predictions are shown for\ndistributions of diphoton pairs produced at the energy of the Large Hadron\nCollider (LHC). Distributions of the diphoton pairs from the decay of a Higgs\nboson are contrasted with those produced from QCD processes at the LHC, showing\nthat enhanced sensitivity to the signal can be obtained with judicious\nselection of events.\n',
  'authors': "C. Bal\\'azs, E. L. Berger, P. M. Nadolsky, C.-P. Yuan",
  'categories': 'hep-ph',
  'created': 'Mon, 2 Apr 2007 19:18:42 GMT',
  'submitter': 'Pavel Nadolsky',
  'title': 'Calculation of prompt diphoton production cross sections at Tevatron and\n  LHC energies',
  'versions': [{'created': 'Mon, 2 Apr 2007 19:18:42 GMT', 'version': 'v1'},
   {'created': 'Tue, 24 Jul 2007 20:10:27 GMT', 'version': 'v2'}]
} for i in range(0, 10000)]

async_db = db.begin_async_execution()

for i in range(0, 100):
    async_db.collection('lexnet_node').import_bulk(docs)

i.e loaded 1 million docs of your sample format into lexnet_node collection.

Then ran this:

from arango import ArangoClient
import nx_arangodb as nxadb

db = ArangoClient("...").db('_system', password="...", verify=True)

G = nxadb.Graph(name='lexnet', db=db)

nodes = G.nodes()

print(len(nodes)) # prints 1 million

Unfortunately not yet hitting the error.

If the dataset is open source, could you maybe provide instructions on how to get it?

And if possible, additional information about your ArangoDB deployment. Was it created via the free-trial deployment sign-up?

[bm777]

Hi @aMahanna ,

I created the data myself for the ArangoDB hackathon, but I couldn't submit on time for some reasons.
I processed over 2.7TB of pdf to generate the citation network (directed graph). 2.7TB of raw pdf, then extracted text from each pdf and store them temporary in Google Cloud Storage gs reducing to 70GB of txt file, and then extract arxiv ID in each paper.

And I plan to make it open source, once it is purely ready. I'm trying now, to add steaming and include callback inside langchain_community.graphs.arangodb_graph.ArangoGraph.

But i can give you access if you want, so that you can connect to the graph.
You give me your email adress:)

It is a self hosted in my instance with a T4 GPU.

---

Workaround

The workaround was to stream nodes/edges and pre-compute the equivalentundirected_graph, and the G.degree which worked perfectly.

If you need anything else let me know.
I was realy sad i could not submit on time.

---

Just to give you a taste of what i'm still refining ->

[aMahanna]

@bm777 understood, feel free to send me the information: anthony.mahanna[at]arangodb.com

[bm777]

@aMahanna sent!