Creating new collections for appId to DataSource mapping

[vishalag001]

Hi @wajda,
We have a usecase where we want to track appId to dataSource uri mapping. In current design, dataSource Uri is available via 2 hops progress -> progressOf -> executionPlan -> [affects, depends] -> dataSource
But if we creates a new edge collection which maps (appId, DataSource) we can directly get the relationship and the graph.
For this I created two new collections - jobs document to store appId and owns edge collection to store the mapping between appId and datasource.
I tweaked the progress inserts code to following

const dependsDSKey = db._query(aql`
    WITH executionPlan,dataSource, depends
    FOR v in 1 OUTBOUND ${ep._id} depends return distinct v._key
`).toArray()
const jobsAffectsEdge: Partial<ArangoDB.Edge> =
    edge('jobRuns',progress.extra.appId,CollectionName.DataSource,dataSourceKey)

const jobsDependsEdge = []
dependsDSKey.forEach(d=>{
    const ed = edge(CollectionName.DataSource,d,'jobRuns',progress.extra.appId)
    jobsDependsEdge.push(ed)
})
try {
      store.insertOne(progressWithPlanDetails, CollectionName.Progress, wtxInfo)
      store.insertOne(progressEdge, CollectionName.ProgressOf, wtxInfo)
      store.insertOne({'_key':progress.extra.appId,'_created':progress._created}, CollectionName.JobRuns, wtxInfo)
      store.insertOne(jobsAffectsEdge, CollectionName.Owns, wtxInfo)
      store.insertMany(jobsDependsEdge, CollectionName.Owns, wtxInfo)
      TxManager.commit(wtxInfo)
  }
  catch (e) {
      TxManager.rollback(wtxInfo)
      throw e
  }

This works, but the problem is Arango is getting stuck due to QueueFull error.
Any idea/suggestions why is this happening ?

[vishalag001]

Hi @wajda can you help with the above usecase ?

[wajda]

I'll respond to this, no worries. It just requires some more time to analyse the question and the code, which I cannot afford right now. I'll respond to this later, hope for understanding.

[wajda]

At the first glance, I don't see anything wrong with the code above. The arango performance issue that you experience might be caused by something else. I guess the real cause might be actually the one you outlined in another topic - #1264. If I'm wrong and the issues are independent then more info is needed.