Implement automated storing to db/backend

[jpwahle]

Is your feature request related to a problem? Please describe.
We need to store author, venue, and publication data into our backend automatically when the next d3 version is released.

Describe the solution you'd like
Implement a backend class that:

• creates papers and authors
• updates papers and authors
• delets papers and authors

Additional context
https://www.mongodb.com/docs/database-tools/mongoimport/#std-label-ex-mongoimport-merge

[github-actions]

Branch issue-70 created!

[jpwahle]

The final layer missing here is the automatic update of the backend.
One of the main issues is that more than 6 million queries have to be sent to the backend to check whether a paper exists and has to be updated/inserted.

One solution would be to hash all papers and let the backend return a list of all hashes that the crawler can compare to without sending any other request. Then the crawler can decide what to update/write which results in few requests per update.

[trannel]

So far we used the code I created on branch https://github.com/gipplab/cs-insights-crawler/tree/data-upload-full in the file upload/d3_full.py. There might be some helpful things there, that could help with this issue.

[jpwahle]

@muhammadtalha242 Is the new data ingestion through SemanticScholar already ready?