-
Notifications
You must be signed in to change notification settings - Fork 495
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
helper procedures to use import-tool headers for node/relationship creation #489
Comments
Status report: I started working on this. So far, I implemented an initial version that generates a Cypher query based on a CSV header. Currently, I develop the code in a separate project at https://github.com/szarnyasg/cypher-csv-header
These are not yet supported.
OK
OK, even for arrays
What do you mean by this?
What should we do, if there these properties do not have a name? I'd use a named such as
This is a TODO as well. |
@szarnyasg thanks a lot for the update
This could be done by a post-processing step where we aggregate the nodes/rels by label/type and set it manually or with a cypher statement with UNWIND. btw. we have to make sure to not exceed memory limits for tx-sizes. So probably makes most sense to run this in batches. I.e. pass batch-size rows to a separate tx (in a separate thread) to create the data. |
@szarnyasg how is it going? |
@jexp sorry I went missing for some time. It's still on my todolist, but I took some vacation and did some research/proposal/paper writing recently. I will get back to it this week - it's not that far to have a working prototype, I hope. |
@jexp I got back to working on this recently and was joined by my Master's student, @hegyibalint. One thing that occurred to us is that the approach is not entirely clear yet. Our original understanding was that we should generate a |
I think the idea was to allow an efficient import similar to neo4j-import just as procedure? I.e. the procedure would take care of full control of creating the data (nodes and rels), constraints etc. |
We have an initial implementation in my fork: https://github.com/szarnyasg/neo4j-apoc-procedures/commit/d3fcdc1c50737028439a15d441ee8ef60ce75c73 We used the One thing that is not entirely clear to us is how the progress reporting should be handled. Our current approach is a quick-and-dirty hack: it uses a As the loading is carried out by an internal Cypher command*, it can be difficult to track progress. Maybe we should use
|
I'll have a look. Just quickly from your comments. Not sure if we had an misunderstanding with "export" ? If you generate the load csv anyway, you could also generate a multi-pass for different labels / rel-type or generate the labels/rel-types in. Yes I know the query statistics bit is quite annoying. You can at least use apoc.create.addLabels, so the node-counts are correct. |
I found out that it's possible to override the package name for a given procedure such as |
One think that occurred to me though is that we could simplify the code a bit by requiring all files to have a header - this is currently optional. If a certain CSV file does not have a header, users can still use As we have to access the file anyways (to get the number of lines), this is quite straighforward to implement. Also, it makes the public API simpler, as the user does not have to provide the header in the |
@jexp I have a conceptual question. In the thread starting post, you mentioned that:
AFAIK, indexes can only be defined for a certain label, so the user has to create the appropriate ID index for all labels. I tried to create them from APOC, but I don't think this is possible due to the "Cannot perform schema updates in a transaction that has performed data updates." limitation. |
My previous question leads to another, more important question: can we start new (and parallel) transactions from a transaction calling an APOC procedure? I am starting new transaction for all batches with |
Some new lessons learnt:
|
Really that shouldn't be the case for :ID at least it didn't used to be. |
Yes you can start new transactions in new threads (but you need to inject GraphDatabaseAPI), see examples here which Util methods help: Those background tx could even be used to create the schema indexes upfront. |
Given the file
And the following import sequence rm -rf data/databases/graph.db/
bin/neo4j-import --into data/databases/graph.db --nodes:Person import/id-test.csv --relationships:KNOWS import/rel-test.csv It works with both
or
|
Ya, it doesn't need the id-names. |
btw. something that might be relevant here too is groups. To handle conflicting id-spaces. i.e.
|
Feel free to send your commit as a PR |
The example is getting bigger (4 CSV files now), so I created a repository for it: https://github.com/szarnyasg/neo4j-csv-ids Here is how the CSVs look:
The problem is that if we respect the ID attributes' name (
I believe this will also work with id spaces. |
Okay, so we are getting close.
I added a check for indexes: https://github.com/szarnyasg/neo4j-apoc-procedures/blob/9babeafa963d6c832084e130d1364aca06a21e7a/src/main/java/apoc/export/csv/ImportCSV.java#L108 (partially inspired by https://stackoverflow.com/questions/23443337/java-check-if-index-already-exists-neo4j-if-clause) And I create them like this: Does this look OK? |
(I originally added this comment to the PR, but it rather belongs here.) @jexp I finally got back to this. I revamped the code a bit and now have one fundamental question left: how should we pass the URI of the CSV file to the procedure? Let me elaborate on that one a bit. The procedure needs to use the filename at least twice: look into it to read the CSV header @ https://github.com/neo4j-contrib/neo4j-apoc-procedures/pull/581/files#diff-df1b88452931fbee58e22a3f1d27b3a8R181 There is a surprising contradiction in this:
So in both cases, the |
Okay, it seems I almost reinvented the wheel here. FileUtils should take care of those issues. |
…ort tool's specification (#581) * Add 'apoc.import.csv' procedure This allows users to load graphs to an online database from CSVs with headers that follow the conventions of the neo4j-import tool * Introduce batch transactions to 'apoc.import.csv'
…ort tool's specification (#581) * Add 'apoc.import.csv' procedure This allows users to load graphs to an online database from CSVs with headers that follow the conventions of the neo4j-import tool * Introduce batch transactions to 'apoc.import.csv'
the neo4j-import tool headers already contain information about label, id-property, property-types, skipping etc.
It would be great to load these csv files with load csv and use the header information internally to set labels/rel-types, convert types for properties or skip columns
apoc.load.csv
should also support thatFrom @szarnyasg:
Is there a simple way to load this from Cypher? the best approach I can think of is to use
```LOAD CSV WITH HEADERS FROM ... AS line FIELDTERMINATOR
|
The text was updated successfully, but these errors were encountered: