Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide functionality for use case regarding exporting all data for import into a FindMyCourt lookup database (e.g., CSV export) #4

Closed
Tracked by #1
bguayante opened this issue Nov 10, 2022 · 6 comments

Comments

@bguayante
Copy link
Contributor

No description provided.

@bguayante bguayante mentioned this issue Nov 10, 2022
4 tasks
@bguayante bguayante closed this as not planned Won't fix, can't repro, duplicate, stale Nov 10, 2022
@bguayante bguayante reopened this Nov 28, 2022
@bguayante
Copy link
Contributor Author

World Wide @JDziurlaj

This is the issue associated with what we were discussing earlier today. Contrary to the Issue title, we are scrapping the csv export but still need to output a json file from the RDF store. I tagged you in a comment to @jungshadow, but it might be better to create a new .py file for the export rather than adding it to the existing script.

In either case, the import script is located in scripts and is named db_importer.py. Let me know if you have any questions.

@bguayante
Copy link
Contributor Author

Also indev is the most current branch. The scripts in main are mostly the same, but indev is where all the action is.

@bguayante
Copy link
Contributor Author

@JDziurlaj I've put everything together but your db_importer.py script hangs forever when I run it. When I installed the updated Pipfile I got an error that said Ignoring importlib-metadata: markers 'python_version < "3.10"' don't match your environment. I'm pretty sure it's the library related to oxigraph. I'm on Python 3.10 or 3.11 and I remember that you were around 3.6 a few weeks ago. I tried to cheese the Pipfile by changing the < to <= and was able to install everything but the script still hung. I tried both the pre-merge feature/json-ld branch and the merged indev branch. Do you mind comparing what's on your local machine to the db_exporter script that's here on GH?

@JDziurlaj
Copy link
Contributor

I pulled the latest indev from GitHub, and even installed Python 3.10.8, and encountered no issues, other than one with relative paths in db_exporter.py (pull latest). Keep in mind I am running in WSL/Debian.

@bguayante
Copy link
Contributor Author

@jungshadow Do you mind pulling the most recent indev and running the exporter script? You might need to run the pipeline first. @JDziurlaj reports it's working with WSL/Debian but I can't get the same code to run on macOS. Do you mind verifying if it's an OS issues?

@bguayante
Copy link
Contributor Author

We sorted this out. It was an issue with using duplicate data in the db. The export probably would work with duplicates, but it would take a very, very long time. I created additional test data to eliminate duplicates and all works as expected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants