Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

updates to support quads in the release graph function #32

Merged
merged 3 commits into from
Mar 16, 2023
Merged

updates to support quads in the release graph function #32

merged 3 commits into from
Mar 16, 2023

Conversation

fils
Copy link
Member

@fils fils commented Mar 2, 2023

Changed include

  • ADR: https://github.com/gleanerio/nabu/blob/dev/decisions/0001-URN-decision.md
  • bulkloader (both bulk and release) now checks if single file, skips skolimization if true. Not needed with
    a single file since no chance for blank node URN collision
  • example.yaml updated to include the context parameters and contextmap node
  • comment removed in jsonldToNQ.go (pointless to do, but I did it anyway) ;)
  • mod ldproc.go to read and use the context map node in the config
  • Makefile order changed, for no good reason
  • pipcopy update to support quads as output
  • skolemize.go comment removed to simply pad the number of files in the commit. ;)
  • urnToPrefix more pointless comment updates to make me feel good
  • tofromRDF.go added which does some various RDF transforms, there are some unused functions in here (JsonldToNQ,NqToJSONLD,NqToNTCtx) but left them in for now.

So really only pipecopy, ldproc toFromRDF and the two bulkloader files have any changes of value in them.

@fils fils requested a review from valentinedwv March 2, 2023 17:51
Copy link
Member

@valentinedwv valentinedwv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Get this error... but file get's uploaded.
{"file":"/Users/valentin/development/dev_earthcube/gleanerio/nabu/internal/services/bulk/bulkLoader.go:54","func":"github.com/gleanerio/nabu/internal/services/bulk.BulkAssembly","level":"info","msg":"Put "?graph=urn:gleaner:scratch:iris_bulk": unsupported protocol scheme ""","time":"2023-03-02T12:25:10-08:00"}

@fils
Copy link
Member Author

fils commented Mar 2, 2023

@valentinedwv good catch, I didn't test the services-bulk as close as releases-bulk.

I'll look into it!

@fils
Copy link
Member Author

fils commented Mar 3, 2023

@valentinedwv pushed updates to resolve (and hopefully improve) the bulk command that loads into the triplestores. Docs updated to detail this.

  • bulkLoader.go cleaned up and viper read moved to the docfunc.go
  • example.yaml some clean up and generalizing then add in the endpointMethod and contentType params in sparql node
  • function.go trap errors better and cleanup, read new parameters in the config for different triplestores deprecated ioutils.ReadAll replaced with io.ReadAll
  • README.md doc updates
  • sparql.go update in pkg file to read in the new config parameters
  • allGraphs.rq testing query modified
  • allLimit100.rq testing query modified

Copy link
Member

@valentinedwv valentinedwv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works.
Running a test using the branch in glcon testing, but can't see that it would fail.

@valentinedwv
Copy link
Member

Maybe upload to latest without the datestamp on the filename?

@fils fils merged commit b68225c into dev Mar 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants