Fix 124 identifiers sha #135

valentinedwv · 2023-01-11T19:25:19Z

Add ability to set an identifier per source:

      identifierpath: []
      identifiertype: identifiersha
      # identifiertype: ""   # default normalizedjsonsha flight test if we get a different if
      # identifiertype: identifierstring  # needs sanitizing...

identifiertype

(normalizedjsonsha) present default is the sha of the normlized json
- (Generate an identifier based on the whole json, if normalize produces ""
(identifiersha) Generate identifiers based on jsonpath.

identifierpath is a jsonpath that gets prefied to the default jsonpath.

Metadata about identifier is written to s3 user metadata.

includes the context repairs since that was the most likely normalize issue.

updated cobra and viper package versions

…_sha # Conflicts: # go.mod # go.sum

…o/gleaner into fix_129_context

Tests to be sure arrays are not mangled by string and that strings are not mangled by array.

This reverts commit 613b709.

Tests to be sure arrays are not mangled by string and that strings are not mangled by array.

return an identifier sha if someone asks for an unsupported string add user metadatadetails to s3

…the original later Clean up LdProc Normalize Tests. Empty triples will be empty triples

add some comments to the test

fix workflow

and glcon tools jsonld: jsonld > corrected jsonld rdf: jsonld > rdf

Added ability to just generate a sha of the json. Was option, but not coded.

* Cleaned up tests. Files now used. Added ability to just generate a sha of the json. Was option, but not coded.

…x_124_identifiers_sha # Conflicts: # go.mod # go.sum # internal/common/identifier_test.go # internal/config/sources.go # pkg/cli/gleaner.go # pkg/cli/root.go

… results.

…one. What we need is the contexts, for the triples if it falls through to a normalizedsha

…one. We need is the contexts, for the triples if it falls through to a normalizedsha makefile. copy binaries up to that assets are in correct location relative to binaries

But for integration testing file is not run from this directory, so the generation may fail, and gives a warning which muck with an approval test.

nein09 · 2023-03-06T22:33:00Z

This seems to be working for me! Ship it.

Thanks!

valentinedwv and others added 30 commits July 25, 2022 10:49

run configurations

f075d91

adding a uuid command to get uuid for a file sent to stdin.

e9e1d09

updated cobra and viper package versions

create a tools cmd

4b46a93

clean up some logging CLI.

20f1fe1

Choose a JsonPath parser

f22a891

Choose a JsonPath parser

44534ed

Example JSON Paths and arrays of paths

4cb9cbe

Example JSON Paths and arrays of paths

671aad3

Update add an identifierType. aka how to generate the indentifier.

2abaa72

Merge remote-tracking branch 'origin/dv_urn' into fix_124_identifiers…

82428a3

…_sha # Conflicts: # go.mod # go.sum

glcon tool id --idtype indentifiersha --jsonld some file

53acdac

test for #129.

58267aa

Implemented changes to prevent context mangling

fc3013d

Implemented changes to prevent context mangling

fd514a7

Merge branch 'fix_124_identifiers_sha' of https://github.com/gleaneri…

6f2252e

…o/gleaner into fix_129_context

If it's an array, we standardized.

5f9b9e6

Tests to be sure arrays are not mangled by string and that strings are not mangled by array.

If it's an array, we standardized.

a8df3fb

Tests to be sure arrays are not mangled by string and that strings are not mangled by array.

WIP (including stray files)

bc15b12

Get things working and DRY code up a bit

b40605d

Extract out a constant and a json processing function

d6e5383

Add test for json finding method

68dc5e2

Test for the api config helper

2e27954

Try to make this work for DataStream as well

613b709

Identifiers fix for @graph

9e7b3cf

Revert "Try to make this work for DataStream as well"

4bbf6ea

This reverts commit 613b709.

Remove stray files

5970a0e

Remove another stray file

9f782aa

fix for empty normalized triples

ad8dbbb

add some testing to ldproc for normalization

2ddc8c4

FIxing tests

860fc96

valentinedwv added 27 commits March 2, 2023 15:06

If it's an array, we standardized.

cc2dd1c

Tests to be sure arrays are not mangled by string and that strings are not mangled by array.

Identifiers fix for @graph

6a6b196

fix for empty normalized triples

d1617c8

FIxing tests

4f016f2

rename filesha to jsonsha, since that is what it is.

e795192

return an identifier sha if someone asks for an unsupported string add user metadatadetails to s3

implement identifierstring... add config.SourceUrl type

298fdc9

cleanup a couple of items to use the string constants

66466e6

add an idea example resource

1492012

IdentiferPath is a comma separated set of strings, or one string.

e093606

If you are trying not to recorder an array... and copy it, don't use …

8e8701d

…the original later Clean up LdProc Normalize Tests. Empty triples will be empty triples

Move expectations struct to a tests_common_struct file

b273d31

add some comments to the test

Update nabu dependencies during merge with dev

0d670a8

fix workflow

added test for a couple of problem children.

19fddf7

methods to allow dumping data before upload

8e6e3fb

and glcon tools jsonld: jsonld > corrected jsonld rdf: jsonld > rdf

methods to allow dumping data before upload

c7a9b53

and glcon tools jsonld: jsonld > corrected jsonld rdf: jsonld > rdf

add a couple resources for cli testing

1c07820

Cleaned up tests. Files now used. (#161)

e3bf70a

Added ability to just generate a sha of the json. Was option, but not coded.

Fix 124 identifiers tests (#162)

294337f

* Cleaned up tests. Files now used. Added ability to just generate a sha of the json. Was option, but not coded.

put in approvals into jsonutils

5e4d270

use log.info not println so this does not show up in console output.

41a02c0

sync with dev

4fe7a8b

add approval test .approved

89e3423

Merge remote-tracking branch 'origin/fix_124_identifiers_sha' into fi…

a8570d2

…x_124_identifiers_sha # Conflicts: # go.mod # go.sum # internal/common/identifier_test.go # internal/config/sources.go # pkg/cli/gleaner.go # pkg/cli/root.go

Sort the jsonpath results because UUID generation depends on order of…

3b7c092

… results.

One more fix for tool id. If no config is passed, just use a generic …

073a189

…one. What we need is the contexts, for the triples if it falls through to a normalizedsha

One more fix for tool id. If no config is passed, just use a generic …

18306a5

…one. We need is the contexts, for the triples if it falls through to a normalizedsha makefile. copy binaries up to that assets are in correct location relative to binaries

fix issue with tool id where is works with a configs/local.

87a0414

But for integration testing file is not run from this directory, so the generation may fail, and gives a warning which muck with an approval test.

valentinedwv merged commit 5049528 into dev Mar 6, 2023

valentinedwv deleted the fix_124_identifiers_sha branch September 8, 2023 22:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix 124 identifiers sha #135

Fix 124 identifiers sha #135

valentinedwv commented Jan 11, 2023

nein09 commented Mar 6, 2023

Fix 124 identifiers sha #135

Fix 124 identifiers sha #135

Conversation

valentinedwv commented Jan 11, 2023

nein09 commented Mar 6, 2023