Issue when repeatedly expanding and compacting jsonld with nested/type-scoped contexts #119

Panaetius · 2020-04-17T06:58:13Z

Trying out PyLD 2.0.1 (Also happens with PyLD 2.0.0) on Python 3.6-3.8 and everything works great, performance improves significantly, but tests in our test suite started failing randomly.

After a lot of trial and error, it seems like when expanding and compacting, at random, contexts aren't applied correctly.

For instance

{
"@context": { 
    "schema": "http://schema.org/", 
    "name": "schema:name"
},
"name": "test"
}

ends up being

{
"@context": { 
    "schema": "http://schema.org/", 
    "name": "schema:name"
},
"schema:name": "test"
}

which breaks our processing down the line.

I made a minimal example that showcases the issue.
pyld_reproduce_error.zip

The example loads the attached jsonld file, expands it and compacts it again with the original context. This is repeated until the data after expansion&compaction does not match the output of the last iteration of the loop.

The problem crops up anywhere between iteration 1 and 32 (average 6.5) when I ran it 100 times, it seems completely random when it crops up (Which in our test suite caused tests to fail at random).

Any idea what could be causing this?

The text was updated successfully, but these errors were encountered:

Panaetius · 2020-04-17T07:18:22Z

The problem seems to only happen with more complex contexts, I can run it with the following document 10'000 times without the problem cropping up:

{
    "@context": {
        "@version": 1.1,
        "_id": "@id",
        "_label": "rdfs:label",
        "_message": "rdfs:comment",
        "creator": {
            "@context": {
                "@version": 1.1,
                "_id": "@id",
                "affiliation": "schema:affiliation",
                "alternate_name": "schema:alternateName",
                "email": "schema:email",
                "label": "rdfs:label",
                "name": "schema:name",
                "prov": "http://www.w3.org/ns/prov#",
                "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
                "schema": "http://schema.org/"
            },
            "@id": "schema:creator"
        },
        "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
        "schema": "http://schema.org/",
        "wfprov": "http://purl.org/wf4ever/wfprov#"
    },
    "creator": {
        "@type": [
            "prov:Person",
            "schema:Person"
        ],
        "_id": "mailto:test@example.com",
        "affiliation": null,
        "alternate_name": null,
        "email": "test@example.com",
        "label": "Creator Name",
        "name": "Creator Name"
    }
}

Panaetius · 2020-04-17T07:33:27Z

This als works even though I saw the problem pop up in this `_project` object in the full example:


{
    "@context": {
        "@version": 1.1,
        "_id": "@id",
        "_label": "rdfs:label",
        "_message": "rdfs:comment",
        "_project": {
            "@context": {
                "@version": 1.1,
                "_id": "@id",
                "created": "schema:dateCreated",
                "creator": {
                    "@context": {
                        "@version": 1.1,
                        "_id": "@id",
                        "affiliation": "schema:affiliation",
                        "alternate_name": "schema:alternateName",
                        "email": "schema:email",
                        "label": "rdfs:label",
                        "name": "schema:name",
                        "prov": "http://www.w3.org/ns/prov#",
                        "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
                        "schema": "http://schema.org/"
                    },
                    "@id": "schema:creator"
                },
                "name": "schema:name",
                "prov": "http://www.w3.org/ns/prov#",
                "schema": "http://schema.org/",
                "updated": "schema:dateUpdated",
                "version": "schema:schemaVersion"
            },
            "@id": "schema:isPartOf"
        },
        "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
        "schema": "http://schema.org/",
        "wfprov": "http://purl.org/wf4ever/wfprov#"
    },
    "@type": [
        "prov:Activity",
        "wfprov:ProcessRun"
    ],
    "_id": "https://localhost/activities/commit/3c2fb22cc8119e45baa97dde8cca92b2958da7e5",
    "_label": "/tmp/tmp3ae6nixd/.renku/workflow/6061b7fc3297453c8d048ef01ddabea0_touch.yaml@3c2fb22cc8119e45baa97dde8cca92b2958da7e5",
    "_message": "renku run: committing 1 newly added files",
    "_project": {
        "@type": [
            "prov:Location",
            "schema:Project"
        ],
        "_id": "https://localhost/projects/creator.lastname/tmp3ae6nixd",
        "created": "2020-04-17T05:55:53.595428+00:00",
        "creator": {
            "@type": [
                "prov:Person",
                "schema:Person"
            ],
            "_id": "mailto:test@example.com",
            "affiliation": null,
            "alternate_name": null,
            "email": "test@example.com",
            "label": "Creator Name",
            "name": "Creator Name"
        },
        "name": "tmp3ae6nixd",
        "updated": "2020-04-17T05:55:53.606050+00:00",
        "version": "5"
    }
}

This also works even though it has nested/type scoped contexts:


{

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_message": "rdfs:comment",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"generated": {

"@context": {

"@version": 1.1,

"_id": "@id",

"entity": {

"@context": [

{

"@version": 1.1,

"renku.core.models.entities.Entity_prov_Entity": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "http://www.w3.org/ns/prov#Entity"

}

},

{

"@version": 1.1,

"renku.core.models.entities.Entity_wfprov_Artifact": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "http://purl.org/wf4ever/wfprov#Artifact"

}

},

{

"@version": 1.1,

"renku.core.models.entities.Collection_prov_Collection": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"members": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "prov:hadMember"

},

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "http://www.w3.org/ns/prov#Collection"

}

},

{

"@version": 1.1,

"renku.core.models.entities.Collection_prov_Entity": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"members": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "prov:hadMember"

},

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "http://www.w3.org/ns/prov#Entity"

}

},

{

"@version": 1.1,

"renku.core.models.entities.Collection_wfprov_Artifact": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"members": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "prov:hadMember"

},

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "http://purl.org/wf4ever/wfprov#Artifact"

}

},

{

"@version": 1.1,

"renku.core.models.datasets.Dataset_prov_Entity": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"date_published": "schema:datePublished",

"description": "schema:description",

"files": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"added": "schema:dateCreated",

"based_on": "schema:isBasedOn",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/",

"url": "schema:url",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "schema:hasPart"

},

"identifier": "schema:identifier",

"in_language": {

"@context": {

"@version": 1.1,

"alternate_name": "schema:alternateName",

"name": "schema:name",

"schema": "http://schema.org/"

},

"@id": "schema:inLanguage"

},

"keywords": "schema:keywords",

"license": "schema:license",

"name": "schema:name",

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"same_as": {

"@context": {

"@version": 1.1,

"_id": "@id",

"schema": "http://schema.org/",

"url": "schema:url"

},

"@id": "schema:sameAs"

},

"schema": "http://schema.org/",

"short_name": "schema:alternateName",

"tags": {

"@context": {

"@version": 1.1,

"_id": "@id",

"commit": "schema:location",

"created": "schema:startDate",

"dataset": "schema:about",

"description": "schema:description",

"name": "schema:name",

"schema": "http://schema.org/"

},

"@id": "schema:subjectOf"

},

"url": "schema:url",

"version": "schema:version",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "http://www.w3.org/ns/prov#Entity"

}

},

{

"@version": 1.1,

"renku.core.models.datasets.Dataset_schema_Dataset": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"date_published": "schema:datePublished",

"description": "schema:description",

"files": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"added": "schema:dateCreated",

"based_on": "schema:isBasedOn",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/",

"url": "schema:url",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "schema:hasPart"

},

"identifier": "schema:identifier",

"in_language": {

"@context": {

"@version": 1.1,

"alternate_name": "schema:alternateName",

"name": "schema:name",

"schema": "http://schema.org/"

},

"@id": "schema:inLanguage"

},

"keywords": "schema:keywords",

"license": "schema:license",

"name": "schema:name",

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"same_as": {

"@context": {

"@version": 1.1,

"_id": "@id",

"schema": "http://schema.org/",

"url": "schema:url"

},

"@id": "schema:sameAs"

},

"schema": "http://schema.org/",

"short_name": "schema:alternateName",

"tags": {

"@context": {

"@version": 1.1,

"_id": "@id",

"commit": "schema:location",

"created": "schema:startDate",

"dataset": "schema:about",

"description": "schema:description",

"name": "schema:name",

"schema": "http://schema.org/"

},

"@id": "schema:subjectOf"

},

"url": "schema:url",

"version": "schema:version",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "http://schema.org/Dataset"

}

},

{

"@version": 1.1,

"renku.core.models.datasets.Dataset_wfprov_Artifact": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"date_published": "schema:datePublished",

"description": "schema:description",

"files": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"added": "schema:dateCreated",

"based_on": "schema:isBasedOn",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/",

"url": "schema:url",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "schema:hasPart"

},

"identifier": "schema:identifier",

"in_language": {

"@context": {

"@version": 1.1,

"alternate_name": "schema:alternateName",

"name": "schema:name",

"schema": "http://schema.org/"

},

"@id": "schema:inLanguage"

},

"keywords": "schema:keywords",

"license": "schema:license",

"name": "schema:name",

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"same_as": {

"@context": {

"@version": 1.1,

"_id": "@id",

"schema": "http://schema.org/",

"url": "schema:url"

},

"@id": "schema:sameAs"

},

"schema": "http://schema.org/",

"short_name": "schema:alternateName",

"tags": {

"@context": {

"@version": 1.1,

"_id": "@id",

"commit": "schema:location",

"created": "schema:startDate",

"dataset": "schema:about",

"description": "schema:description",

"name": "schema:name",

"schema": "http://schema.org/"

},

"@id": "schema:subjectOf"

},

"url": "schema:url",

"version": "schema:version",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "http://purl.org/wf4ever/wfprov#Artifact"

}

},

{

"@version": 1.1,

"renku.core.models.datasets.DatasetFile_prov_Entity": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"added": "schema:dateCreated",

"based_on": "schema:isBasedOn",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/",

"url": "schema:url",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "http://www.w3.org/ns/prov#Entity"

}

},

{

"@version": 1.1,

"renku.core.models.datasets.DatasetFile_schema_DigitalDocument": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"added": "schema:dateCreated",

"based_on": "schema:isBasedOn",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/",

"url": "schema:url",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "http://schema.org/DigitalDocument"

}

},

{

"@version": 1.1,

"renku.core.models.datasets.DatasetFile_wfprov_Artifact": {

"@context": {

"@version": 1.1,

"_id": "@id",

"_label": "rdfs:label",

"_project": {

"@context": {

"@version": 1.1,

"_id": "@id",

"created": "schema:dateCreated",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"schema": "http://schema.org/",

"updated": "schema:dateUpdated",

"version": "schema:schemaVersion"

},

"@id": "schema:isPartOf"

},

"added": "schema:dateCreated",

"based_on": "schema:isBasedOn",

"creator": {

"@context": {

"@version": 1.1,

"_id": "@id",

"affiliation": "schema:affiliation",

"alternate_name": "schema:alternateName",

"email": "schema:email",

"label": "rdfs:label",

"name": "schema:name",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/"

},

"@id": "schema:creator"

},

"name": "schema:name",

"path": "prov:atLocation",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/",

"url": "schema:url",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@id": "http://purl.org/wf4ever/wfprov#Artifact"

}

}

],

"@reverse": "prov:qualifiedGeneration"

},

"prov": "http://www.w3.org/ns/prov#",

"role": "prov:hadRole"

},

"@reverse": "prov:activity"

},

"oa": "http://www.w3.org/ns/oa#",

"prov": "http://www.w3.org/ns/prov#",

"rdfs": "http://www.w3.org/2000/01/rdf-schema#",

"schema": "http://schema.org/",

"wfprov": "http://purl.org/wf4ever/wfprov#"

},

"@type": [

"prov:Activity",

"wfprov:ProcessRun"

],

"_id": "https://localhost/activities/commit/3c2fb22cc8119e45baa97dde8cca92b2958da7e5",

"_label": "/tmp/tmp3ae6nixd/.renku/workflow/6061b7fc3297453c8d048ef01ddabea0_touch.yaml@3c2fb22cc8119e45baa97dde8cca92b2958da7e5",

"_message": "renku run: committing 1 newly added files",

"_project": {

"@type": [

"prov:Location",

"schema:Project"

],

"_id": "https://localhost/projects/creator.lastname/tmp3ae6nixd",

"created": "2020-04-17T05:55:53.595428+00:00",

"creator": {

"@type": [

"prov:Person",

"schema:Person"

],

"_id": "mailto:test@example.com",

"affiliation": null,

"alternate_name": null,

"email": "test@example.com",

"label": "Creator Name",

"name": "Creator Name"

},

"name": "tmp3ae6nixd",

"updated": "2020-04-17T05:55:53.606050+00:00",

"version": "5"

},

"generated": [

{

"@type": "prov:Generation",

"_id": "https://localhost/activities/commit/3c2fb22cc8119e45baa97dde8cca92b2958da7e5/tree/baz",

"entity": {

"@type": [

"renku.core.models.entities.Entity_prov_Entity",

"renku.core.models.entities.Entity_wfprov_Artifact"

],

"_id": "https://localhost/blob/3c2fb22cc8119e45baa97dde8cca92b2958da7e5/baz",

"_label": "baz@3c2fb22cc8119e45baa97dde8cca92b2958da7e5",

"_project": {

"@type": [

"prov:Location",

"schema:Project"

],

"_id": "https://localhost/projects/creator.lastname/tmp3ae6nixd",

"created": "2020-04-17T05:55:53.595428+00:00",

"creator": {

"@type": [

"prov:Person",

"schema:Person"

],

"_id": "mailto:test@example.com",

"affiliation": null,

"alternate_name": null,

"email": "test@example.com",

"label": "Creator Name",

"name": "Creator Name"

},

"name": "tmp3ae6nixd",

"updated": "2020-04-17T05:55:53.606050+00:00",

"version": "5"

},

"path": "baz"

},

"role": null

}

]

}

(expand to see the full thing, don't want to clutter the conversation)

gkellogg · 2020-04-17T17:32:43Z

Thanks, @Panaetius, I'll take a look. It's really useful to have something repeatable.

The update uses internal dicts and an LRU cache for retaining referenced contexts, and the result of processing them relative to an active context. Keys are all created via UUIDs or the actual body of the context, so it's hard to see where there could be a collision, but there may be some internal memory issues that are getting triggered.

As the ContextResolver can be specified at runtime, we can probably come up with a version that doesn't actually cache easily enough, or parameterize the creation of the ContextResolver to not cache. This would be a big performance loss, but would likely be a short-term solution to the problem you're seeing.

…the object id of the active context. Fixes #119.

Panaetius · 2020-04-20T06:41:01Z

Thank you very much! 🚀

…the object id of the active context. Fixes #119.

Panaetius mentioned this issue Apr 17, 2020

Update to PyLD 2.0 SwissDataScienceCenter/renku-python#1195

Closed

Panaetius changed the title ~~Caching issue when repeatedly expanding and compacting~~ Issue when repeatedly expanding and compacting jsonld with nested/type-scoped contexts Apr 17, 2020

gkellogg added the bug label Apr 17, 2020

gkellogg self-assigned this Apr 17, 2020

gkellogg added a commit that referenced this issue Apr 17, 2020

Fix inverse context cache indexing to use the uuid field, instead of …

3f6b2e9

…the object id of the active context. Fixes #119.

gkellogg mentioned this issue Apr 17, 2020

Fix inverse context cache indexing to use the uuid field #120

Merged

gkellogg added a commit that referenced this issue Apr 17, 2020

Fix inverse context cache indexing to use the uuid field, instead of …

3e0b2c7

…the object id of the active context. Fixes #119.

davidlehn pushed a commit that referenced this issue Apr 20, 2020

Fix inverse context cache indexing to use the uuid field, instead of …

c198592

…the object id of the active context. Fixes #119.

davidlehn closed this as completed in #120 Apr 20, 2020

davidlehn pushed a commit that referenced this issue Apr 20, 2020

Fix inverse context cache indexing to use the uuid field, instead of …

c3fc929

…the object id of the active context. Fixes #119.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue when repeatedly expanding and compacting jsonld with nested/type-scoped contexts #119

Issue when repeatedly expanding and compacting jsonld with nested/type-scoped contexts #119

Panaetius commented Apr 17, 2020 •

edited

Loading

Panaetius commented Apr 17, 2020

Panaetius commented Apr 17, 2020

gkellogg commented Apr 17, 2020

Panaetius commented Apr 20, 2020

Issue when repeatedly expanding and compacting jsonld with nested/type-scoped contexts #119

Issue when repeatedly expanding and compacting jsonld with nested/type-scoped contexts #119

Comments

Panaetius commented Apr 17, 2020 • edited Loading

Panaetius commented Apr 17, 2020

Panaetius commented Apr 17, 2020

gkellogg commented Apr 17, 2020

Panaetius commented Apr 20, 2020

Panaetius commented Apr 17, 2020 •

edited

Loading