Allow URLInputSource to get content-negotiation links from the Link headers #1436

ashleysommer · 2021-10-11T15:12:43Z

Allow URLInputSource to get content-negotiation links from the Link headers of HTTP responses.
Use those Links to resolve schema.org-style json-ld conneg redirections.
Fix the ability to run the remote-url arm of the JSON-LD test suite (got most of them working!)
Fixes Failed to parse graph (json-ld, JSONDecodeError) #1423

…eaders of HTTP responses. Use Links to resolve schema.org-style json-ld conneg redirections. Fix the ability to run the `remote-url` arm of the JSON-LD test suite (got most of them working!)

rdflib/parser.py

removed unused import os

cclauss

A few OPTIONAL ideas...

rdflib/parser.py

test/jsonld/runner.py

test/jsonld/test_onedotone.py

aucampia · 2021-10-19T20:39:05Z

@ashleysommer @nicholascar may be worth merging this, I'm hitting at least one of the problems this fixes in the move to pytest in #1452 - see https://drone.rdflib.ashs.dev/RDFLib/rdflib/520/1/2

2717	self = <rdflib.plugins.shared.jsonld.context.Context object at 0x7f9651f87690>
226s
2718	source = {'@import': 'so08-context.jsonld', '@protected': True, '@version': 1.1, 'term': 'http://example.org/redefined'}
226s
2719	source_url = None, referenced_contexts = set()
226s
2720	
226s
2721	    def _read_source(self, source, source_url=None, referenced_contexts=None):
226s
2722	        imports = source.get(IMPORT)
226s
2723	        if imports:
226s
2724	            if not isinstance(imports, str):
226s
2725	                raise INVALID_CONTEXT_ENTRY
226s
2726	    
226s
2727	            imported = self._fetch_context(
226s
2728	                imports, self.base, referenced_contexts or set()
226s
2729	            )
226s
2730	            if not isinstance(imported, dict):
226s
2731	>               raise INVALID_CONTEXT_ENTRY
226s
2732	E               NameError: name 'INVALID_CONTEXT_ENTRY' is not defined

aucampia · 2021-10-19T20:39:59Z

not sure why drone is failing here though, also there may be other problems causing failures on my branch - but this PR looks good to me at least.

Update 1

Actually the issue I had did not really have anything to do with what is fixed here, and tests there pass fine now. Still may be good to merge this before we merge the migration to pytest.

Co-authored-by: Christian Clauss <cclauss@me.com>

nicholascar · 2021-11-21T11:39:21Z

Still may be good to merge this before we merge the migration to pytest.

Ooops... As noted in another PR, we are going to have to update PRs with old-style tests in them to pytest before the next release.

aucampia · 2021-11-24T22:31:41Z

@nicholascar will try make a PR to fix this the coming weekend.

Also ping me if there are more PRs that need updating.

aucampia · 2021-11-24T23:57:18Z

Just to clarify regarding this:

Still may be good to merge this before we merge the migration to pytest.

Ooops... As noted in another PR, we are going to have to update PRs with old-style tests in them to pytest before the next release.

This will be a problem because it does yield tests which are not picked up by pytest but is a nosetest thing, and this is what I will fix. python unittest.UnitTest based tests integrate fine with pytest.

nicholascar · 2021-11-25T00:12:08Z

@aucampia great, thanks. There are only a couple of current, pending PRs with tests that may need consideration since the pytest switch - PRs here, https://github.com/rdflib/rdflib/pulls, since about September.

For most pending PRs, I'm waiting for reviews from other maintainers or feedback from proponents.

rdflib/parser.py

@ashleysommer

test/jsonld/test_onedotone.py got a bit messed up with a merge from master. Looking at the original changes from @ashleysommer, all he did was change a condition. This applies the same change but essentially rebased on master. For comparison see: ab31c5e...c4b679f Also add back import os in rdflib/parser.py This is now needed after #1441 was merged.

aucampia · 2021-11-28T12:40:47Z

@nicholascar #1480 fixes tests

…onneg Adapt for pytest and add back import of os in rdflib/parser.py

aucampia · 2021-12-19T19:39:36Z

@ashleysommer @nicholascar mind if I rebase this onto latest master keep it mergeable?

nicholascar · 2021-12-20T01:18:57Z

mind if I rebase...

Please do!

This is being used and should be imported.

aucampia · 2021-12-28T02:18:51Z

Merged master back in and I went through the code again, I don't see any problems with it, but there are parts not covered by tests. I'm fine merging it though, we need to expand our testing of json-ld quite a bit anyway. But we can also look into that first before merging it.

rdflib/plugins/shared/jsonld/util.py

aucampia · 2022-01-21T12:24:25Z

I think we should clarify what the conditions is for this to be integrated into master, like I said I would like to see more testing but I think this is okay without that, but if others want to see more tests I don't mind working to add them. I want to at some point work on the test suite compliance but that will still be in some time.

white-gecko

Looks good to me. Also the example mentioned in #1423 failed on master and works on this branch for me.

Note re `_urlopen(full_link)` -> `_urlopen(Request(full_link))` `_urlopen` expects a request object and will potentially use `Request.full_url` which won't be available on a string.

aucampia · 2022-03-16T21:15:31Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

aucampia · 2022-03-16T21:37:33Z

Planning to merge this weekend if not merged yet and if there is no objections.

Allow URLInputSource to get content-negotiation links from the Link h…

0b5e5d4

…eaders of HTTP responses. Use Links to resolve schema.org-style json-ld conneg redirections. Fix the ability to run the `remote-url` arm of the JSON-LD test suite (got most of them working!)

ashleysommer mentioned this pull request Oct 11, 2021

Failed to parse graph (json-ld, JSONDecodeError) #1423

Closed

niklasl reviewed Oct 11, 2021

View reviewed changes

rdflib/parser.py Outdated Show resolved Hide resolved

ashleysommer commented Oct 11, 2021

View reviewed changes

rdflib/parser.py Show resolved Hide resolved

ashleysommer mentioned this pull request Oct 13, 2021

Undefined name: INVALID_CONTEXT_ENTRY #1440

Closed

moved http.client imports into a TYPE_CHECKING guard

c4b679f

removed unused import os

cclauss reviewed Oct 14, 2021

View reviewed changes

aucampia approved these changes Oct 19, 2021

View reviewed changes

aucampia mentioned this pull request Oct 28, 2021

Fix JSON-LD data import adds trailing slashes to IRIs (#1443) #1456

Merged

nicholascar and others added 6 commits November 21, 2021 21:24

Merge branch 'master' into jsonld_conneg

619b7c3

Update rdflib/parser.py

19b5cf4

Co-authored-by: Christian Clauss <cclauss@me.com>

Update rdflib/parser.py

831b4cf

Co-authored-by: Christian Clauss <cclauss@me.com>

Update test/jsonld/runner.py

c996e75

Co-authored-by: Christian Clauss <cclauss@me.com>

Update test/jsonld/test_onedotone.py

c6a0de6

Co-authored-by: Christian Clauss <cclauss@me.com>

Update test/jsonld/test_onedotone.py

39fe561

Co-authored-by: Christian Clauss <cclauss@me.com>

nicholascar self-requested a review November 21, 2021 11:39

cclauss reviewed Nov 25, 2021

View reviewed changes

rdflib/parser.py Show resolved Hide resolved

nicholascar added 2 commits December 1, 2021 12:49

Merge pull request #1480 from iafork/iwana-20211128T1326-fix_jsonld_c…

f3ba37c

…onneg Adapt for pytest and add back import of os in rdflib/parser.py

Merge branch 'master' into jsonld_conneg

cc6641e

Merge remote-tracking branch 'origin/master' into jsonld_conneg

862ec67

Import urljoin

36afe6d

This is being used and should be imported.

aucampia requested a review from white-gecko December 28, 2021 00:29

nicholascar added 2 commits December 29, 2021 08:38

Merge branch 'master' into jsonld_conneg

3b1174a

Merge branch 'master' into jsonld_conneg

df1fffc

mwatts15 reviewed Jan 3, 2022

View reviewed changes

rdflib/plugins/shared/jsonld/util.py Outdated Show resolved Hide resolved

nicholascar added 2 commits January 4, 2022 14:08

deduplicate import

d952004

add type to URLInputSource.links

061a465

Add doc string

682f33d

white-gecko approved these changes Mar 16, 2022

View reviewed changes

white-gecko and others added 3 commits March 16, 2022 21:45

Merge branch 'master' into jsonld_conneg

da0a6b6

Merge remote-tracking branch 'upstream/master' into jsonld_conneg

5fd8080

Fix typing errors

f328d7e

Note re `_urlopen(full_link)` -> `_urlopen(Request(full_link))` `_urlopen` expects a request object and will potentially use `Request.full_url` which won't be available on a string.

[pre-commit.ci] auto fixes from pre-commit.com hooks

e00529f

for more information, see https://pre-commit.ci

white-gecko merged commit cc9ac04 into master Mar 17, 2022

white-gecko deleted the jsonld_conneg branch March 17, 2022 09:17

ghost mentioned this pull request Mar 31, 2022

JSON-LD schema.org parsing fails with JSONDecodeError("Expecting value", s, err.value) from None #1781

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow URLInputSource to get content-negotiation links from the Link headers #1436

Allow URLInputSource to get content-negotiation links from the Link headers #1436

ashleysommer commented Oct 11, 2021

cclauss left a comment

aucampia commented Oct 19, 2021

aucampia commented Oct 19, 2021 •

edited

Loading

nicholascar commented Nov 21, 2021

aucampia commented Nov 24, 2021 •

edited

Loading

aucampia commented Nov 24, 2021

nicholascar commented Nov 25, 2021

aucampia commented Nov 28, 2021

aucampia commented Dec 19, 2021

nicholascar commented Dec 20, 2021

aucampia commented Dec 28, 2021 •

edited

Loading

aucampia commented Jan 21, 2022

white-gecko left a comment

aucampia commented Mar 16, 2022

aucampia commented Mar 16, 2022

Allow URLInputSource to get content-negotiation links from the Link headers #1436

Allow URLInputSource to get content-negotiation links from the Link headers #1436

Conversation

ashleysommer commented Oct 11, 2021

cclauss left a comment

Choose a reason for hiding this comment

aucampia commented Oct 19, 2021

aucampia commented Oct 19, 2021 • edited Loading

Update 1

nicholascar commented Nov 21, 2021

aucampia commented Nov 24, 2021 • edited Loading

aucampia commented Nov 24, 2021

nicholascar commented Nov 25, 2021

aucampia commented Nov 28, 2021

aucampia commented Dec 19, 2021

nicholascar commented Dec 20, 2021

aucampia commented Dec 28, 2021 • edited Loading

aucampia commented Jan 21, 2022

white-gecko left a comment

Choose a reason for hiding this comment

aucampia commented Mar 16, 2022

aucampia commented Mar 16, 2022

aucampia commented Oct 19, 2021 •

edited

Loading

aucampia commented Nov 24, 2021 •

edited

Loading

aucampia commented Dec 28, 2021 •

edited

Loading