Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Question about EARL reports #155

Closed
hsolbrig opened this issue May 25, 2021 · 8 comments
Closed

Question about EARL reports #155

hsolbrig opened this issue May 25, 2021 · 8 comments

Comments

@hsolbrig
Copy link

When I run python tests/runtests.py ../json-ld-api/tests/ ../json-ld-framing/tests/ -e pyld-earl.jsonld, I get
FAILED (failures=18, errors=168, skipped=7)

This doesn't seem to line up with the summary printed in https://w3c.github.io/json-ld-api/reports/

Digging into the report itself, the first failure shows at:

{
      "@type": "earl:Assertion",
      "earl:assertedBy": "https://github.com/dlongley",
      "earl:mode": "earl:automatic",
      "earl:test": "https://w3c.github.io/json-ld-api/tests/compact-manifest#tc009",
      "earl:result": {
        "@type": "earl:TestResult",
        "dc:date": "2021-05-25T21:31:04Z",
        "earl:outcome": "earl:failed"
      }
    },

Which, if I understand the reports correctly, appeared to have passed previously. Help!
@hsolbrig
Copy link
Author

At least some of the failures seem to fit the pattern below:

EXPECTED:  {
  "@context": {
    "@vocab": "http://example/",
    "Foo": {
      "@context": {
        "baz": {
          "@type": "@vocab"
        }
      }
    }
  },
  "@type": "Foo",
  "bar": {
    "baz": "buzz"
  }
}
ACTUAL:  {
  "@context": {
    "@vocab": "http://example/",
    "Foo": {
      "@context": {
        "baz": {
          "@type": "@vocab"
        }
      }
    }
  },
  "@type": "Foo",
  "bar": {
    "baz": {
      "@id": "http://example/buzz"
    }
  }
}```

@davidlehn
Copy link
Member

I haven't looked into this, but it may in part be a case of the the test suites changing and pyld hasn't caught up. But looks like you got a bunch of failures and errors so maybe it's something else too.

The reports really should be clear that results are for a test suite and implementation as of a certain date.

As far as tracking down and fixing the issues, help would be appreciated.

@hsolbrig
Copy link
Author

Happy to pitch in -- just wanted to be certain I understood the starting point.

@hsolbrig
Copy link
Author

Ok - part of the issue is that URL's of the form https://json-ld.org/test-suite/tests/frame/0005-in.jsonld are no longer valid.

curl https://json-ld.org/test-suite/tests/frame/0005-in.jsonld
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at json-ld.org Port 80</address>
</body></html>

If I do
curl https://json-ld.org/test-suite/
I get

<div class="hero-unit">
    <h1>JSON-LD Test Suite</h1>
    <br>
    <p>Please use the Working Group version of the test suite at <a href="https://w3c.github.io/json-ld-api/tests/">https://w3c.github.io/json-ld-api/tests/</a> and <a href="https://w3c.github.io/json-ld-framing/tests/">https://w3c.github.io/json-ld-framing/tests/</a> instead.</p>
  </div>

So - do we update the manifest and any embedded links? How to proceed?

@gkellogg
Copy link
Collaborator

Tests should now all be at https://w3c.github.io/json-ld-api/tests/ and /json-ld-framing/tests/. Those in json-ld.org are obsolete.

@hsolbrig
Copy link
Author

Will update the manifests accordingly.

@hsolbrig
Copy link
Author

Ah - I see what is going on. The instructions for running the tests are a bit ambiguous....

@hsolbrig
Copy link
Author

Was using https://github.com/digitalbazaar/pyld/blob/master/CONTRIBUTING.rst , which wasn't clear on where the tests came from. Will file a separate issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gkellogg @davidlehn @hsolbrig