test(wpt): add results to an existing WPT Report

[panva]

I wish to extend the daily Node.js WPT Report with your WPTs.

I plan to extend the workflow file with roughly the following

UNDICI_VERSION = cat deps/undici/src/package.json | jq -r '.version'
if (UNDICI_VERSION_INCLUDES_THIS_CHANGE(UNDICI_VERSION))
  actions/checkout nodejs/undici at $UNDICI_VERSION tag
  WPT_REPORT=$(pwd)/out/wpt/wptreport.json npm run test:wpt

In order for the WPTs to be comparable with other wpt.fyi results they must respect the variants (1st commit), must sanitize the subtest names and messages the same (part of the 2nd commit).

NB: per commit no whitespace review recommended

[panva]

cc @KhafraDev please give it a rough first look

[KhafraDev]

the changes look fine so far, although the runner is erroring out

[panva]

Fixed that, but what are these outside the wpt runner about?

[KhafraDev]

jest, unfortunately (#1945)

[panva]

jest get rid of it already ;)

[KhafraDev]

I'd love to, but we need to make sure undici works with jest 😵‍💫

[codecov-commenter]

Codecov Report

Base: 90.43% // Head: 90.42% // Decreases project coverage by -0.02% ⚠️

Coverage data is based on head (f3de626) compared to base (d8d9a96).
Patch has no changes to coverable lines.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1944      +/-   ##
==========================================
- Coverage   90.43%   90.42%   -0.02%     
==========================================
  Files          71       71              
  Lines        6119     6119              
==========================================
- Hits         5534     5533       -1     
- Misses        585      586       +1     

Impacted Files	Coverage Δ
lib/fetch/file.js	89.65% <0.00%> (-1.15%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[panva]

@KhafraDev Do you have a handy command in the undici repo to update the WPT test files?

[panva]

Ok, so with updated WPT files the test:wpt doesn't even run. That is an issue seeing how the daily job runs with the latest wpt files...

[panva]

So the issue is that as soon as I pull in the full WPTs there are tests executed that crash your runner.

I dealt with that in xhr and mime-sniff but fetch and FileAPI are too cherry-picked in your checkout.

I don't know how to proceed from here. The runner walks the whole directory to discover tests but I think it might need a filter list in the case of fetch and FileAPI which would allow new WPTs to be pulled in but it will automatically exclude the ones from new directories to run.

[panva]

I've pushed the one commit that can go alone as #1951.

As for the WPT Report, this is also good to go but I cannot use it unless the Runner is ready for the fixtures to be replaced by a full web-platform-tests/wpt checkout.

Issues i've encountered while doing so:

• new {{ }} GET replace block to be handled by handlePipes
• rather than logging wpt setup failures the runner stops the run with no logs

Feel free to close this as needed, I don't unfortunately have the time needed to fix the runner issues and for fetch related WPTs to be included in the daily wpt.fyi upload they need to represent a given wpt checkout, not the cherry picked one the runner expects at the moment.

[KhafraDev]

I've dmed you on twitter, I'd like to take care of those issues (including a way to automatically update the WPTs). What are you doing to update/replce them?

[anonrig]

Is there a specific reason for why we are not running fetch wpt in node.js core?

[KhafraDev]

@anonrig I outlined some difficulties with doing so here: nodejs/node#46498 (comment)

[panva]

I'm not sure how the wpt service will deal with duplicate FileAPI/blob and FileAPI/file results, we run them in Node.js too...

[KhafraDev]

I can remove disable the blob tests, but undici implements its own File class (predecessor to node's), so we should keep those in. Maybe the easiest fix is to disable reports being generated for File/Blob tests?

[panva]

Yeah, I'll pass the WPT_REPORT optionally straight from the start file and skip doing so for FileAPI.

[KhafraDev]

We do also implement FileReader (at the time it made sense), but I guess reporting coverage from it isn't that important.

[panva]

We do also implement FileReader (at the time it made sense), but I guess reporting coverage from it isn't that important.

Not unless it's made a global in Node. To be honest a lot of the FileWPTs in Node are failing just because we don't FileReader to support them...

[panva]

Also skipped WebSocket since it's not a Node.js global (yet?).

[panva]

I will do a test submit to staging.wpt.fyi before you land this so that you can see the results.

[panva]

Please have a look / review

• report https://staging.wpt.fyi/api/runs/5247378118410240
• fetch https://staging.wpt.fyi/results/fetch?run_id=5247378118410240&run_id=6086989598162944
• xhr/formdata https://staging.wpt.fyi/results/xhr/formdata?run_id=5247378118410240&run_id=6086989598162944
• mimesniff/mime-types https://staging.wpt.fyi/results/mimesniff/mime-types?run_id=5247378118410240&run_id=6086989598162944
• complete merged report https://staging.wpt.fyi/results/?run_id=5247378118410240&run_id=6086989598162944

[panva]

I found some untitled subtests, looking into it.

[KhafraDev]

Just wondering, since our tests (& node's) run in a worker, should the *.worker.html tests be marked as passing too?

[panva]

Just wondering, since our tests (& node's) run in a worker, should the *.worker.html tests be marked as passing too?

It's not the same and just duplicating the results based on the path seems wrong. You can use the rich wpt query language to filter only tests that are present in one of the products to compare.

[panva]

I found some untitled subtests, looking into it.

Worked around in the runner.

[ronag]

RSLGTM, @KhafraDev I will have to trust you on this one.