fix(dispatch): upgrade to helix-fetch v2 #398

tripodsan · 2021-02-04T10:10:24Z

No description provided.

codecov · 2021-02-04T10:11:48Z

Codecov Report

Merging #398 (adbd1df) into main (90af496) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main      #398   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            5         5           
  Lines          232       242   +10     
=========================================
+ Hits           232       242   +10

Impacted Files	Coverage Δ
src/fetchers.js	`100.00% <100.00%> (ø)`
src/index.js	`100.00% <100.00%> (ø)`
src/redirects.js	`100.00% <100.00%> (ø)`
src/utils.js	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 90af496...adbd1df. Read the comment docs.

trieloff · 2021-02-04T10:46:47Z

package.json

  },
  "devDependencies": {
    "@adobe/eslint-config-helix": "1.1.3",
-    "@adobe/helix-deploy": "https://github.com/adobe/helix-deploy.git#no-epsagon",


github-actions · 2021-02-04T16:01:52Z

This PR will trigger a patch release when merged.

stefan-guggisberg · 2021-02-04T16:12:36Z

src/utils.js

@@ -51,7 +49,7 @@ function getFetchOptions(options) {
  delete headers.host;
  delete headers.connection;
  return {
-    cache: 'no-cache',


This cache option was never supported by helix-fetch (v1 & v2). Using the correct cache: 'no-store' option causes the following test failure:

1) Index Tests index produces 404 when fetcher fails due to missing action (with 404 handler).: TypeError: Stream had error: The operation was aborted.

The same error was also triggered with helix-fetch v1 but was hidden due the incorrect cache option.

Seems like this code is responsible for the failure:
https://github.com/adobe/helix-dispatch/blob/helix-v2/src/index.js#L227-L232

no. this error was due to another problem: for the 404 page, we receive a 200 status, but need to tweak it to look like a 404. with helix-fetch v1, we could just override the response.status but with v2 this is protected (even worse, the object is not frozen, so no error is thrown, but even if you set res.status=404, it remains 200). so we wrap it with a Proxy to override the status. later in the finally block, it aborts all pending requests, if they are have not produced the response that is returned. since the proxied response is now not the same response again that we check against, it was aborted.

hmm, I debugged the test with helix-fetch v1 and cache: 'no-store'. I got the exact same failure (aborted). So it doesn't seem to be related to v2 vs v1.

hmm... AFAICR, I added the Proxy recently, as I couldn't overwrite the status.... but I think it was for the universal deploy changes. this code is so complicated, there might be several layers of potential errors :-(

so we wrap it with a Proxy to override the status

Why not

const { headers, url } = resp; const resp404 = new Response(res.body, { status: 404, headers, url });

?

I try... but the problem remains, it's not the same response as we captured during the fetch. so we don't know if we need to abort or not.

but with v2 this is protected (even worse, the object is not frozen, so no error is thrown, but even if you set res.status=404, it remains 200)

FWIW: I agree, it is confusing. However it's the same behavior as with the most popular nodejs fetch implementation (node-fetch, 21m weekly downloads) and in the browser.

stefan-guggisberg · 2021-02-04T17:18:32Z

related: adobe/helix-home#182

tripodsan · 2021-02-05T07:17:50Z

now with the recent fixes, it hangs again on openwhisk (and epsagon)....

tripodsan · 2021-02-05T08:20:58Z

now with the recent fixes, it hangs again on openwhisk (and epsagon)....

looking into it....

stefan-guggisberg · 2021-02-05T08:50:03Z

now with the recent fixes, it hangs again on openwhisk (and epsagon)....

hmm, just pulled the latest changes and run npm test:

64 passing (12s)

----------------------|---------|----------|---------|---------|-------------------
File                  | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
----------------------|---------|----------|---------|---------|-------------------
All files             |     100 |      100 |     100 |     100 |                   
 fetchers.js          |     100 |      100 |     100 |     100 |                   
 index.js             |     100 |      100 |     100 |     100 |                   
 redirects.js         |     100 |      100 |     100 |     100 |                   
 resolve-preferred.js |     100 |      100 |     100 |     100 |                   
 utils.js             |     100 |      100 |     100 |     100 |                   
----------------------|---------|----------|---------|---------|-------------------

No hanging tests.

tripodsan · 2021-02-05T09:02:47Z

No hanging tests.

yes. but it hangs on openwhisk and espagon enabled...

eg:

$ curl -sD - 'https://adobeioruntime.net/api/v1/web/helix/helix-services/dispatch@4.4.7-test?static.owner=trieloff&static.repo=helix-demo&static.ref=master&path=/index.md&content.owner=trieloff&content.repo=helix-demo&content.ref=master&a=7'
HTTP/2 202
content-length: 86
content-type: application/json
x-request-id: mbt3f5GRZQYfxvzkGnB3k3saFYRhAeIa
access-control-allow-origin: *
access-control-allow-methods: OPTIONS, GET, DELETE, POST, PUT, HEAD, PATCH
access-control-allow-headers: Authorization, Origin, X-Requested-With, Content-Type, Accept, User-Agent
x-openwhisk-activation-id: 788c55c7f66b45a78c55c7f66b25a763
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
strict-transport-security: max-age=7884000; includeSubDomains
x-envoy-upstream-service-time: 60036
x-azure-ref: 07AYdYAAAAAAUWSsDaswjRJ27zsPVI0tOVFlPMDFFREdFMDQyMABiNzdiNjYzMy04ZjcwLTQyNWItOTJiZC03NmFlYmU5YWI5YzU=
date: Fri, 05 Feb 2021 08:51:51 GMT

{
  "code": "mbt3f5GRZQYfxvzkGnB3k3saFYRhAeIa",
  "error": "Response not yet ready."
}

It's hard to debug - I don't think it's related to helix-fetch, as we had this problems before. but rather with the promise-wars inside helix-dispatch :-) - or a new behaviour of epsagon....

tripodsan · 2021-02-05T10:24:43Z

using a slightly modified epsagon would resolve the hanging actions: epsagon/epsagon-node#428

stefan-guggisberg · 2021-02-08T13:59:39Z

@tripodsan 3 branch-deploy tests are still timing out. Any idea?

tripodsan · 2021-02-09T02:39:04Z

@tripodsan 3 branch-deploy tests are still timing out. Any idea?

as I said... I'm still waiting for the espagon release...

## [4.4.8](v4.4.7...v4.4.8) (2021-02-09) ### Bug Fixes * **dispatch:** upgrade to helix-fetch v2 ([#398](#398)) ([ac929cb](ac929cb))

trieloff · 2021-02-09T08:34:52Z

🎉 This PR is included in version 4.4.8 🎉

The release is available on:

v4.4.8
GitHub release

Your semantic-release bot 📦🚀

fix(dispatch): upgrade to helix-fetch v2

5253e01

tripodsan requested review from trieloff and stefan-guggisberg February 4, 2021 10:10

tripodsan mentioned this pull request Feb 4, 2021

fix(dispatch): upgrade to helix-fetch v2 and avoid haning requests in… #397

Closed

trieloff approved these changes Feb 4, 2021

View reviewed changes

stefan-guggisberg mentioned this pull request Feb 4, 2021

Migrate to helix-fetch v2 adobe/helix-home#182

Closed

31 tasks

fix: incorrect cache option

7e11150

stefan-guggisberg suggested changes Feb 4, 2021

View reviewed changes

tripodsan added 2 commits February 5, 2021 16:04

chore(index): don't abort proxied response

9c761ce

chore(test): fix coverage

2083e46

tripodsan self-assigned this Feb 5, 2021

stefan-guggisberg approved these changes Feb 5, 2021

View reviewed changes

chore: update hleix-deploy

adbd1df

tripodsan merged commit ac929cb into main Feb 9, 2021

tripodsan deleted the helix-v2 branch February 9, 2021 08:32

trieloff pushed a commit that referenced this pull request Feb 9, 2021

chore(release): 4.4.8 [skip ci]

c7b5582

## [4.4.8](v4.4.7...v4.4.8) (2021-02-09) ### Bug Fixes * **dispatch:** upgrade to helix-fetch v2 ([#398](#398)) ([ac929cb](ac929cb))

trieloff added the released label Feb 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(dispatch): upgrade to helix-fetch v2 #398

fix(dispatch): upgrade to helix-fetch v2 #398

tripodsan commented Feb 4, 2021

codecov bot commented Feb 4, 2021 •

edited

trieloff Feb 4, 2021

github-actions bot commented Feb 4, 2021

stefan-guggisberg Feb 4, 2021 •

edited

tripodsan Feb 5, 2021

stefan-guggisberg Feb 5, 2021

tripodsan Feb 5, 2021

stefan-guggisberg Feb 5, 2021

tripodsan Feb 5, 2021 •

edited

stefan-guggisberg Feb 5, 2021 •

edited

stefan-guggisberg commented Feb 4, 2021

tripodsan commented Feb 5, 2021

tripodsan commented Feb 5, 2021

stefan-guggisberg commented Feb 5, 2021

tripodsan commented Feb 5, 2021

tripodsan commented Feb 5, 2021

stefan-guggisberg commented Feb 8, 2021

tripodsan commented Feb 9, 2021

trieloff commented Feb 9, 2021

fix(dispatch): upgrade to helix-fetch v2 #398

fix(dispatch): upgrade to helix-fetch v2 #398

Conversation

tripodsan commented Feb 4, 2021

codecov bot commented Feb 4, 2021 • edited

Codecov Report

trieloff Feb 4, 2021

Choose a reason for hiding this comment

github-actions bot commented Feb 4, 2021

stefan-guggisberg Feb 4, 2021 • edited

Choose a reason for hiding this comment

tripodsan Feb 5, 2021

Choose a reason for hiding this comment

stefan-guggisberg Feb 5, 2021

Choose a reason for hiding this comment

tripodsan Feb 5, 2021

Choose a reason for hiding this comment

stefan-guggisberg Feb 5, 2021

Choose a reason for hiding this comment

tripodsan Feb 5, 2021 • edited

Choose a reason for hiding this comment

stefan-guggisberg Feb 5, 2021 • edited

Choose a reason for hiding this comment

stefan-guggisberg commented Feb 4, 2021

tripodsan commented Feb 5, 2021

tripodsan commented Feb 5, 2021

stefan-guggisberg commented Feb 5, 2021

tripodsan commented Feb 5, 2021

tripodsan commented Feb 5, 2021

stefan-guggisberg commented Feb 8, 2021

tripodsan commented Feb 9, 2021

trieloff commented Feb 9, 2021

codecov bot commented Feb 4, 2021 •

edited

stefan-guggisberg Feb 4, 2021 •

edited

tripodsan Feb 5, 2021 •

edited

stefan-guggisberg Feb 5, 2021 •

edited