🐛 BUG: pages publish very slow / crashing #1194

revmischa · 2022-06-06T20:01:24Z

What version of `Wrangler` are you using?

2.0.8

What operating system are you using?

Mac

Describe the Bug

I'm trying to upload my site, it's about 300MB, it uploads to KV fine with wrangler 1

When I upload to pages with wrangler 2 it goes reaaallly slow and then crashes

▲ [WARNING] Warning: Your working directory is a git repo and has uncommitted changes

  To silence this warning, pass in --commit-dirty=true


🌍  Uploading... (809/8009)

/usr/local/lib/node_modules/wrangler/wrangler-dist/cli.js:16264
            throw ex;
            ^

FatalError: Failed to upload files. Please try again.
    at /usr/local/lib/node_modules/wrangler/wrangler-dist/cli.js:123689:31
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async run2 (/usr/local/lib/node_modules/wrangler/wrangler-dist/cli.js:122127:26) {
  code: 1
}

I posted it on discord https://discord.com/channels/595317990191398933/799437470004412476/983442112534175815

If I try to upload it via web it errors telling me not to upload more than 1000 files. If I zip it it says the zip is too big. I don't see a way to upload a zip file with the CLI.
No idea how to get my files into Pages.

The text was updated successfully, but these errors were encountered:

sidharthachatterjee · 2022-06-07T15:15:10Z

Hey there! Sorry to hear this. What you're describing with the dashboard is expected (we'll be bumping that up soon).

What you're seeing with wrangler however is a bug. Our hypothesis is that your auth token is expiring during the upload because of low upload bandwidth? I've opened a PR with a fix for this.

Can you try this version of wrangler and let us know if it helps?

npx https://prerelease-registry.developers.workers.dev/runs/2455435541/npm-package-wrangler-1199 dev path/to/script.js

Cheers.

ElijahKotyluk · 2022-06-07T19:08:16Z

@sidharthachatterjee I have tested this a couple times, I am having no issues when running the dev command you provided, but still am not seeing any improvements when trying to run the pages publish command. It still hangs and then throws the same error as above.

revmischa · 2022-06-07T20:47:22Z

I ran npx https://prerelease-registry.developers.workers.dev/runs/2455435541/npm-package-wrangler-1199 pages publish public and had the same issue

sidharthachatterjee · 2022-06-08T13:40:12Z

Interesting. Thank you for testing this. In that case, my hypothesis was incorrect. I'll ship that anyway since it's an improvement.

I'll add some verbose debug logs and ship a version for you'll so we can get some more info on what's up. Cheers.

ElijahKotyluk · 2022-06-08T16:18:08Z

Thank you for actively working on this.

sidharthachatterjee · 2022-06-09T14:41:58Z

Hi folks! Can you please try npx https://prerelease-registry.developers.workers.dev/runs/2469110657/npm-package-wrangler-1217 pages publish <dir> and post your logs here?

This will print some debugging information for us (HTTP requests, how long they took, the response code)

ElijahKotyluk · 2022-06-09T15:08:40Z

@sidharthachatterjee fresh logs:

▲ [WARNING] HTTP 200 /accounts/:account_id/pages/projects/:project/upload-token 0.50s


▲ [WARNING] HTTP 200 /pages/assets/check-missing 1.36s


▲ [WARNING] HTTP 200 /pages/assets/upload 139.55s


▲ [WARNING] HTTP 200 /pages/assets/upload 153.21s


▲ [WARNING] HTTP 200 /pages/assets/upload 172.97s


▲ [WARNING] HTTP 200 /pages/assets/upload 145.21s


▲ [WARNING] HTTP 200 /pages/assets/upload 138.10s


▲ [WARNING] HTTP 200 /pages/assets/upload 128.20s


▲ [WARNING] HTTP 403 /pages/assets/upload 129.70s


▲ [WARNING] HTTP 403 /pages/assets/upload 153.34s


▲ [WARNING] HTTP 403 /pages/assets/upload 144.38s


▲ [WARNING] HTTP 403 /pages/assets/upload 128.99s


▲ [WARNING] HTTP 403 /pages/assets/upload 127.93s

▲ [WARNING] HTTP 403 /pages/assets/upload 144.57s


▲ [WARNING] HTTP 403 /pages/assets/upload 134.71s


▲ [WARNING] HTTP 403 /pages/assets/upload 154.54s


▲ [WARNING] HTTP 403 /pages/assets/upload 142.15s


▲ [WARNING] HTTP 403 /pages/assets/upload 148.50s


▲ [WARNING] HTTP 403 /pages/assets/upload 152.17s


▲ [WARNING] HTTP 403 /pages/assets/upload 136.69s


▲ [WARNING] HTTP 403 /pages/assets/upload 138.48s


🌍  Uploading... (255/7962)

/Users/ek/.npm/_npx/8be3397ebf390916/node_modules/wrangler/wrangler-dist/cli.js:16264
            throw ex;
            ^

FatalError: Failed to upload files. Please try again.
    at /Users/ek/.npm/_npx/8be3397ebf390916/node_modules/wrangler/wrangler-dist/cli.js:123709:31
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async run2 (/Users/elijahkotyluk/.npm/_npx/8be3397ebf390916/node_modules/wrangler/wrangler-dist/cli.js:122141:26) {
  code: 1
}

sidharthachatterjee · 2022-06-09T16:45:25Z

Okay, interesting. So that would imply that the JWT expiry fix didn't work. Can you try this now?

npx https://prerelease-registry.developers.workers.dev/runs/2469819200/npm-package-wrangler-1217

Added a few more logs.

revmischa · 2022-06-09T17:25:35Z

Using
npx https://prerelease-registry.developers.workers.dev/runs/2469819200/npm-package-wrangler-1217 pages publish public

Result:


▲ [WARNING] HTTP 403 /pages/assets/upload 1.09s


▲ [WARNING] Expired JWT


▲ [WARNING] HTTP 403 /pages/assets/upload 1.40s


▲ [WARNING] Expired JWT


🌎  Uploading... (809/8009)

/Users/cyber/.npm/_npx/727f611967a5fdf5/node_modules/wrangler/wrangler-dist/cli.js:16264
            throw ex;
            ^

FatalError: Failed to upload files. Please try again.
    at /Users/cyber/.npm/_npx/727f611967a5fdf5/node_modules/wrangler/wrangler-dist/cli.js:123705:31
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async run2 (/Users/cyber/.npm/_npx/727f611967a5fdf5/node_modules/wrangler/wrangler-dist/cli.js:122136:26) {
  code: 1

It really seems to be much much slower than the K/V upload btw

revmischa · 2022-06-13T19:58:50Z

I'm able to upload everything now eventually but I think it's only because I upgraded my internet connection. It still seems to take a bit longer than it ought to

My site is 339MB
I'm uploading at an average 2.5MB/s
Should take ~135s

But I get: ✨ Success! Uploaded 8009 files (215.32 sec)

ElijahKotyluk · 2022-06-13T23:28:14Z

npx https://prerelease-registry.developers.workers.dev/runs/2469819200/npm-package-wrangler-1217 pages publish is giving me:

▲ [WARNING] HTTP 403 /pages/assets/upload 14.28s


▲ [WARNING] Expired JWT


▲ [WARNING] HTTP 403 /pages/assets/upload 15.91s


▲ [WARNING] Expired JWT


🌏  Uploading... (167/7960)

/Users/ek/.npm/_npx/8be3397ebf390916/node_modules/wrangler/wrangler-dist/cli.js:16264
            throw ex;
            ^

FatalError: Failed to upload files. Please try again.
    at /Users/ek/.npm/_npx/8be3397ebf390916/node_modules/wrangler/wrangler-dist/cli.js:123709:31
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async run2 (/Users/ek/.npm/_npx/8be3397ebf390916/node_modules/wrangler/wrangler-dist/cli.js:122141:26) {
  code: 1
}

ElijahKotyluk · 2022-06-30T12:54:53Z

@sidharthachatterjee I took some time to debug and find a solve for the expiring JWT on a fork, this seems to only be an issue for my larger projects I am deploying through direct uploads. Even with the JWT issue resolved, the uploads are still taking quite a bit longer than I would hope.

GregBrimble · 2022-07-06T14:55:46Z

Hi @ElijahKotyluk , glad to hear you got it working on a fork. If you can, please put up a PR with your changes, but we have added some refresh logic a little while ago, so maybe this won't be necessary anymore with the latest wrangler versions?

We'll close this issue out in the meantime, but please reopen if there's an ongoing issue. Thanks!

ElijahKotyluk · 2022-07-06T15:25:26Z

Hi @GregBrimble, thank you for the response. I just tested the latest and still hit issues with the refetch and speed.

I will get something up today, need to spend some time figuring out the test I broke.

The issue with the current implementation is that the way the error is being checked in the case the JWT expires won't pass, the error caught is a ParseError and the error.code that's being checked does not exist on the error caught, which results in the JWT not getting refetched. I just wrote a helper function isJWTExpired: (error: unknown) => boolean but that broke the refetch JWT test.

GregBrimble · 2022-07-06T15:33:51Z

Ah, okay, thanks for that additional info! Absolutely feel free to put up whatever you have right now and we can help get it over the line 😊

ElijahKotyluk · 2022-07-06T15:56:13Z

@GregBrimble, PR is up. Thanks again.

maisfeldchen · 2022-10-02T20:44:20Z

Hi, i'm having the exact same problem. Any chance for a fix in the near future?

rohit-gohri · 2022-10-21T15:33:12Z

I'm having the same issue with a 90MB project, with around 5840 files

aexol · 2022-10-31T21:32:01Z

same here

ElijahKotyluk · 2022-11-01T16:15:30Z

Still seeing this happen on a regular basis, starting to become a bit of a slowdown for development.

adamsiwiec · 2023-02-28T22:21:14Z

Can not upload a ~300 file project. Maybe the CLI should gzip a project then unzip it on the server?

mbouclas · 2023-04-18T16:17:00Z

Having the same issue. Publishes are hit or miss. Sometimes they finish, most times they don't. Here's what a failed attempt looks like

PBhadoo · 2023-06-14T10:18:56Z

This puts the entire project on hold. Tried Several times.

🌎  Uploading... (380/4308)

C:\Users\bhadoo\AppData\Roaming\npm\node_modules\wrangler\wrangler-dist\cli.js:30632
            throw a;
            ^

FatalError: Failed to upload files. Please try again. Error: {"text":"Received a malformed response from the API","notes":[{"text":"Expired JWT"},{"text":"POST /pages/assets/upload -> 403 Forbidden"}],"kind":"error","name":"ParseError"})
    at C:\Users\bhadoo\AppData\Roaming\npm\node_modules\wrangler\wrangler-dist\cli.js:141131:13
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async run (C:\Users\bhadoo\AppData\Roaming\npm\node_modules\wrangler\wrangler-dist\cli.js:140737:26) {
  code: 1
}

Node.js v18.16.0

anaclumos · 2023-07-06T02:06:47Z

Same

FatalError: Failed to upload files. Please try again. Error: {"cause":{"errno":-32,"code":"EPIPE","syscall":"write"}})
    at /Users/spaceship/Downloads/www/node_modules/.pnpm/wrangler@3.1.1/node_modules/wrangler/wrangler-dist/cli.js:141191:13
    at async run (/Users/spaceship/Downloads/www/node_modules/.pnpm/wrangler@3.1.1/node_modules/wrangler/wrangler-dist/cli.js:140797:26) {
  code: 1
}

What's weird is that on a particular Wi-fi network, it works flawlessly.

PBhadoo · 2023-08-16T04:00:19Z

`🌏 Uploading... (166/2249)

C:\Users\bhadoo\AppData\Roaming\npm\node_modules\wrangler\wrangler-dist\cli.js:30632
throw a;
^

FatalError: Failed to upload files. Please try again. Error: {"text":"Received a malformed response from the API","notes":[{"text":"Expired JWT"},{"text":"POST /pages/assets/upload -> 403 Forbidden"}],"kind":"error","name":"ParseError"})
at C:\Users\bhadoo\AppData\Roaming\npm\node_modules\wrangler\wrangler-dist\cli.js:141734:13
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async run (C:\Users\bhadoo\AppData\Roaming\npm\node_modules\wrangler\wrangler-dist\cli.js:141340:26) {
code: 1
}`

Wrangler 3.5.1

Can't refresh JWT Token. (happened after 10 minutes maybe?)
@sidharthachatterjee Please fix this problem.

PBhadoo · 2023-08-16T11:34:19Z

@sidharthachatterjee

I guess, you added this

workers-sdk/packages/wrangler/src/pages/upload.tsx

Line 320 in 478bed3

if ((e as { code: number }).code === 8000013 || isJwtExpired(jwt)) {

but this doesn't fix the problem
yet this triggers

workers-sdk/packages/wrangler/src/pages/upload.tsx

Line 338 in 478bed3

(error) => {

Which can be fixed by some method where the error code can be extracted and checked and returned with

jwt = await fetchJwt();
return doUpload();

For me this temporary Fixed Code Worked.

  for (const bucket of buckets) {
    if (bucket.files.length === 0)
      continue;
    attempts = 0;
    const doUpload = /* @__PURE__ */ __name(async () => {
      const payload = await Promise.all(
        bucket.files.map(async (file) => ({
          key: file.hash,
          value: (await (0, import_promises12.readFile)(file.path)).toString("base64"),
          metadata: {
            contentType: file.contentType
          },
          base64: true
        }))
      );
      try {
        logger.debug("POST /pages/assets/upload");
        const res = await fetchResult(`/pages/assets/upload`, {
          method: "POST",
          headers: {
            "Content-Type": "application/json",
            Authorization: `Bearer ${jwt}`
          },
          body: JSON.stringify(payload)
        });
        logger.debug("result:", res);
      } catch (e2) {
        if (attempts < 1000) {
          logger.log("failed:", e2, "retrying... ", attempts);
          await new Promise(
            (resolvePromise) => setTimeout(resolvePromise, Math.pow(2, attempts++) * 1e3)
          );
          if (e2.code === 8000013 || isJwtExpired(jwt)) {
            jwt = await fetchJwt();
          }
          return doUpload();
        } else {
          logger.debug("failed:", e2);
          throw e2;
        }
      }
    }, "doUpload");
    void queue.add(() => {
      return Promise.resolve(doUpload())
        .then(
          () => {
            counter += bucket.files.length;
            rerender(counter, fileMap.size);
          },
          async (error) => {
            logger.log("caught error, retrying with failed problem");
            // Fetch JWT token here
            jwt = await fetchJwt();
            return doUpload();
          /*return Promise.reject(
            new FatalError(
              `Failed to upload files. Please try again. Error: ${JSON.stringify(
                error
              )})`,
              error.code || 1
            )
          );*/
          }
        );
    });
  }

also MAX_UPLOAD_ATTEMPTS is set to 5 is very low.
The user should be able send this as parameters.
because i uploaded 2000 files
and for first 660 files it took around 11 attempts while 10 of them are jwt expire.

I hope this helps.

thangisme · 2023-10-03T09:58:53Z

I'm also experiencing this problem:

/root/project/node_modules/wrangler/wrangler-dist/cli.js:30947
            throw a;
            ^

FatalError: Failed to upload files. Please try again. Error: {"text":"Received a malformed response from the API","notes":[{"text":"Expired JWT"},{"text":"POST /pages/assets/upload -> 403 Forbidden"}],"kind":"error","name":"ParseError"})
    at /root/project/node_modules/wrangler/wrangler-dist/cli.js:142760:13
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async run (/root/project/node_modules/wrangler/wrangler-dist/cli.js:142289:26) {
  code: 1
}

Node.js v20.7.0

Exited with code exit status 7

GregBrimble · 2024-01-05T16:00:08Z

Hey folks! We've been incrementally making reliability improvements to our service and to Wrangler, and we're hopefully in a place where any issues with wrangler pages deploy are now very uncommon. If you see consistent failures, please let us know, but I'll close out this issue in the meantime.

nolanlawson · 2024-01-15T03:04:50Z

I'm running into this same issue with wrangler pages deploy, on a large directory (6k+ files) with several large files (0.5MB max file size). Not sure if any of this information helps, but here's what I'm seeing:

FatalError: Failed to upload files. Please try again. Error: {"text":"Received a malformed response from the API","notes":[{"text":"Expired JWT"},{"text":"POST /pages/assets/upload -> 403 Forbidden"}],"kind":"error","name":"ParseError"})
    at .../tools/image/packages/wrangler/lib/node_modules/wrangler/wrangler-dist/cli.js:140972:13
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async run (.../tools/image/packages/wrangler/lib/node_modules/wrangler/wrangler-dist/cli.js:140496:26) {
  code: 1
}

Node.js v20.9.0

Wrangler version is 3.22.4. I've retried several times, and it always fails after ~15 minutes of trying to upload the files.

revmischa added the bug Something that isn't working label Jun 6, 2022

threepointone added the pages Relating to Pages label Jun 6, 2022

sidharthachatterjee mentioned this issue Jun 7, 2022

fix: Refresh JWT in wrangler pages publish when it expires #1199

Merged

GregBrimble closed this as completed Jul 6, 2022

GregBrimble reopened this Jul 6, 2022

WalshyDev added the pages:validated A validated Pages issue label Oct 19, 2022

PBhadoo mentioned this issue Aug 16, 2023

🐛 BUG: Expired JWT #3773

Closed

GregBrimble closed this as completed Jan 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 BUG: pages publish very slow / crashing #1194

🐛 BUG: pages publish very slow / crashing #1194

revmischa commented Jun 6, 2022 •

edited

sidharthachatterjee commented Jun 7, 2022

ElijahKotyluk commented Jun 7, 2022

revmischa commented Jun 7, 2022

sidharthachatterjee commented Jun 8, 2022

ElijahKotyluk commented Jun 8, 2022

sidharthachatterjee commented Jun 9, 2022

ElijahKotyluk commented Jun 9, 2022 •

edited

sidharthachatterjee commented Jun 9, 2022

revmischa commented Jun 9, 2022

revmischa commented Jun 13, 2022

ElijahKotyluk commented Jun 13, 2022

ElijahKotyluk commented Jun 30, 2022

GregBrimble commented Jul 6, 2022

ElijahKotyluk commented Jul 6, 2022

GregBrimble commented Jul 6, 2022

ElijahKotyluk commented Jul 6, 2022

maisfeldchen commented Oct 2, 2022

rohit-gohri commented Oct 21, 2022

aexol commented Oct 31, 2022

ElijahKotyluk commented Nov 1, 2022

adamsiwiec commented Feb 28, 2023 •

edited

mbouclas commented Apr 18, 2023

PBhadoo commented Jun 14, 2023

anaclumos commented Jul 6, 2023

PBhadoo commented Aug 16, 2023

PBhadoo commented Aug 16, 2023 •

edited

thangisme commented Oct 3, 2023

GregBrimble commented Jan 5, 2024

nolanlawson commented Jan 15, 2024 •

edited

🐛 BUG: pages publish very slow / crashing #1194

🐛 BUG: pages publish very slow / crashing #1194

Comments

revmischa commented Jun 6, 2022 • edited

What version of Wrangler are you using?

What operating system are you using?

Describe the Bug

sidharthachatterjee commented Jun 7, 2022

ElijahKotyluk commented Jun 7, 2022

revmischa commented Jun 7, 2022

sidharthachatterjee commented Jun 8, 2022

ElijahKotyluk commented Jun 8, 2022

sidharthachatterjee commented Jun 9, 2022

ElijahKotyluk commented Jun 9, 2022 • edited

sidharthachatterjee commented Jun 9, 2022

revmischa commented Jun 9, 2022

revmischa commented Jun 13, 2022

ElijahKotyluk commented Jun 13, 2022

ElijahKotyluk commented Jun 30, 2022

GregBrimble commented Jul 6, 2022

ElijahKotyluk commented Jul 6, 2022

GregBrimble commented Jul 6, 2022

ElijahKotyluk commented Jul 6, 2022

maisfeldchen commented Oct 2, 2022

rohit-gohri commented Oct 21, 2022

aexol commented Oct 31, 2022

ElijahKotyluk commented Nov 1, 2022

adamsiwiec commented Feb 28, 2023 • edited

mbouclas commented Apr 18, 2023

PBhadoo commented Jun 14, 2023

anaclumos commented Jul 6, 2023

PBhadoo commented Aug 16, 2023

PBhadoo commented Aug 16, 2023 • edited

thangisme commented Oct 3, 2023

GregBrimble commented Jan 5, 2024

nolanlawson commented Jan 15, 2024 • edited

revmischa commented Jun 6, 2022 •

edited

What version of `Wrangler` are you using?

ElijahKotyluk commented Jun 9, 2022 •

edited

adamsiwiec commented Feb 28, 2023 •

edited

PBhadoo commented Aug 16, 2023 •

edited

nolanlawson commented Jan 15, 2024 •

edited