Progress UploadStates no longer emitted

[Spiral1401]

Describe the bug
This is a strange one, this hasn't been a problem until very recently but also not sure exactly when it began.
Only happens for configuration/AOT built angular (ie ng build --configuration=dev etc.). Doesn't happen when using app via "ng serve".

I use PrimeNG's fileupload component combined with ngx-uploadx UploadService.

Did some simple debugging with console.log messages:

this.options = {
      endpoint: APP_CONFIG.apiEndpoint + '/api/v1/document/docFileUploadInit',
      chunkSize: 2097152,
      concurrency: 5,
      uploaderClass: UploaderXCustom,
      autoUpload: false,
      token: () => this.authService.getToken()
    };
this.uploadService.init(this.options).subscribe((item: UploadState) => {
      console.log(JSON.stringify(item, null, 2));
      ...
});

When local via ng serve (this is where things work normally):

{
  "file": {},
  "name": "dummy2.txt",
  "size": 10485760,
  "status": "added",
  "uploadId": "f02b88a6",
  "url": ""
}
{
  "file": {},
  "name": "dummy2.txt",
  "size": 10485760,
  "status": "queue",
  "uploadId": "f02b88a6",
  "url": ""
}
{
  "file": {},
  "name": "dummy2.txt",
  "responseStatus": 0,
  "size": 10485760,
  "status": "uploading",
  "uploadId": "f02b88a6",
  "url": ""
}
{
  "file": {},
  "name": "dummy2.txt",
  "progress": 6.41,
  "remaining": 67,
  "responseStatus": 0,
  "size": 10485760,
  "speed": 147798,
  "status": "uploading",
  "uploadId": "f02b88a6",
  "url": "https://localhost:44356/api/v1/document/docFileUpload?tokenId=51baf61c-9284-47e7-9c3f-137bbc1f8be5"
}
{
  "file": {},
  "name": "dummy2.txt",
  "progress": 40,
  "remaining": 11,
  "responseStatus": 0,
  "size": 10485760,
  "speed": 582704,
  "status": "uploading",
  "uploadId": "f02b88a6",
  "url": "https://localhost:44356/api/v1/document/docFileUpload?tokenId=51baf61c-9284-47e7-9c3f-137bbc1f8be5"
}
{
  "file": {},
  "name": "dummy2.txt",
  "progress": 60,
  "remaining": 7,
  "responseStatus": 0,
  "size": 10485760,
  "speed": 644881,
  "status": "uploading",
  "uploadId": "f02b88a6",
  "url": "https://localhost:44356/api/v1/document/docFileUpload?tokenId=51baf61c-9284-47e7-9c3f-137bbc1f8be5"
}
{
  "file": {},
  "name": "dummy2.txt",
  "progress": 80,
  "remaining": 4,
  "responseStatus": 0,
  "size": 10485760,
  "speed": 694479,
  "status": "uploading",
  "uploadId": "f02b88a6",
  "url": "https://localhost:44356/api/v1/document/docFileUpload?tokenId=51baf61c-9284-47e7-9c3f-137bbc1f8be5"
}
{
  "file": {},
  "name": "dummy2.txt",
  "progress": 100,
  "remaining": 0,
  "responseStatus": 0,
  "size": 10485760,
  "speed": 729444,
  "status": "uploading",
  "uploadId": "f02b88a6",
  "url": "https://localhost:44356/api/v1/document/docFileUpload?tokenId=51baf61c-9284-47e7-9c3f-137bbc1f8be5"
}
{
  "file": {},
  "name": "dummy2.txt",
  "progress": 100,
  "remaining": 0,
  "response": 1274303,
  "responseStatus": 200,
  "size": 10485760,
  "speed": 729444,
  "status": "complete",
  "uploadId": "f02b88a6",
  "url": "https://localhost:44356/api/v1/document/docFileUpload?tokenId=51baf61c-9284-47e7-9c3f-137bbc1f8be5"
}

When running via angular-http-server on a ng build --configuration=dev build (AOT, etc.):

{
  "file": {},
  "name": "dummy2.txt",
  "size": 10485760,
  "status": "added",
  "uploadId": "182e259e",
  "url": ""
}
{
  "file": {},
  "name": "dummy2.txt",
  "size": 10485760,
  "status": "queue",
  "uploadId": "182e259e",
  "url": ""
}
{
  "file": {},
  "name": "dummy2.txt",
  "responseStatus": 0,
  "size": 10485760,
  "status": "uploading",
  "uploadId": "182e259e",
  "url": ""
}
{
  "file": {},
  "name": "dummy2.txt",
  "progress": 100,
  "remaining": 0,
  "response": 1274303,
  "responseStatus": 200,
  "size": 10485760,
  "status": "complete",
  "uploadId": "182e259e",
  "url": "https://localhost:44356/api/v1/document/docFileUpload?tokenId=39aafa11-20e1-469c-80f6-b12831915c59"
}

For some reason the progress events are skipped right over. Also happens on our deployed Azure DEV environment. It otherwise behaves exactly as it should - the same number of requests, responses, chunks, etc. Just no progress states emitted. Basically the only issue is my progress bar UI no longer shows anything. It just sits at 0% until final chunk is sent when it jumps to 100%.

At first I tried reverting to the default uploaderClass (I had it overridden with my own) by not defining one in UploadxOptions - and altered my server responses to be compatible. No luck.

Then I updated ngx-uploadx from 3.1.4 to 3.3.3. No luck.

To Reproduce
Not sure it'd be worth trying to reproduce my exact scenario as it's kind of involved, more looking for advice / whether anyone has seen anything like this.

Setup details:

• Used ngx-uploadx: 3.1.4 (later updated to 3.3.3)
• Used Angular version : Angular 8
• Used browser and OS: Chrome/Windows
• Used server software .NET WebAPI

Additional context
Strangely seems independent of any relevant version changes/etc., but not 100% sure here.

[kukhariev]

Hi @Spiral1401!

I can't reproduce it yet.
Lib throttles progress events with 500 ms window, but at least one event per chunk.
Angular v8.2.14, CLI v8.3.25, aot=true, buildOptimizer=true.

{
  "file": {},
  "name": "dummy2.txt",
  "progress": 100,
  "remaining": 0,
  "response": 1274303,
  "responseStatus": 200,
  "size": 10485760,
  "status": "complete",
  "uploadId": "182e259e",
  "url": "https://localhost:44356/api/v1/document/docFileUpload?tokenId=39aafa11-20e1-469c-80f6-b12831915c59"
}

There's no "speed" field , which means that either there's no real content upload or there's something wrong with sendFileContent method.

[Spiral1401]

That's interesting, I didn't catch the missing "speed" field. However like I said the upload is working perfectly otherwise - I can breakpoint and see the contents through the chain - and when it completes I can browse for it in our Azure Storage and download the resultant file.

The problem seems to exist regardless of if I override the default uploaderClass with my own implementation, but here is what mine looks like:

import { Uploader, UploadxOptions } from 'ngx-uploadx';

/**
 * Implements XHR/CORS Resumable Upload
 * @see
 * https://developers.google.com/drive/v3/web/resumable-upload
 */
export class UploaderXCustom extends Uploader {
  constructor(readonly file: File, options: UploadxOptions) {
    super(file, options);
    this.responseType = 'json' as XMLHttpRequestResponseType;
  }

  async getFileUrl(): Promise<string> {
    const headers = {
      'Content-Type': 'application/json; charset=UTF-8',
      'X-Upload-Content-Length': `${this.size}`
      // 'X-Upload-Content-Type': `${this.mimeType}`
    };
    const body = JSON.stringify(this.metadata);
    const _ = await this.request({
      method: 'POST',
      body,
      url: this.endpoint,
      headers
    });
    const location =
      this.responseStatus === 200 && this.getValueFromResponse('location');
    this.offset = this.responseStatus === 201 ? 0 : undefined;
    return this.resolveUrl(location, this.endpoint);
  }

  async sendFileContent(): Promise<number> {
    const end = this.chunkSize
      ? Math.min(this.offset + this.chunkSize, this.size)
      : this.size;
    const body = this.file.slice(this.offset, end);
    const headers = {
      'Content-Type': 'application/octet-stream',
      'Content-Range': `bytes ${this.offset}-${end - 1}/${this.size}`
    };
    const _ = await this.request({
      method: 'PUT',
      body,
      url: this.url,
      headers
    });
    console.log('req headers: ' + JSON.stringify(headers, null, 2));
    return this.getOffsetFromResponse();
  }

  async getOffset(): Promise<number> {
    const headers = {
      'Content-Type': 'application/octet-stream',
      'Content-Range': `bytes */${this.size}`
    };
    const _ = await this.request({
      method: 'PUT',
      url: this.url,
      headers
    });
    return this.getOffsetFromResponse();
  }

  protected getOffsetFromResponse() {
    if (this.responseStatus === 308) {
      const str = this.getValueFromResponseCust('Range');
      if (!str) {
        return 0;
      }
      const [match] = str && str.match(/(-1|\d+)$/g);
      return match && +match + 1;
    } else if (this.responseStatus === 200) {
      return this.size;
    }
    return;
  }
  protected onCancel(): void {
    // this.request({ method: 'DELETE' });
  }

  protected setAuth(token: string) {
    this.headers.Authorization = `Bearer ${token}`;
  }

  protected resolveUrl(url, baseURI) {
    if (
      url.indexOf('//') * url.indexOf('https://') * url.indexOf('http://') ===
      0
    ) {
      return url;
    }
    try {
      const res = new URL(url, baseURI).href;
      return res;
    } catch (_a) {
      if (url.indexOf('/') === 0) {
        const matches = baseURI.match(/^(?:https?:)?(?:\/\/)?([^\/\?]+)/g);
        const origin = matches && matches[0];
        return origin + url;
      } else {
        const matches = baseURI.match(
          /^(?:https?:)?(?:\/\/)?([^\/\?]+)?(.*\/)/g
        );
        const path = matches && matches[0];
        return path + url;
      }
    }
  }

  protected getValueFromResponseCust(key) {
    const value = this._xhr.getResponseHeader(key);
    const headers = this._xhr.getAllResponseHeaders();
    console.log('resp headers: ' + JSON.stringify(headers, null, 2));
    if (!value) {
      return null;
    }
    return value;
  }
}

It was based on an earlier version of the library default uploader. My server response is set up to send back a 308 response with no "Range" header when ngx-uploadx is asking for the offset (and there isn't one). Hence why I do what I do in "getValueFromResponseCust". But I can also make the server send back "Range: bytes=0--1" for that scenario and allow it to work with the default class rather than my version.

Here's what the logging looks like for my uploader class and it's console.log statements:

{
  "file": {},
  "name": "dummy2.txt",
  "size": 10485760,
  "status": "added",
  "uploadId": "182e259e",
  "url": ""
}
{
  "file": {},
  "name": "dummy2.txt",
  "size": 10485760,
  "status": "queue",
  "uploadId": "182e259e",
  "url": ""
}
{
  "file": {},
  "name": "dummy2.txt",
  "responseStatus": 0,
  "size": 10485760,
  "status": "uploading",
  "uploadId": "182e259e",
  "url": ""
}
resp headers: "content-length: 7\r\ncontent-type: application/json; charset=utf-8\r\n"
req headers: {
  "Content-Type": "application/octet-stream",
  "Content-Range": "bytes 0-2097151/10485760"
}
resp headers: "content-length: 7\r\ncontent-type: application/json; charset=utf-8\r\nrange: bytes=0-2097151\r\n"
req headers: {
  "Content-Type": "application/octet-stream",
  "Content-Range": "bytes 2097152-4194303/10485760"
}
resp headers: "content-length: 7\r\ncontent-type: application/json; charset=utf-8\r\nrange: bytes=2097152-4194303\r\n"
req headers: {
  "Content-Type": "application/octet-stream",
  "Content-Range": "bytes 4194304-6291455/10485760"
}
resp headers: "content-length: 7\r\ncontent-type: application/json; charset=utf-8\r\nrange: bytes=4194304-6291455\r\n"
req headers: {
  "Content-Type": "application/octet-stream",
  "Content-Range": "bytes 6291456-8388607/10485760"
}
resp headers: "content-length: 7\r\ncontent-type: application/json; charset=utf-8\r\nrange: bytes=6291456-8388607\r\n"
req headers: {
  "Content-Type": "application/octet-stream",
  "Content-Range": "bytes 8388608-10485759/10485760"
}
{
  "file": {},
  "name": "dummy2.txt",
  "progress": 100,
  "remaining": 0,
  "response": 1274308,
  "responseStatus": 200,
  "size": 10485760,
  "status": "complete",
  "uploadId": "182e259e",
  "url": "https://localhost:44356/api/v1/document/docFileUpload?tokenId=4180095c-a37e-499e-9520-f32a0f101874"
}

*Another difference in my server response when using my own uploader class is that the response is the range written for that particular request i.e. "range: bytes=6291456-8388607" rather than "range: bytes=0-8388607" ("all that has been written") with the default class, but I can make it work either way - and the issue persists in both scenarios.

[kukhariev]

Yeah, it'll force the progress events:

async sendFileContent(): Promise<number> {
//  ...
    const _ = await this.request({
      method: 'PUT',
      body,
      url: this.url,
      headers,
      progress: true // <---  force progress events
    });

    console.log(this._xhr.upload.onprogress);  //  should never be null

// ...
}

[Spiral1401]

Unfortunately I'm not seeing a difference. Here's my updated sendFileContent():

async sendFileContent(): Promise<number> {
    const end = this.chunkSize
      ? Math.min(this.offset + this.chunkSize, this.size)
      : this.size;
    const body = this.file.slice(this.offset, end);
    const headers = {
      'Content-Type': 'application/octet-stream',
      'Content-Range': `bytes ${this.offset}-${end - 1}/${this.size}`
    };
    const blobLength = body.size;
    const reqObjForPrint = {
      method: 'PUT',
      bodyLen: blobLength,
      url: this.url,
      headers,
      progress: true
    };
    const _ = await this.request({
      method: 'PUT',
      body,
      url: this.url,
      headers,
      progress: true
    });
    console.log('req object: ' + JSON.stringify(reqObjForPrint, null, 2));
    return this.getOffsetFromResponse();
  }

(I printed out a slightly different object to avoid console.logging the entire body contents)

And the resulting logs:

{
  "file": {},
  "name": "dummy2.txt",
  "size": 10485760,
  "status": "added",
  "uploadId": "182e259e",
  "url": ""
}
{
  "file": {},
  "name": "dummy2.txt",
  "size": 10485760,
  "status": "queue",
  "uploadId": "182e259e",
  "url": ""
}
{
  "file": {},
  "name": "dummy2.txt",
  "responseStatus": 0,
  "size": 10485760,
  "status": "uploading",
  "uploadId": "182e259e",
  "url": ""
}
resp headers: "content-length: 7\r\ncontent-type: application/json; charset=utf-8\r\n"
req object: {
  "method": "PUT",
  "bodyLen": 2097152,
  "url": "https://localhost:44356/api/v1/document/docFileUpload?tokenId=ad0ef69b-ec44-4009-8276-76116da78b59",
  "headers": {
    "Content-Type": "application/octet-stream",
    "Content-Range": "bytes 0-2097151/10485760"
  },
  "progress": true
}
resp headers: "content-length: 7\r\ncontent-type: application/json; charset=utf-8\r\nrange: bytes=0-2097151\r\n"
req object: {
  "method": "PUT",
  "bodyLen": 2097152,
  "url": "https://localhost:44356/api/v1/document/docFileUpload?tokenId=ad0ef69b-ec44-4009-8276-76116da78b59",
  "headers": {
    "Content-Type": "application/octet-stream",
    "Content-Range": "bytes 2097152-4194303/10485760"
  },
  "progress": true
}
resp headers: "content-length: 7\r\ncontent-type: application/json; charset=utf-8\r\nrange: bytes=2097152-4194303\r\n"
req object: {
  "method": "PUT",
  "bodyLen": 2097152,
  "url": "https://localhost:44356/api/v1/document/docFileUpload?tokenId=ad0ef69b-ec44-4009-8276-76116da78b59",
  "headers": {
    "Content-Type": "application/octet-stream",
    "Content-Range": "bytes 4194304-6291455/10485760"
  },
  "progress": true
}
resp headers: "content-length: 7\r\ncontent-type: application/json; charset=utf-8\r\nrange: bytes=4194304-6291455\r\n"
req object: {
  "method": "PUT",
  "bodyLen": 2097152,
  "url": "https://localhost:44356/api/v1/document/docFileUpload?tokenId=ad0ef69b-ec44-4009-8276-76116da78b59",
  "headers": {
    "Content-Type": "application/octet-stream",
    "Content-Range": "bytes 6291456-8388607/10485760"
  },
  "progress": true
}
resp headers: "content-length: 7\r\ncontent-type: application/json; charset=utf-8\r\nrange: bytes=6291456-8388607\r\n"
req object: {
  "method": "PUT",
  "bodyLen": 2097152,
  "url": "https://localhost:44356/api/v1/document/docFileUpload?tokenId=ad0ef69b-ec44-4009-8276-76116da78b59",
  "headers": {
    "Content-Type": "application/octet-stream",
    "Content-Range": "bytes 8388608-10485759/10485760"
  },
  "progress": true
}
{
  "file": {},
  "name": "dummy2.txt",
  "progress": 100,
  "remaining": 0,
  "response": 1274309,
  "responseStatus": 200,
  "size": 10485760,
  "status": "complete",
  "uploadId": "182e259e",
  "url": "https://localhost:44356/api/v1/document/docFileUpload?tokenId=ad0ef69b-ec44-4009-8276-76116da78b59"
}

[kukhariev]

You need at least v3.3.1 or copy request method from uploader.ts to UploaderXCustom.ts.
v3.3.1 fixes angular aot instanceof bug.

[Spiral1401]

I am using ngx-uploadx 3.3.3. I mention in the original post that I started with 3.1.4 but I already upgraded to latest in an attempt to fix the issue.

Tried copying the request method from ngx-uploadx source into my UploaderXCustom class anyway but things in the base Uploader class being private (onProgress(), startTime, stateChange() ) are making it tricky to do without copying several other things into my own class.

Have to switch gears on my end for a few hours at least but I'll jump back on this soon. Appreciate your help.

[Spiral1401]

Actually, I think I just found the issue. It fits too perfectly. And I only just started using the service worker semi recently in this particular app, so it lines up time-wise as well.
angular/angular#24683

ng serve does not use the angular service worker, while all the other methods of running the app I am using do. So... this has to be it. And I imagine, there is nothing you can do to fix it - but it looks like I can modify my service worker configuration to allow it to work.

I'll try to drop some further info in here if I get the fix working (might help with documenting it for other people).

[Spiral1401]

Angular service worker provides a header to allow requests to bypass being handled by it. Mentioned here:
angular/angular#21191

Modified the headers in the sendFileContent() of my custom Uploader class:

const headers = {
      'Content-Type': 'application/octet-stream',
      'Content-Range': `bytes ${this.offset}-${end - 1}/${this.size}`,
      'ngsw-bypass': 'true'
    };

Also (somewhat beside the point) had to edit my serverside web.config "Access-Control-Allow-Headers" to include "ngsw-bypass".

Tested and working, progress events are happening again.