Sending really large response sends duplicate

[DaftMonk]

When responding with a very large json object, a duplicate is sent along with it when using the compression middleware.

E.g. the response on my endpoint in chrome and firefox is {_id: 1212, ...... }{_id: 1212, ...... }. When I remove the compression middleware it correctly sends one object.

Original issue: angular-fullstack/generator-angular-fullstack#380

[dougwilson]

Hi! I need a complete app that I can use to reproduce the issue, as well as what version of this module you are experiencing that behavior with.

[dougwilson]

According to what you referenced (https://stackoverflow.com/questions/24836097/duplicate-chunks-on-json-response) it breaks if the response is over 65536 bytes. It does not. I've added tests that prove the case: a test transfers 1mb of data which is not duplicated and a second test transfers 160kb data in 4 chunks and it is not duplicated.

I have nothing I can act on, so I'm closing this issue since as far as I can tell, it is either a bug in however that enormous app is interacting with this module or something else. I cannot debug that generated app, I need you to provide some app that just uses this module (and maybe express) to reproduce this issue.

You may also just want to verify that you are using the latest version of this module, because maybe there was a bug but it was fixed, IDK.

[kingcody]

@dougwilson I believe I can help reproduce the error that @DaftMonk originally had.

I added a test that tries to send large bodies with 'Content-Type application/json'; which fails with supertest reporting that the callback was called twice.

I'm currently trying to get to the root of the issue but it might be much more apparent to someone with more expirence with the project and its libs.

[dougwilson]

Your test is failing because you never called done. You need to add done as the third argument to your last expect.

[dougwilson]

As in, you need to have your last expect like this:

.expect(200, buf.toString(), done)

[kingcody]

@dougwilson that was actually a last minute error, my mistake. I've corrected and the issue still stands.

Thank you for your reply.

[dougwilson]

I've corrected and the issue still stands.

What is the issue that still stands? You have only provided a link to that broken test with the broken Travis from the forgotten done argument. Are there new links you need to provide me? I don't have additional information here beyond the missing done argument in the test.

[kingcody]

Are there new links you need to provide me?

Updated commit
Updated test

[dougwilson]

Thanks. The error in Travis is yet another supertest error or something. I will not have access to a computer for quite a long time, so unless there is a PR you can make, I cannot promise it will be looked into anytime soon.

[dougwilson]

It's too bad that stack trace says nothing. It almost looks like the error is occurring within Node.js core itself, based on the stack trace, which is only a single entry (at least looking at Travis CI on a mobile phone).

[dougwilson]

Oh, I see the issue. The problem is that supertest is actually trying to JSON.parse the response, because of the content type. Your test is failing because of that, yet again a supertest issue. If you are going to send JSON, you need to write actual JSON to get through supertest.