Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trying to query documents results in a Parse Error: HPE_HEADER_OVERFLOW #221

Closed
petmat opened this issue Jan 18, 2019 · 22 comments
Closed

Trying to query documents results in a Parse Error: HPE_HEADER_OVERFLOW #221

petmat opened this issue Jan 18, 2019 · 22 comments
Labels
bug

Comments

@petmat
Copy link

@petmat petmat commented Jan 18, 2019

I am using the @azure/cosmos NPM package to query documents from CosmosDB and it results in the following error:

{ Error: Parse Error
    at TLSSocket.socketOnData (_http_client.js:442:20)
    at TLSSocket.emit (events.js:182:13)
    at addChunk (_stream_readable.js:283:12)
    at readableAddChunk (_stream_readable.js:264:11)
    at TLSSocket.Readable.push (_stream_readable.js:219:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:94:17)
  bytesParsed: 0,
  code: 'HPE_HEADER_OVERFLOW',
  headers:
   { 'x-ms-throttle-retry-count': 1,
     'x-ms-throttle-retry-wait-time-ms': 4059 } }

@azure/cosmos@2.1.1
Node.js 10.15.0

I'm using the following code:

const cosmos = require('@azure/cosmos');
const config = require('./local.settings.json');

const { CosmosClient } = cosmos;

const { endpoint, key } = config;
console.log(endpoint, key);
const client = new CosmosClient({ endpoint, auth: { key } });

const dbId = 'DB';
const containerId = 'Items';

async function getChats() {
  const querySpec =
    'SELECT * FROM c WHERE c.type = "ChatDefinition" AND c.roomId = "062db3be-339e-11e6-bebb-00163e75537f" AND c.lastEntry >= "2019-01-01"';
  const { result } = await client
    .database(dbId)
    .container(containerId)
    .items.query(querySpec)
    .toArray();
  return result;
}

async function run() {
  console.log('Gettings chats...');
  const chats = await getChats();
  console.log(chats.length);
}

run().catch(console.error);
@southpolesteve

This comment has been minimized.

Copy link
Member

@southpolesteve southpolesteve commented Jan 18, 2019

@petmat Thanks for the detailed report. It was super helpful in tracking the issue down. It appears that node decreased the max header size from 80kb to 8kb in the last release nodejs/node@1860352 and https://github.com/nodejs/node/releases/tag/v10.15.0

Could you try on node 10.14.10 and see if the error still exists? It also looks like you can work around by setting the --max-http-header-size flag on the node CLI.

I'll work with the API team over here too. Unfortunately the Cosmos REST API sends a decent amount of data via headers. Changing that would take some time.

@southpolesteve

This comment has been minimized.

Copy link
Member

@southpolesteve southpolesteve commented Jan 18, 2019

@petmat One more Q. Roughly how much data is in this collection and what is the partition key? Feel free to email me if you don't want to disclose here: stfaul@microsoft.com

@Rafael-Ferreiro

This comment has been minimized.

Copy link

@Rafael-Ferreiro Rafael-Ferreiro commented Feb 27, 2019

I have the same problem, the functio call to cosmos api and return

"message": "Error en function{\"bytesParsed\":16383,\"code\":\"HPE_HEADER_OVERFLOW\",\"headers\":{\"x-ms-throttle-retry-count\":0,\"x-ms-throttle-retry-wait-time-ms\":0}}"
@southpolesteve

This comment has been minimized.

Copy link
Member

@southpolesteve southpolesteve commented Feb 27, 2019

@rafa98765 Could you try on node 10.14.10 and see if the error still exists? I've not been able to reproduce the error yet or confirm it is related to the upstream change that nodejs made

@Rafael-Ferreiro

This comment has been minimized.

Copy link

@Rafael-Ferreiro Rafael-Ferreiro commented Feb 27, 2019

Yes, the error is in 10.14.1 version of node. But only when I get values for 2 cosmos db. And we made a migration between 2 cosmos, with datafactory v2, it is relevant?

@bertschneider

This comment has been minimized.

Copy link

@bertschneider bertschneider commented Feb 28, 2019

FYI: We got around this issue adding the --max-http-header-size node flag.

@bialad

This comment has been minimized.

Copy link

@bialad bialad commented Mar 21, 2019

I've also encountered this error while querying with a result of just 150 values+timestamp.

I don't understand how such a small payload can overload the header limit. I'm using cosmos db as a IoT telemetry storage, where I'm expecting to query for thousands of data points at a time, which doesn't seem to be what it's made for with this limit.

@southpolesteve

This comment has been minimized.

Copy link
Member

@southpolesteve southpolesteve commented Mar 21, 2019

@bialad Can you provide a sample query or some data? I'd really like to figure out this problem but I've had trouble replicating this issue. My email is a few comments above if that is preferable.

@bialad

This comment has been minimized.

Copy link

@bialad bialad commented Apr 5, 2019

This is the error I get:

{ Error: Parse Error
    at TLSSocket.socketOnData (_http_client.js:447:20)
    at TLSSocket.emit (events.js:197:13)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at TLSSocket.Readable.push (_stream_readable.js:224:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:150:17)
  bytesParsed: 16383,
  code: 'HPE_HEADER_OVERFLOW',
  headers:
   { 'x-ms-throttle-retry-count': 0,
     'x-ms-throttle-retry-wait-time-ms': 0 } }
(node:30605) UnhandledPromiseRejectionWarning: TypeError: Cannot destructure property `result` of 'undefined' or 'null'.
    at Object.queryContainer (/data/projects/tobias/cloud-iot-test/azure/website/data-api/db.service.js:27:31)
    at processTicksAndRejections (internal/process/next_tick.js:81:5)
(node:30605) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 4)

This is the query being sent:

  const querySpec = {
    query: `SELECT TOP 150 c["value"], c["sourcetimestamp"]
              FROM opc c 
              WHERE c.nodeid = @nodeid AND c.sourcetimestamp > @from
              ORDER BY c.sourcetimestamp DESC`,
    parameters: [
      {
        name: "@nodeid",
        value: req.body.nodeid
      },
      {
        name: "@from",
        value: req.body.from
      }
    ]
  };

This is the response of the same query with TOP 10 (which works):

[{"value":25,"sourcetimestamp":"2019-04-05T09:32:09.95375Z"},{"value":27,"sourcetimestamp":"2019-04-05T09:31:59.95375Z"},{"value":30,"sourcetimestamp":"2019-04-05T09:31:49.95375Z"},{"value":32,"sourcetimestamp":"2019-04-05T09:31:39.95375Z"},{"value":38,"sourcetimestamp":"2019-04-05T09:31:29.95375Z"}]
@tony-gutierrez

This comment has been minimized.

Copy link

@tony-gutierrez tony-gutierrez commented Apr 9, 2019

Same problem with pretty small query in node 10.14.1

@tony-gutierrez

This comment has been minimized.

Copy link

@tony-gutierrez tony-gutierrez commented Apr 9, 2019

@southpolesteve Verified that this issue was caused by the newer version of cosmosdb that has the database wide dtu limit. The older version with collection scope dtu works fine. I can make a query work on the newer db by vastly restricting the result set, the same size is handled fine by the old.

@tony-gutierrez

This comment has been minimized.

Copy link

@tony-gutierrez tony-gutierrez commented Apr 9, 2019

Be advised that Azure App Service does not support any node version with the "--max-http-header-size" flag. So your DB is now useless on your own app service.

@tony-gutierrez

This comment has been minimized.

Copy link

@tony-gutierrez tony-gutierrez commented Apr 9, 2019

For those wondering, node 10.13.0 is before the introduced 8kb limit. On App services the newest before the limit is 10.6.0.

@m4recek

This comment has been minimized.

Copy link

@m4recek m4recek commented May 2, 2019

Hi, did this moved somewhere?

We have replicated the issue on Node 8.15.0 for fairly trivial queries. We are not able to set --max-http-header-size since we are running in a cloud environment not managed by us.

@tony-gutierrez

This comment has been minimized.

Copy link

@tony-gutierrez tony-gutierrez commented May 2, 2019

Yeah, this basically means if you are using the azure sdk, you are locked to older node.

@southpolesteve

This comment has been minimized.

Copy link
Member

@southpolesteve southpolesteve commented May 10, 2019

@christopheranderson can you tag the app service people so they are aware of this on their side?

@tony-gutierrez

This comment has been minimized.

Copy link

@tony-gutierrez tony-gutierrez commented May 10, 2019

Node 10.15.2 is now available on app service. You should be able to somehow execute it with the --max-http-header-size=81000 flag, however I have yet to figure out how.

@christopheranderson

This comment has been minimized.

Copy link
Member

@christopheranderson christopheranderson commented May 10, 2019

I just emailed the App Service team to ask them how to do that and if they can potentially do it automatically.

@tony-gutierrez

This comment has been minimized.

Copy link

@tony-gutierrez tony-gutierrez commented May 10, 2019

Eliminating the quotes works, as described here: tjanczuk/iisnode#315

So by creating iisnode.yml in root, adding nodeProcessCommandLine: D:\Program Files (x86)\nodejs\10.15.2\node.exe --max-http-header-size=81000 you should be able to avoid the header problem and continue to use newer node versions. You can verify by doing a console.log(http.maxHeaderSize);

Also it is possible that a smaller value than 81000 will also work.

You will also want to set the WEBSITE_NODE_DEFAULT_VERSION app setting to 10.15.2 as I believe that controls the default npm version.

It would be nice if the cosmos node sdk could work inside of the 8k header bounds that are now the default in node, but I am sure that is easier said than done.

@rramachand21

This comment has been minimized.

Copy link

@rramachand21 rramachand21 commented May 10, 2019

@leo-buneev

This comment has been minimized.

Copy link

@leo-buneev leo-buneev commented Jul 22, 2019

For the record (I'm constructing REST queries manually, not using this SDK) - if setting "--max-http-header-size" is not possible for you, you can send additional header , x-ms-documentdb-responsecontinuationtokenlimitinkb, that will force cosmosdb server to not send too much information in cont. token. For example:

headers['x-ms-documentdb-responsecontinuationtokenlimitinkb'] = '7'

The downside is - RU usage for getting subsequent pages may increase (because there's less info in cont. token to help cosmosDb retrieve data efficiently)

@southpolesteve

This comment has been minimized.

Copy link
Member

@southpolesteve southpolesteve commented Jul 26, 2019

I've chatted with the backend team and determined we can safely default the header to 1kb. I've done this in #384 and exposed the option to make it larger. This should help significantly with header limit issues. It is still possible for collections with VERY large numbers of partitions to have a large session token, but continuation token is the main culprit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.