[FLINK-12781] [Runtime/REST] include the whole stack trace in response payload

[stevenzwu]

No description provided.

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit f01bf7b (Tue Aug 06 16:02:23 UTC 2019)

Warnings:

• No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[stevenzwu]

test failed due to this change. increased max content length in test code

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
content-length: 6469 Try to raise [rest.client.max-content-length]```

[stevenzwu]

latest travis build seems to fail with build timeout

[StephanEwen]

Having the stack trace is indeed helpful.

The response can however still fail due to messages with a stack trace longer than the content length.
I think we need to cut the stack trace to be below the max contents length. That may loose some parts of the stack trace in the end, but otherwise the whole response can fail.

[stevenzwu]

@StephanEwen that is a great point. I should have looked into what the test failure really means, instead of just increase the max content length in the test class. there are already a few test methods verify the max content length.

I can get the maxContentLength config from FlinkHttpObjectAggregator ChannelHandler. However, I found it tricky to determine the truncation size, because there are some other byte overhead in the payload. e.g. for the RestServerEndpointITCase, the magic number seems to be 81. I will update PR with a 1 KB overhead buffer. Let me know if you have any other suggestion.

FlinkHttpObjectAggregator flinkHttpObjectAggregator = ctx.pipeline().get(FlinkHttpObjectAggregator.class);
String stackTrace = Ascii.truncate(ExceptionUtils.stringifyException(rhe), flinkHttpObjectAggregator.maxContentLength() - 81, "...");

Also note that default maxContent length is 100 MB
https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#rest-server-max-content-length

[flinkbot]

CI report for commit f01bf7b: ERROR Build

[flinkbot]

CI report:

• f01bf7b : FAILURE Build

[StephanEwen]

This looks good. Will rebase and test this...