[FLINK-24020][web] Aggregate HTTP requests before custom netty handers are getting the data

[gaborgsomogyi]

What is the purpose of the change

Custom netty handlers can do authentication (amongst other possibilities).
This requires that the handlers are getting the whole HttpRequest content and not just partial data.
At the moment it's not implemented this way which ends-up in flaky behaviour.
Namely sometimes for example History server responds properly (when the request fits into one netty chunk) but sometimes authentication fails (when the request split into multiple netty chunks).

In this PR I've moved/added FlinkHttpObjectAggregator before custom netty handlers.

Brief change log

• Added FlinkHttpObjectAggregator in WebFrontendBootstrap before custom netty handlers.
• Moved FlinkHttpObjectAggregator in RestServerEndpoint before custom netty handlers.

Important note that upload handlers are streaming the uploadable content so they must be before FlinkHttpObjectAggregator.

Verifying this change

• Existing unit tests.
• Endurance test on cluster.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[gaborgsomogyi]

cc @gyfora

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit f954017 (Fri Aug 27 11:29:24 UTC 2021)

Warnings:

• No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[flinkbot]

CI report:

• f35088a Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run travis re-run the last Travis build
• @flinkbot run azure re-run the last Azure build

[gyfora]

@flinkbot run azure

[zentol]

The HistoryServerTest is getting stuck on CI which is likely caused by this PR.

[gaborgsomogyi]

@zentol having a look...

[gaborgsomogyi]

The last issue is unrelated:

E: Unsupported file ./libssl1.0.0_1.0.2n-1ubuntu5.6_amd64.deb given on commandline

[gaborgsomogyi]

@flinkbot run azure

[zentol]

You need to rebase the branch to fix the libssl issue.

[gaborgsomogyi]

@zentol thanks for the guidance, just did it.

[gyfora]

@flinkbot approve all