Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-32488][REST] Introduce separate configuration to control expiry of E… #22906

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

hlteoh37
Copy link
Contributor

…xecutionGraph cache

What is the purpose of the change

Introduce a new configuration to control the cache expiry for ExecutionGraph cache. This should be separate from the web.refresh-interval, as that controls the Flink dashboard refresh rate.

Brief change log

  • Introduced a new configuration rest.cache.execution-graph.expiry to control ExecutionGraph expiry

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs

@hlteoh37
Copy link
Contributor Author

@dmvk Would you be keen on reviewing this PR to improve the configuration of the REST endpoint execution graph cache?

P.S.: I couldn't find an easy way to fallback to the web.refresh-interval if the rest.cache.execution-graph.expiry is not specified. As such, I just kept the same default value.

@flinkbot
Copy link
Collaborator

flinkbot commented Jun 29, 2023

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@hlteoh37 hlteoh37 changed the title [FLINK-32488] Introduce separate configuration to control expiry of E… [FLINK-32488][rest] Introduce separate configuration to control expiry of E… Jun 30, 2023
@hlteoh37 hlteoh37 changed the title [FLINK-32488][rest] Introduce separate configuration to control expiry of E… [FLINK-32488][REST] Introduce separate configuration to control expiry of E… Jun 30, 2023
@@ -20,6 +20,12 @@
<td>Long</td>
<td>The time in ms that the client waits for the leader address, e.g., Dispatcher or WebMonitorEndpoint</td>
</tr>
<tr>
<td><h5>rest.cache.execution-graph.expiry</h5></td>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this config be meaningful to end users? Which REST endpoints does this control? For example, it is not intuitive that this controls the checkpoint list API

@@ -56,7 +55,6 @@ WebMonitorEndpoint<T> createRestEndpoint(
static ExecutionGraphCache createExecutionGraphCache(
RestHandlerConfiguration restConfiguration) {
return new DefaultExecutionGraphCache(
restConfiguration.getTimeout(),
Time.milliseconds(restConfiguration.getRefreshInterval()));
restConfiguration.getTimeout(), restConfiguration.getExecutionGraphCacheExpiry());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is a backwards incompatible change, that might require a FLIP

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants