Skip to content

Conversation

@caodizhou
Copy link

@caodizhou caodizhou commented Nov 14, 2023

What is the purpose of the change

  • Support configured akka remote dispatcher thread pool size.
  • Move the serialization of ShuffleDescriptorGroup out of the RPC main thread

Brief change log

  • Add config akka.remote-fork-join-executor.parallelism-factor,akka.remote-fork-join-executor.parallelism-max and akka.remote-fork-join-executor.parallelism-min
  • Move the serialization logic of ShuffleDescriptorGroup to akka's default-remote-dispatcher thread instead of the default-dispatcher thread.

Verifying this change

  • Add getConfigDefaultsToRemoteForkJoinExecutor in PekkoUtilsTest.
  • The tests related to ShuffleDescriptorGroup have already been covered by existing tests, such as TaskDeploymentDescriptorFactoryTest.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@flinkbot
Copy link
Collaborator

flinkbot commented Nov 14, 2023

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@caodizhou caodizhou force-pushed the FLINK-33532 branch 2 times, most recently from c4d0378 to 1948e49 Compare November 15, 2023 06:53
.build());

public static final ConfigOption<Double> REMOTE_FORK_JOIN_EXECUTOR_PARALLELISM_FACTOR =
ConfigOptions.key("akka.remote-fork-join-executor.parallelism-factor")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pekko

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as below

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revised

/**
* The raw value that is not offloaded to the {@link org.apache.flink.runtime.blob.BlobServer}.
*
* @param <T> type of the serialized value
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

type of the raw value

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revised

@KarmaGYZ KarmaGYZ merged commit d18a4bf into apache:master Nov 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants