Skip to content

Allow scheduler to preserve worker hostnames#4111

Open
bbockelm wants to merge 2 commits into
dask:mainfrom
bbockelm:preserve_worker_hostname
Open

Allow scheduler to preserve worker hostnames#4111
bbockelm wants to merge 2 commits into
dask:mainfrom
bbockelm:preserve_worker_hostname

Conversation

@bbockelm

Copy link
Copy Markdown

This provides a configuration option allowing the scheduler to preserve worker hostnames (as opposed to the current & default behavior of resolving hostnames immediately to IP address).

Preserving the hostname should convey advantages in two situations:

  1. Dual-network setups where some hosts have IPv4-only or IPv6-only. This allows the host that wants to communicate with the worker to determine the best network to route over (as opposed to the scheduler).
  2. Clusters that route TLS connections based on SNI. If the hostname is kept internally, it will be pased to OpenSSL when making a connection. This allows the OpenSSL client to embed the desired hostname in a SNI header in the TLS handshake. If the worker is behind a TLS-router (not entirely uncommon in a Kubernetes setup), then this SNI is required to successfully route the request.

Closes #4070

This provides a configuration option allowing the scheduler to
preserve worker hostnames (as opposed to the current & default
behavior of resolving hostnames immediately to IP address).

Preserving the hostname should convey advantages in two situations:

1. Dual-network setups where some hosts have IPv4-only or IPv6-only.
   This allows the host that wants to communicate with the worker
   to determine the best network to route over (as opposed to the
   scheduler).
2. Clusters that route TLS connections based on SNI.  If the hostname
   is kept internally, it will be pased to OpenSSL when making a
   connection.  This allows the OpenSSL client to embed the desired
   hostname in a SNI header in the TLS handshake.  If the worker is
   behind a TLS-router (not entirely uncommon in a Kubernetes setup),
   then this SNI is required to successfully route the request.
Comment thread distributed/distributed.yaml Outdated
@mrocklin

Copy link
Copy Markdown
Member

Also cc'ing @jcrist who might have experience here to suggest a default choice.

Base automatically changed from master to main March 8, 2021 19:04
@oshadura

Copy link
Copy Markdown

@bbockelm I opened #4938 to replace your PR since it has conflicts and it was not anymore easy to rebase on master. I took patch directly from our distributed.git fork.

@bbockelm bbockelm requested a review from fjetter as a code owner January 23, 2024 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Preserve hostnames in worker addresses

4 participants