Azure private endpoint support #1817

m1kola · 2022-07-08T16:45:13Z

https://issues.redhat.com/browse/HIVE-1955

openshift-ci · 2022-07-08T16:45:23Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

2uasimojo · 2022-07-08T17:07:42Z

Cool. So this new field is per spoke cluster, not per hub?

m1kola · 2022-07-11T14:43:16Z

@2uasimojo yes, that's the idea. Each cluster has their own private endpoint so it has to be on per cluster basis.

m1kola · 2022-07-15T15:45:29Z

I tested this in context of ARO and it seems to work as expected: without apiServerEndpointIP Hive fails to reach to the cluster. With apiServerEndpointIP populated Hive can talk to the cluster via private endpoint.

codecov · 2022-07-15T15:52:31Z

Codecov Report

Merging #1817 (70383b9) into master (b87d3cb) will increase coverage by 0.00%.
The diff coverage is 46.42%.

@@           Coverage Diff           @@
##           master    #1817   +/-   ##
=======================================
  Coverage   41.68%   41.69%           
=======================================
  Files         353      353           
  Lines       33128    33156   +28     
=======================================
+ Hits        13810    13823   +13     
- Misses      18155    18170   +15     
  Partials     1163     1163

Impacted Files	Coverage Δ
pkg/remoteclient/mock/remoteclient_generated.go	`54.79% <0.00%> (-14.18%)`	⬇️
...shift/hive/apis/hive/v1/clusterdeployment_types.go	`100.00% <ø> (ø)`
pkg/remoteclient/remoteclient.go	`70.00% <100.00%> (+2.65%)`	⬆️

2uasimojo

make update should rebuild the saas template that's causing verify to fail.

But first...

I've been staring at the REST/HTTP code/docs for a while now, and I can't get my head around how Dial plays in relation to Host. My goal was really just to vet that APIServerEndpointIP is a sensible name for the new field. Because the way it's used makes it feel more like a "proxy". Or does Host not get used at all when Dial is set? Would the original code still work if instead of using APIURLOverride to set Host, we used it to do this DialContext thing? And if so, would that mean that we wouldn't need the new field, and you could just use APIURLOverride for this purpose? If not, what happens if you set both fields?

All of this ignorance makes me leery of introducing this field to the permanent API we have to support until we retire v1.

We have a couple of ways we could make this field "softer" to mitigate those concerns.

We could wrap it in an alpha feature gate, so in order to use it, you have to explicitly opt into it in HiveConfig, and you're agreeing it might go away at any point without a version bump. (Of course we would consult with you before considering such a change.)
We could feed it in via an annotation instead of making it a real API field. That way it's not part of the documented API, and we've got the flexibility to change/remove it if we come up with something better later on.

Thoughts?

2uasimojo · 2022-07-15T18:00:57Z

apis/hive/v1/clusterdeployment_types.go

@@ -668,6 +668,11 @@ type ControlPlaneConfigSpec struct {
 	// active, Hive will use the override URL for further communications with the API server of the remote cluster.
 	// +optional
 	APIURLOverride string `json:"apiURLOverride,omitempty"`
+
+	// APIServerEndpointIP is the optional override of the API server IP address.
+	// Hive will use this IP address for creating tcp connections using client-go.


nit, I wouldn't mention implementation details like client-go in this docstring.

m1kola · 2022-07-18T12:06:18Z

@2uasimojo verify should be fine now. Pleaes take another look.

I've been staring at the REST/HTTP code/docs for a while now, and I can't get my head around how Dial plays in relation to Host.

Let's imagine that we have an API server in kubeconfig set to example.com:6443 and DNS resolves to 93.184.216.34.

Without a custom dialer we will be making a TCP connection to 93.184.216.34 at port 6443 and sending an HTTP request with Host: example.com header.

With apiServerEndpointIP set to say 10.0.0.10 we will be using a custom dialer and making a TCP connection to 10.0.0.10 at port 6443 (original port from the kubeconfig). We still will be sending a HTTP request with Host: example.com header.

Or does Host not get used at all when Dial is set?

Host will not be used in transport/dial (TCP, layer 4), but will still be used on the application layer (in the Host header of HTTP request, layer 7).

Would the original code still work if instead of using APIURLOverride to set Host, we used it to do this DialContext thing? And if so, would that mean that we wouldn't need the new field, and you could just use APIURLOverride for this purpose?

We considered replacing API server hostname in kubeconfig with IP address such as example.com:6443 -> 10.0.0.10:6443, but it won't work because in this case we will be sending HTTP requests with Host: 10.0.0.10 header.

I believe it will be exact the same issue if we set APIURLOverride to 10.0.0.10.

If not, what happens if you set both fields?

Given that we have the following:

Kubeconfig has example.com:6443 (DNS resolves to 93.184.216.34)
APIURLOverride is set to bar.com:6443 (DNS resolves to 104.21.1.6)

I think this is what is going to happen:

Without apiServerEndpointIP field we will be making TCP connections to 104.21.1.6 at port 6443 and sending HTTP Host: bar.com in headers.
With apiServerEndpointIP set to 10.0.0.10 we will be making TCP connections to 10.0.0.10 at port 6443 and sending HTTP Host: bar.com in headers.

All of this ignorance makes me leery of introducing this field to the permanent API we have to support until we retire v1.

We have a couple of ways we could make this field "softer" to mitigate those concerns.
* We could wrap it in an alpha feature gate, so in order to use it, you have to explicitly opt into it in HiveConfig, and you're agreeing it might go away at any point without a version bump. (Of course we would consult with you before considering such a change.)

* We could feed it in via an annotation instead of making it a real API field. That way it's not part of the documented API, and we've got the flexibility to change/remove it if we come up with something better later on.
Thoughts?

I'll leave the decision on how to support this up to you. For ARO it is important to get guarantees that support won't get dropped and that this thing is tested (minimising chances of a regression).

2uasimojo · 2022-07-18T13:17:02Z

Without a custom dialer we will be making a TCP connection to 93.184.216.34 at port 6443 and sending an HTTP request with Host: example.com header.

With apiServerEndpointIP set to say 10.0.0.10 we will be using a custom dialer and making a TCP connection to 10.0.0.10 at port 6443 (original port from the kubeconfig). We still will be sending a HTTP request with Host: example.com header.

And then what happens? Does the other end of the connection use example.com from that header? Does the DNS-resolved address 93.184.216.34 ever get used again?

IIUC the point is to allow the console and cluster workloads to be public, but hide the API behind a private endpoint, right? Hypothetically, would the effect here be the same as if DNS resolved console.example.com to 93.184.216.XX and api.example.com to 10.0.0.10?

Regarding unit testing, I thought I had left this comment last week, sorry:
We should be able to do similar to this just to prove that when the override is set in the ClusterDeployment, it makes its way into the REST Config. Though without some fancy mocking, I guess the only thing we would be able to check is that Config.Dial isn't nil...

m1kola · 2022-07-18T14:27:43Z

And then what happens? Does the other end of the connection use example.com from that header? Does the DNS-resolved address 93.184.216.34 ever get used again?

There could potentially be multiple hosts at the same IP address. It is like setting up example.com and bar.com to be served by the same nginx at 10.0.0.10. If we do curl https://10.0.0.10/ we will end up with a request which as Host: 10.0.0.10 and nginx will have no idea wether it is a request for example.com or for bar.com.

IIUC the point is to allow the console and cluster workloads to be public, but hide the API behind a private endpoint, right? Hypothetically, would the effect here be the same as if DNS resolved console.example.com to 93.184.216.XX and api.example.com to 10.0.0.10?

We also support fully private clusters where both api server and ingress are private. To access the cluster you have to be on the same network as the cluster.

From ARO resource provider perspective we treat all the cluster as if they were private: we connect to all of them (even public ones) via private endpoint.

m1kola · 2022-07-18T16:29:09Z

@2uasimojo I updated Test_builder_RESTConfig to check for nil/not nil and added Test_DialContext to check that we correctly replace the address.

Please take another look.

2uasimojo

I dig the unit tests ✓

After talking with the team:
We're cool making it a real API field, as it seems like it would be generally useful,
BUT
We (okay, I) would like to understand why this wouldn't be solved simply by repointing DNS for the API server URL to the desired IP

Inline suggestions for a couple of renames.

2uasimojo · 2022-07-18T20:21:10Z

apis/hive/v1/clusterdeployment_types.go

+	// APIServerEndpointIP is the optional override of the API server IP address.
+	// Hive will use this IP address for creating TCP connections.
+	// +optional
+	APIServerEndpointIP string `json:"apiServerEndpointIP,omitempty"`


Talked about this with the team.
We would like to propose that the name include the word "Override"; and suggest that "Endpoint" is probably redundant.
Probably also useful to mention that the port from the original API server URL will be used.

Suggested change

// APIServerEndpointIP is the optional override of the API server IP address.

// Hive will use this IP address for creating TCP connections.

// +optional

APIServerEndpointIP string `json:"apiServerEndpointIP,omitempty"`

// APIServerIPOverride is the optional override of the API server IP address.

// Hive will use this IP address (with the port from the original API URL) for creating TCP connections.

// +optional

APIServerIPOverride string `json:"apiServerEndpointIP,omitempty"`

2uasimojo · 2022-07-18T20:33:16Z

pkg/remoteclient/remoteclient_test.go

+
+type dialerMock func(ctx context.Context, network, address string)
+
+func (d dialerMock) DialContext(ctx context.Context, network, address string) (net.Conn, error) {


Okay, so: This DialContext gets called by the DialContext in remoteclient.go:345 as a method on the dialer we pass into the DialContext called in the test driver (L352 below) which is actually calling the DialContext in remoteclient.go:334 -- phew!

I think this would be (slightly) more scrutable if the func in remoteclient.go:334 were named something different, like CreateDialContext.

@2uasimojo Np, I renamed to createDialContext.

CreateDialContext will be returning DialContext which is a wrapper around go's standard library DialContext from Dialer.

I also noticed that in this repo mocks are being generated with GoMock for every interface. I was going to use these mocks, but then realised that I cant: there will be a circular import.

If APIServerEndpointIP is present

m1kola

We (okay, I) would like to understand why this wouldn't be solved simply by repointing DNS for the API server URL to the desired IP

We were considering solving it with the DNS however the issue is that we have customers who:

Use our managed DNS aroapp.io
Who use their own custom domain.

To handle clusters with custom domains we would have to create private DNS zone on the ARO RP side and there is a max limit of 1k private DNS zones per subscription. And we group multiple regions supported by ARO RP within a single subscription. So in every ARO RP sub, we would have an upper limit of 1k clusters with custom DNS. Sooner or later we will hit this limit.

m1kola · 2022-07-19T09:16:39Z

pkg/remoteclient/remoteclient_test.go

+
+type dialerMock func(ctx context.Context, network, address string)
+
+func (d dialerMock) DialContext(ctx context.Context, network, address string) (net.Conn, error) {


@2uasimojo Np, I renamed to createDialContext.

CreateDialContext will be returning DialContext which is a wrapper around go's standard library DialContext from Dialer.

m1kola · 2022-07-19T09:26:21Z

pkg/remoteclient/remoteclient.go

@@ -51,6 +53,10 @@ type Builder interface {
 	UseSecondaryAPIURL() Builder
 }

+type Dialer interface {


Will make Dialer interface and DialContext/CreateDialContext non-exported. These are internal implementation details and don't need to be used externally.

m1kola · 2022-07-19T09:44:31Z

pkg/remoteclient/remoteclient_test.go

+
+type dialerMock func(ctx context.Context, network, address string)
+
+func (d dialerMock) DialContext(ctx context.Context, network, address string) (net.Conn, error) {


I also noticed that in this repo mocks are being generated with GoMock for every interface. I was going to use these mocks, but then realised that I cant: there will be a circular import.

openshift-ci · 2022-07-19T11:41:59Z

@m1kola: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

2uasimojo

Functionally this is good, so I'm happy to land it. Thanks for picking this up @m1kola!
Couple items for a followon PR:

Let's add to the docstring a statement like, "This field can be used when repointing the APIServer's DNS is not viable option."
I'd like to clean up the mocking. I think moving the Dialer interface to a separate file would probably be the first move; then we can either continue to generate that mock if we're going to use it (moving the interface should let us avoid the circular import) or just get rid of it.

/lgtm

openshift-ci · 2022-07-19T13:57:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 2uasimojo, m1kola

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [2uasimojo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

2uasimojo · 2022-07-19T14:12:44Z

/override tide

openshift-ci · 2022-07-19T14:12:49Z

@2uasimojo: Overrode contexts on behalf of 2uasimojo: tide

In response to this:

/override tide

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

2uasimojo · 2022-07-19T14:13:31Z

This passed CI and nothing has merged since. ARO team is waiting on it, so I'm going to merge it manually. Otherwise it'd end up retesting.

It includes support for private endpoint: openshift/hive#1817

2uasimojo · 2022-07-19T15:02:54Z

pkg/remoteclient/remoteclient.go

@@ -289,6 +295,14 @@ func (b *builder) RESTConfig() (*rest.Config, error) {
 		}
 	}

+	if override := b.cd.Spec.ControlPlaneConfig.APIServerIPOverride; override != "" {
+		dialer := &net.Dialer{


nts: net.Dialer implements DialContext -- that's the prod path. We define the interface so we can swap out the impl for UT.

It includes support for private endpoint: openshift/hive#1817

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 8, 2022

openshift-ci bot requested review from abutcher and dlom July 8, 2022 16:48

m1kola mentioned this pull request Jul 15, 2022

Adds and documents a way to manually test a custom hive image Azure/ARO-RP#2252

Closed

m1kola marked this pull request as ready for review July 15, 2022 15:39

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 15, 2022

2uasimojo reviewed Jul 15, 2022

View reviewed changes

m1kola force-pushed the azure-private-endpoint-support branch 2 times, most recently from 6901df7 to 0018c56 Compare July 18, 2022 11:47

m1kola force-pushed the azure-private-endpoint-support branch from 0018c56 to 5e3b2e4 Compare July 18, 2022 16:26

m1kola requested a review from 2uasimojo July 18, 2022 16:27

m1kola mentioned this pull request Jul 18, 2022

Adds an unit test covering DialContext Azure/ARO-RP#2257

Merged

2uasimojo reviewed Jul 18, 2022

View reviewed changes

m1kola added 2 commits July 19, 2022 10:39

Adds a new field into ControlPlaneConfigSpec

87698ed

Adds a custom dialer for rest client-go

70383b9

If APIServerEndpointIP is present

m1kola force-pushed the azure-private-endpoint-support branch from 5e3b2e4 to 70383b9 Compare July 19, 2022 09:48

m1kola commented Jul 19, 2022

View reviewed changes

m1kola requested a review from 2uasimojo July 19, 2022 13:40

2uasimojo reviewed Jul 19, 2022

View reviewed changes

openshift-ci bot assigned 2uasimojo Jul 19, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jul 19, 2022

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 19, 2022

2uasimojo merged commit c63c9b0 into openshift:master Jul 19, 2022

m1kola deleted the azure-private-endpoint-support branch July 19, 2022 14:23

m1kola mentioned this pull request Jul 19, 2022

Azure private endpoint support - follow-up #1822

Merged

m1kola added a commit to m1kola/ARO-RP that referenced this pull request Jul 19, 2022

Updates hive to the latest version

e5cfba4

It includes support for private endpoint: openshift/hive#1817

m1kola added a commit to m1kola/ARO-RP that referenced this pull request Jul 19, 2022

Updates hive to the latest version

841b741

It includes support for private endpoint: openshift/hive#1817

m1kola mentioned this pull request Jul 19, 2022

Updates Hive to the latest version Azure/ARO-RP#2262

Merged

2uasimojo reviewed Jul 19, 2022

View reviewed changes

m1kola added a commit to Azure/ARO-RP that referenced this pull request Jul 19, 2022

Updates hive to the latest version

961f8ed

It includes support for private endpoint: openshift/hive#1817

This was referenced Jul 19, 2022

Vendoring: update Hive to the latest version Azure/ARO-RP#2263

Merged

Makes ClusterDeployment private endpoint aware Azure/ARO-RP#2268

Merged

m1kola mentioned this pull request Jul 29, 2022

[M5] Initial storage of ClusterDocument in Hive Azure/ARO-RP#2215

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure private endpoint support #1817

Azure private endpoint support #1817

m1kola commented Jul 8, 2022 •

edited

Loading

openshift-ci bot commented Jul 8, 2022

2uasimojo commented Jul 8, 2022

m1kola commented Jul 11, 2022

m1kola commented Jul 15, 2022

codecov bot commented Jul 15, 2022 •

edited

Loading

2uasimojo left a comment

2uasimojo Jul 15, 2022

m1kola commented Jul 18, 2022

2uasimojo commented Jul 18, 2022

m1kola commented Jul 18, 2022 •

edited

Loading

m1kola commented Jul 18, 2022

2uasimojo left a comment

2uasimojo Jul 18, 2022

2uasimojo Jul 18, 2022

m1kola Jul 19, 2022 •

edited

Loading

m1kola Jul 19, 2022

m1kola left a comment

m1kola Jul 19, 2022 •

edited

Loading

m1kola Jul 19, 2022

m1kola Jul 19, 2022

openshift-ci bot commented Jul 19, 2022

2uasimojo left a comment

openshift-ci bot commented Jul 19, 2022

2uasimojo commented Jul 19, 2022

openshift-ci bot commented Jul 19, 2022

2uasimojo commented Jul 19, 2022

2uasimojo Jul 19, 2022


		type dialerMock func(ctx context.Context, network, address string)

		func (d dialerMock) DialContext(ctx context.Context, network, address string) (net.Conn, error) {

Azure private endpoint support #1817

Azure private endpoint support #1817

Conversation

m1kola commented Jul 8, 2022 • edited Loading

openshift-ci bot commented Jul 8, 2022

2uasimojo commented Jul 8, 2022

m1kola commented Jul 11, 2022

m1kola commented Jul 15, 2022

codecov bot commented Jul 15, 2022 • edited Loading

Codecov Report

2uasimojo left a comment

Choose a reason for hiding this comment

2uasimojo Jul 15, 2022

Choose a reason for hiding this comment

m1kola commented Jul 18, 2022

2uasimojo commented Jul 18, 2022

m1kola commented Jul 18, 2022 • edited Loading

m1kola commented Jul 18, 2022

2uasimojo left a comment

Choose a reason for hiding this comment

2uasimojo Jul 18, 2022

Choose a reason for hiding this comment

2uasimojo Jul 18, 2022

Choose a reason for hiding this comment

m1kola Jul 19, 2022 • edited Loading

Choose a reason for hiding this comment

m1kola Jul 19, 2022

Choose a reason for hiding this comment

m1kola left a comment

Choose a reason for hiding this comment

m1kola Jul 19, 2022 • edited Loading

Choose a reason for hiding this comment

m1kola Jul 19, 2022

Choose a reason for hiding this comment

m1kola Jul 19, 2022

Choose a reason for hiding this comment

openshift-ci bot commented Jul 19, 2022

2uasimojo left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Jul 19, 2022

2uasimojo commented Jul 19, 2022

openshift-ci bot commented Jul 19, 2022

2uasimojo commented Jul 19, 2022

2uasimojo Jul 19, 2022

Choose a reason for hiding this comment

m1kola commented Jul 8, 2022 •

edited

Loading

codecov bot commented Jul 15, 2022 •

edited

Loading

m1kola commented Jul 18, 2022 •

edited

Loading

m1kola Jul 19, 2022 •

edited

Loading

m1kola Jul 19, 2022 •

edited

Loading