Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lack of Internet Gateway is Fatal Error on Existing Private VPC #780

Closed
druidsbane opened this issue Nov 2, 2016 · 33 comments
Closed

Lack of Internet Gateway is Fatal Error on Existing Private VPC #780

druidsbane opened this issue Nov 2, 2016 · 33 comments

Comments

@druidsbane
Copy link

@druidsbane druidsbane commented Nov 2, 2016

Hi,

It would be useful to not require an internet gateway when setting up on a pre-existing VPC, especially if it is intended to be completely internal and not have public IP's. Currently it fails with:

internetgateway.go:197] ID must be set, if InternetGateway is shared:
 *awstasks.InternetGateway 
{
  "Name":"cluster.mydomain.com",
  "ID":null,
  "VPC":
  {
    "Name":"cluster.mydomain.com",
    "ID":"vpc-xxxxxxxx",
    "CIDR":"10.2.0.0/16",
    "EnableDNSHostnames":true,
    "EnableDNSSupport":true,
    "Shared":true
   },
  "Shared":true
}

Is there a flag that we can use today to not require it?

Also might be useful option to ensure works when implementing: #428 #694

@kris-nova

This comment has been minimized.

Copy link
Member

@kris-nova kris-nova commented Nov 3, 2016

@druidsbane - I will look into how much it will take to get this into #694 - if it's a minor change I can shoehorn it in for you.

Otherwise this might have to be a feature added after the initial private networking PR goes through.

Either way I think it's a valid point, and worthwhile to get into the code base.

@chrislovecnm

This comment has been minimized.

Copy link
Member

@chrislovecnm chrislovecnm commented Dec 19, 2016

We have PR inbound thay may help with this ... I am looking through issues to test against #1183 - no promises but we need to test.

@kris-nova thoughts?

@chrislovecnm chrislovecnm mentioned this issue Dec 19, 2016
10 of 27 tasks complete
@chrislovecnm

This comment has been minimized.

Copy link
Member

@chrislovecnm chrislovecnm commented Dec 19, 2016

See #1206 ~ The PR is in. Who can test?

@justinsb justinsb modified the milestone: 1.5.0 Dec 28, 2016
@justinsb

This comment has been minimized.

Copy link
Member

@justinsb justinsb commented Jan 29, 2017

We do have --topology private now, but we do (in general) still require internet access to download e.g. docker images. It would be great to have an option to have that entirely driven from an S3 bucket, and to let you mirror that S3 bucket if you want to. I think that would be most of what is needed to drop the internet gateway.

@druidsbane can you comment on whether --topology private fits your use case, and whether you have found a way around requiring internet access (i.e. I think things would break if you didn't have an internet gateway)

@justinsb justinsb modified the milestones: 1.5.1, 1.5.0 Jan 29, 2017
@chrislovecnm chrislovecnm modified the milestones: backlog, 1.5.2 Apr 30, 2017
@fraenkel

This comment has been minimized.

Copy link

@fraenkel fraenkel commented May 26, 2017

If an internet gateway is required for downloading, can we support egress only internet gateways?

This was referenced Jul 21, 2017
@justinsb

This comment has been minimized.

Copy link
Member

@justinsb justinsb commented Jul 24, 2017

@fraenkel egress-only internet gateways were new to me; it sounds like they are IPv6 only (reading http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/egress-only-internet-gateway.html). But we do support NAT gateways (use --topology private) and it sounds like this is what you are asking for?

@fraenkel

This comment has been minimized.

Copy link

@fraenkel fraenkel commented Jul 24, 2017

@justinsb Not exactly but there doesn't seem to be a way to install without an inbound internet connection which is my requirement.

@chrislovecnm

This comment has been minimized.

Copy link
Member

@chrislovecnm chrislovecnm commented Jul 24, 2017

@fraenkel have a PR in with a fix, client needs it as well

@justinsb

This comment has been minimized.

Copy link
Member

@justinsb justinsb commented Jul 24, 2017

@fraenkel cool, yes @chrislovecnm is doing some work here. Can you expand on your requirements though? Where should we source docker images from? The three primary options I know of so far:

  • An S3 bucket which we then docker load
  • A private registry running insider your VPC, and we rewrite the image names so that they point to that private registry
  • A private registry in conjunction with docker registry mirroring

Do you have a preferred approach @fraenkel ?

@fraenkel

This comment has been minimized.

Copy link

@fraenkel fraenkel commented Jul 24, 2017

@justinsb Yes. I see bootstrapping the system needs s3 since I would not have a private registry or proxy running yet. Once I have a minimal system running, I would want to run either a registry or proxy. I have use cases for both, one for a dev environment (registry) and the other for all other environments which pull from a CI registry.

@chrislovecnm

This comment has been minimized.

Copy link
Member

@chrislovecnm chrislovecnm commented Jul 24, 2017

@fraenkel I know users have a use case where:

  1. provide kops with vpc id
  2. provide kops with subnet id(s)
  3. kops does not create or modify any networking components: vpc, subnet, route tables, igw, etc

@justinsb private container registry, file repository, and HTTP proxies are often used in conjunction with these requirements, but I think @fraenkel is just concerned about the network. I think this requirement could be fulfilled by allowing our new network phase not touch any networking components.

@fraenkel if kops created only IAM and EC2 components, would that fulfill your needs? Also, I assume that we need to not update or delete any networking components as well.

@fraenkel

This comment has been minimized.

Copy link

@fraenkel fraenkel commented Jul 24, 2017

@chrislovecnm Not really. There are many pieces that have hard coded where images and other bits are downloaded from. Providing a locked down VPC would fail immediately because there is no way to easily configure where the bits and pieces required to stand up the various components would come from.

@chrislovecnm

This comment has been minimized.

Copy link
Member

@chrislovecnm chrislovecnm commented Jul 24, 2017

@fraenkel so that is a different set of PRs that are being work on as well

  1. Custom container registry location
  2. Custom file repository location
  3. Capability to upload to those locations

This includes basically everything. All bits/containers for kops and k8s.

Also working on using an http proxy.

But, back to the topic at hand with networks. Do you want kops to create or update any network components.

@fraenkel

This comment has been minimized.

Copy link

@fraenkel fraenkel commented Jul 24, 2017

What is there is fine with me. I always have a pre-created VPC and subnet(s), just no IGW by default.

@adrianmkng

This comment has been minimized.

Copy link

@adrianmkng adrianmkng commented Aug 4, 2017

I can give my 2 cents on the issue, even though I don't have much of an idea how kops works, but I can explain my use case.
I'm using direct connect to connect to instances in my VPC and routing all other traffic from the VPC to the virtual private gateway so that it's only accessible from within the corporate network. Hence internet access from the EC2 instances will be routed through the corporate network and behind the corporate proxy at all times. Like the above examples I'm creating the VPCs and subnets myself and just using kops to setup the Kube cluster.

@chrislovecnm

This comment has been minimized.

Copy link
Member

@chrislovecnm chrislovecnm commented Oct 14, 2017

My phase work will fix this

/assign

@wadeholler

This comment has been minimized.

Copy link

@wadeholler wadeholler commented Oct 24, 2017

+1 very interested in this

@cb-crucks

This comment has been minimized.

Copy link

@cb-crucks cb-crucks commented Oct 30, 2017

+1 Also very interested.

@tewing

This comment has been minimized.

Copy link

@tewing tewing commented Nov 12, 2017

+1 very interested.

1 similar comment
@dreampuf

This comment has been minimized.

Copy link

@dreampuf dreampuf commented Nov 16, 2017

+1 very interested.

@towens

This comment has been minimized.

Copy link

@towens towens commented Jan 5, 2018

+1 Also very interested.

@gozer2222

This comment has been minimized.

Copy link

@gozer2222 gozer2222 commented Jan 26, 2018

+1 Also very very interested.

@zuyong

This comment has been minimized.

Copy link

@zuyong zuyong commented Feb 2, 2018

+1 Also very very very interested.

@chrislovecnm

This comment has been minimized.

Copy link
Member

@chrislovecnm chrislovecnm commented Feb 13, 2018

If anyone has some time to test #3031, it would be much appreciated. I have tested the PR, but I do not have a VPN or other solution setup to test with.

@dreampuf

This comment has been minimized.

Copy link

@dreampuf dreampuf commented Feb 13, 2018

@chrislovecnm I tested it with an InternetGateWay but didn't work with an internal VPN. We have a simular environment.

@chrislovecnm

This comment has been minimized.

Copy link
Member

@chrislovecnm chrislovecnm commented Feb 13, 2018

@dreampuf can I get more details please?

@dreampuf

This comment has been minimized.

Copy link

@dreampuf dreampuf commented Feb 13, 2018

@chrislovecnm Sorry for the misleading. I thought the wrong issue. It's worked both having an IGW to let the kops finish the setting up progress or utilizing this patch to skip forcing create new IGW.

We have a VPN and route the out bandwidth to that VGW. Kops with this patch can work with that. I can send you the history echo if you need it.

@chrislovecnm

This comment has been minimized.

Copy link
Member

@chrislovecnm chrislovecnm commented Feb 13, 2018

@dreampuf yes please set the API value and test for me. Let me know if the IGW is created and if it isn't if we try to delete one.

  networkRequireGateway: false

Needs to be set either via an edit or use a YAML manifest. You will need to build kops off of my branch or apply the patch to master. https://patch-diff.githubusercontent.com/raw/kubernetes/kops/pull/3031.patch

The help is very much appreciated!!

@dreampuf

This comment has been minimized.

Copy link

@dreampuf dreampuf commented Feb 13, 2018

@chrislovecnm

I just created a cluster in a non-IGW VPC.

$ git clone https://github.com/kubernetes/kops.git
Cloning into 'kops'...
remote: Counting objects: 167284, done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 167284 (delta 6), reused 9 (delta 3), pack-reused 167247
Receiving objects: 100% (167284/167284), 168.43 MiB | 33.77 MiB/s, done.
Resolving deltas: 100% (97678/97678), done.
$ cd kops
$ patch -p1 < <(curl -s https://patch-diff.githubusercontent.com/raw/kubernetes/kops/pull/3031.patch)
patching file pkg/apis/kops/cluster.go
patching file pkg/apis/kops/v1alpha1/cluster.go
patching file pkg/apis/kops/v1alpha1/zz_generated.conversion.go
patching file pkg/apis/kops/v1alpha1/zz_generated.deepcopy.go
patching file pkg/apis/kops/v1alpha2/cluster.go
patching file pkg/apis/kops/v1alpha2/zz_generated.conversion.go
patching file pkg/apis/kops/v1alpha2/zz_generated.deepcopy.go
patching file pkg/apis/kops/zz_generated.deepcopy.go
patching file pkg/model/network.go
$ docker run --rm -it -v "$PWD":/go/src/k8s.io/kops -w /go/src/k8s.io/kops golang make kops
mkdir -p /go/src/k8s.io/kops/.build/local
go build  -ldflags "" -o /go/src/k8s.io/kops/.build/local/go-bindata k8s.io/kops/vendor/github.com/jteeuwen/go-bindata/go-bindata
cd /go/src/k8s.io/kops; /go/src/k8s.io/kops/.build/local/go-bindata -o upup/models/bindata.go -pkg models -ignore="\\.DS_Store" -ignore="bindata\\.go" -ignore="vfs\\.go" -prefix upup/models/ upup/models/...
go build  -ldflags "-X k8s.io/kops.Version=1.8.1 -X k8s.io/kops.GitVersion=ff5c625a8 " -o /go/src/k8s.io/kops/.build/local/kops k8s.io/kops/cmd/kops/
$ .build/local/kops version
Version 1.8.1 (git-ff5c625a8)

$ kops version
Version 1.8.1 (git-ff5c625a8)
$ aws s3 cp s3://${BUCKETNAME}/${CLUSTERNAME}/config - | grep networkRequire
  networkRequireGateway: false
$ kops update cluster --yes --name ${CLUSTERNAME}
W0213 22:14:31.816094   10090 network.go:116] kops is skipping the creation of an Internet Gateway, as networkRequireGateway is set to false.
I0213 22:14:31.979327   10090 executor.go:91] Tasks: 0 done / 81 total; 33 can run
...
kops has set your kubectl context to kube-canal.adsdev.aws.ec2.internal

Cluster is starting.  It should be ready in a few minutes.
@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented May 14, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Jun 14, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Jul 14, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@michaelajr

This comment has been minimized.

Copy link

@michaelajr michaelajr commented Apr 9, 2019

@chrislovecnm Did this ever make it into the main branch?
@austinmoore-

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

You can’t perform that action at this time.