CfnCluster fails to start a cluster with COMPUTE nodes in a PRIVATE subnet

[vbosquier]

Hi,

I want to start a cfncluster with the MASTER node in a public Subnet and the COMPUTE nodes in a private Subnet. I don't want my compute nodes to be exposed, and even if you can close all possible ways to connect to the nodes through the Security Group, I'm convinced that having a Public IP on the compute nodes is an unnecessary weakness in terms of security.
My question is:
HOW CAN I START A CLUSTER HAVING THE COMPUTE NODES IN A STRICTLY PRIVATE NETWORK (with no Internet access at all)?

Complementary info:
When trying to start such cluster, the ComputeFleet fails to initialize, and the cluster fails to start with the following error:

Status: cfncluster-Test - CREATE_FAILED                             
Cluster creation failed.  Failed events:
  - AWS::CloudFormation::Stack cfncluster-Test The following resource(s) failed to create: [ComputeFleet]. 
  - AWS::AutoScaling::AutoScalingGroup ComputeFleet Received 2 FAILURE signal(s) out of 2.  Unable to satisfy 100% MinSuccessfulInstancesPercent requirement

In cloud-init.log, I could see that the script named /var/lib/cloud/instance/scripts/part-002 fails to execute properly on the COMPUTE nodes (no error on the MASTER, and no error if the COMPUTE nodes are configured to start in a public Subnet) :

2018-08-28 15:36:38,160 - util.py[DEBUG]: Running command ['/var/lib/cloud/instance/scripts/part-002'] with allowed return codes [0] (shell=False, capture=False)
2018-08-28 15:47:11,727 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-002 [1]
2018-08-28 15:47:11,728 - util.py[DEBUG]: Failed running /var/lib/cloud/instance/scripts/part-002 [1]
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 802, in runparts
subp(prefix + [exe_path], capture=False)
File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 1858, in subp
cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ['/var/lib/cloud/instance/scripts/part-002']
Exit code: 1
Reason: -
Stdout: -
Stderr: -

Please, find below the configuration that I use:

[aws]
aws_region_name = eu-west-1
aws_access_key_id = xxx
aws_secret_access_key = xxx

[global]
update_check = true
sanity_check = true
cluster_template = TorqueCluster

[cluster TorqueCluster]
vpc_settings = VPC-test
key_name = mykey
scheduler = torque
base_os = centos7
initial_queue_size = 2
max_queue_size = 2
maintain_initial_size = true
master_instance_type = m4.xlarge
compute_instance_type = t2.micro

[vpc VPC-test]
vpc_id = vpc-xxx
master_subnet_id = subnet-<public>
compute_subnet_id = subnet-<private>
vpc_security_group_id = sg-xxx

Thanks in advance for your help.

[enrico-usai]

Hi @vbosquier,
you can use the use_public_ips = false parameter in the vpc section:
https://cfncluster.readthedocs.io/en/latest/configuration.html?highlight=private#use-public-ips
It defines whether or not to assign public IP addresses to EC2 Compute Instances.

You can also take at look at the Network Configurations documentation: https://cfncluster.readthedocs.io/en/latest/networking.html

[vbosquier]

Thanks Enrico for the quick reply! ;-)

I just made a new try with the following configuration file below:

(...)
vpc_id = vpc-xxx
master_subnet_id = subnet-<public>
compute_subnet_id = subnet-<private>
use_public_ips = false
vpc_security_group_id = sg-xxx

Unfortunately, the result is the exact same as before.

[root@SERVER .cfncluster]# cfncluster create -c testconf -nr Test
Beginning cluster creation for cluster: Test
Creating stack named: cfncluster-Test
Status: cfncluster-Test - CREATE_FAILED
Cluster creation failed. Failed events:

• AWS::CloudFormation::Stack cfncluster-Test The following resource(s) failed to create: [ComputeFleet].
• AWS::AutoScaling::AutoScalingGroup ComputeFleet Received 0 SUCCESS signal(s) out of 2. Unable to satisfy 100% MinSuccessfulInstancesPercent requirement

[enrico-usai]

Hi @vbosquier,
to have the Compute Nodes in a private subnet, you need to configure a NAT Gateway.

It enables instances in the private subnet to connect to the internet and other AWS services (e.g. s3), but prevent the internet from initiating a connection with those instances.

[vbosquier]

Ciao Enrico.
To enable the COMPUTE instances to communicate with S3, I have created an endpoint (configured in my the route table of my private subnet).
What is the true need for a NAT GAteway? Which other AWS service should the instance talk to? Could I create another type of endpoint instead of a NAT Gateway (which is not free per se, and which generates trafic out of AWS meaning additional charges)?
Vincent.

[vbosquier]

Ciao Enrico!

Complementary info: I have tried to start a cluster in a brand new VPC with 2 subnets (1 public with an Internet Gateway, and 1 private with a NAT Gateway and a S3 Endpoint).

TEST #1 - I started a cluster in my public subnet only using the following conf file:

[cluster Test]
vpc_settings = VPC-Test
(...)

[vpc VPC-Test]
vpc_id = vpc-xxx
master_subnet_id = subnet-<public>
vpc_security_group_id = sg-xxx

Everything worked perfectly fine, the 2 compute nodes went up normally and were added to the conf of the job scheduler (Torque => nodes were down by default).

TEST #2 - I started a cluster to have the MASTER node in my public subnet and the COMPUTE nodes in my private subnet, using the following conf file:

[cluster Test]
vpc_settings = VPC-Test
(...)

[vpc VPC-Test]
vpc_id = vpc-xxx
master_subnet_id = subnet-<public>
compute_subnet_id = subnet-<private>
use_public_ips = false
vpc_security_group_id = sg-xxx

This attempt failed with the same exact issue as before.

[root@SERVER .cfncluster]# cfncluster create -c testconf -nr Test
Beginning cluster creation for cluster: Test
Creating stack named: cfncluster-Test
Status: cfncluster-Test - CREATE_FAILED
Cluster creation failed. Failed events:

• AWS::CloudFormation::Stack cfncluster-TestJ The following resource(s) failed to create: [ComputeFleet].
• AWS::AutoScaling::AutoScalingGroup ComputeFleet Received 0 SUCCESS signal(s) out of 2. Unable to satisfy 100% MinSuccessfulInstancesPercent requirement

And in the COMPUTE nodes cloud-init.log:

2018-09-04 12:30:31,426 - util.py[DEBUG]: Running command ['/var/lib/cloud/instance/scripts/part-002'] with allowed return codes [0] (shell=False, capture=False)
2018-09-04 13:01:08,396 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-002 [1]
2018-09-04 13:01:08,396 - util.py[DEBUG]: Failed running /var/lib/cloud/instance/scripts/part-002 [1]
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 802, in runparts
subp(prefix + [exe_path], capture=False)
File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 1858, in subp
cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ['/var/lib/cloud/instance/scripts/part-002']
Exit code: 1
Reason: -
Stdout: -
Stderr: -

What did I do wrong? Is there another prerequisite to match for the COMPUTE nodes to start normally without public IPv4?

[vbosquier]

Ciao Enrico!

Thanks for your help on this issue, my NAT Gateway was not created correctly which made it unusable. Using a correct NAT Gateway made the trick!

In case this can help any other AWS user, my problem was that my NAT Gateway was not working, because I created it FOR my private subnet IN my private subnet. This is wrong, you must create a NAT Gateway for your private subnet in your PUBLIC subnet.
Indeed at creation time, the NAT Gateway must be associated to the PUBLIC subnet to make it able to access the internet through your Internet Gateway. Then, through a dedicated entry in the route table of your PRIVATE subnet, it relays to the internet requests from instances in your PRIVATE subnet .

This mechanism is clearly indicated here: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html

[enrico-usai]

For completness I would add that VPC Endpoints can't replace the NAT Gateway because Compute Nodes require web access (e.g. to download cookbooks).

Currently it is required to use a NAT Gateway or an internal PROXY as already mentioned here.

[bwbarrett]

Why can't CfnCluster remove the dependency on internet when using pre-built AMIs? They shouldn't need to download everything, because they're pre-built...

[enrico-usai]

Yes, this is the way to go: remove any dependency on internet and just keep some VPC Endpoints, but alas the current requirements are those described above.