Refactor structure (#1)

* Restructure directories on high-level concepts. * Fix cross references from docs.
BigDataRepublic · Sep 9, 2016 · 9fb3005 · 9fb3005
1 parent 4a568b5
commit 9fb3005
Show file tree

Hide file tree

Showing 49 changed files with 11 additions and 15 deletions.
diff --git a/README.md b/README.md
@@ -35,7 +35,7 @@ ansible-playbook example-all.yml -i example-inventory/dev -e stack=dev-{some-nam
 Note: do not forget to tear down all the components created when done experimenting, see wrap-up section of [Automatic EMR Deployment Guide](docs/2-automatic-emr-deployment/README.md).
 
 ## Ansible role structure and conventions
-The Ansible [roles](roles/) directory contains all the roles that can be used. When following the standard best-practices for Ansible, each role contains a single component and/or service that can be deployed on one or more servers. Since we are focussing on infrastructure instead of software that is installed on a single server, we used the following rules of thumb for creating new roles:
+The Ansible [roles](common/) directory contains all the roles that can be used. When following the standard best-practices for Ansible, each role contains a single component and/or service that can be deployed on one or more servers. Since we are focussing on infrastructure instead of software that is installed on a single server, we used the following rules of thumb for creating new roles:
 
 1. In case of non-service specific, infrastructure related components, use a role per infrastructure layer. Example: network, routing, subnets.
 2. In case of service specific components, use a role per service concept. Example: nat (provides internet service to private subnet VMs), vpn (provides access to the cluster to users), datalake (provides the whole configuration of an entire managed datalake; e.g. AWS EMR / Azure HDInsight as implementation).
@@ -79,4 +79,4 @@ This way it is possible to cross-reference between environments , e.g. hostvars[
 They do not plan to fix this. See also: https://github.com/ansible/ansible/issues/9065
 
 ### Output convention
-Each of the roles can generate output to use later when applying other roles in a sequence (rather than independently, which is also possible). The convention here is to define a dictionary variable equal to the role name and then have a subdictionary named "output". For example ``hostvars.aws_region_ireland.network_aws.output.network_id`` will contain the network ID (underlying VPC id in this case) generated by running network_aws in the [example-all.yml](example-all.yml) playbook for the ``aws_region_ireland`` variable group.
+Each of the roles can generate output to use later when applying other roles in a sequence (rather than independently, which is also possible). The convention here is to define a dictionary variable equal to the role name and then have a subdictionary named "output". For example ``hostvars.aws_region_ireland.network_aws.output.network_id`` will contain the network ID (underlying VPC id in this case) generated by running network_aws in the [example-all.yml](data-science-cluster/example-all.yml) playbook for the ``aws_region_ireland`` variable group.
diff --git a/roles/aws_clients/tasks/main.yml → common/network/aws_clients/tasks/main.yml b/roles/aws_clients/tasks/main.yml → common/network/aws_clients/tasks/main.yml
diff --git a/roles/network/tasks/main.yml → common/network/network/tasks/main.yml b/roles/network/tasks/main.yml → common/network/network/tasks/main.yml
diff --git a/roles/datalake_aws/defaults/main.yml → common/network/network_aws/defaults/main.yml b/roles/datalake_aws/defaults/main.yml → common/network/network_aws/defaults/main.yml
diff --git a/roles/network_aws/meta/main.yml → common/network/network_aws/meta/main.yml b/roles/network_aws/meta/main.yml → common/network/network_aws/meta/main.yml
diff --git a/roles/network_aws/tasks/main.yml → common/network/network_aws/tasks/main.yml b/roles/network_aws/tasks/main.yml → common/network/network_aws/tasks/main.yml
diff --git a/roles/network_aws/templates/vpc.json.j2 → ...network/network_aws/templates/vpc.json.j2 b/roles/network_aws/templates/vpc.json.j2 → ...network/network_aws/templates/vpc.json.j2
diff --git a/roles/network_clients/tasks/main.yml → ...on/network/network_clients/tasks/main.yml b/roles/network_clients/tasks/main.yml → ...on/network/network_clients/tasks/main.yml
diff --git a/roles/routing/meta/main.yml → common/network/routing/meta/main.yml b/roles/routing/meta/main.yml → common/network/routing/meta/main.yml
diff --git a/roles/routing/tasks/main.yml → common/network/routing/tasks/main.yml b/roles/routing/tasks/main.yml → common/network/routing/tasks/main.yml
diff --git a/roles/nat_aws/defaults/main.yml → common/network/routing_aws/defaults/main.yml b/roles/nat_aws/defaults/main.yml → common/network/routing_aws/defaults/main.yml
diff --git a/roles/routing_aws/meta/main.yml → common/network/routing_aws/meta/main.yml b/roles/routing_aws/meta/main.yml → common/network/routing_aws/meta/main.yml
diff --git a/roles/routing_aws/tasks/main.yml → common/network/routing_aws/tasks/main.yml b/roles/routing_aws/tasks/main.yml → common/network/routing_aws/tasks/main.yml
diff --git a/roles/routing_aws/templates/routing.json.j2 → ...ork/routing_aws/templates/routing.json.j2 b/roles/routing_aws/templates/routing.json.j2 → ...ork/routing_aws/templates/routing.json.j2
diff --git a/roles/routing_clients/tasks/main.yml → ...on/network/routing_clients/tasks/main.yml b/roles/routing_clients/tasks/main.yml → ...on/network/routing_clients/tasks/main.yml
diff --git a/roles/subnets/meta/main.yml → common/network/subnets/meta/main.yml b/roles/subnets/meta/main.yml → common/network/subnets/meta/main.yml
diff --git a/roles/subnets/tasks/main.yml → common/network/subnets/tasks/main.yml b/roles/subnets/tasks/main.yml → common/network/subnets/tasks/main.yml
diff --git a/roles/network_aws/defaults/main.yml → common/network/subnets_aws/defaults/main.yml b/roles/network_aws/defaults/main.yml → common/network/subnets_aws/defaults/main.yml
diff --git a/roles/subnets_aws/meta/main.yml → common/network/subnets_aws/meta/main.yml b/roles/subnets_aws/meta/main.yml → common/network/subnets_aws/meta/main.yml
diff --git a/roles/subnets_aws/tasks/main.yml → common/network/subnets_aws/tasks/main.yml b/roles/subnets_aws/tasks/main.yml → common/network/subnets_aws/tasks/main.yml
diff --git a/roles/subnets_aws/templates/subnets.json.j2 → ...ork/subnets_aws/templates/subnets.json.j2 b/roles/subnets_aws/templates/subnets.json.j2 → ...ork/subnets_aws/templates/subnets.json.j2
diff --git a/roles/subnets_clients/tasks/main.yml → ...on/network/subnets_clients/tasks/main.yml b/roles/subnets_clients/tasks/main.yml → ...on/network/subnets_clients/tasks/main.yml
diff --git a/roles/firewall_clients/defaults/main.yml → ...atform/firewall_clients/defaults/main.yml b/roles/firewall_clients/defaults/main.yml → ...atform/firewall_clients/defaults/main.yml
diff --git a/roles/firewall_clients/tasks/main.yml → .../platform/firewall_clients/tasks/main.yml b/roles/firewall_clients/tasks/main.yml → .../platform/firewall_clients/tasks/main.yml
diff --git a/data-science-box/ansible.cfg b/data-science-box/ansible.cfg
@@ -0,0 +1,2 @@
+[defaults]
+roles_path = network:platform:../common/network:../common/platform
diff --git a/data-science-cluster/ansible.cfg b/data-science-cluster/ansible.cfg
@@ -0,0 +1,2 @@
+[defaults]
+roles_path = network:platform:../common/network:../common/platform
diff --git a/example-all.yml → data-science-cluster/example-all.yml b/example-all.yml → data-science-cluster/example-all.yml
diff --git a/example-datalake-only.yml → ...science-cluster/example-datalake-only.yml b/example-datalake-only.yml → ...science-cluster/example-datalake-only.yml
diff --git a/example-inventory/README.md → ...ience-cluster/example-inventory/README.md b/example-inventory/README.md → ...ience-cluster/example-inventory/README.md
diff --git a/example-inventory/dev → data-science-cluster/example-inventory/dev b/example-inventory/dev → data-science-cluster/example-inventory/dev
diff --git a/...e-inventory/group_vars/aws_datalake_1.yml → ...e-inventory/group_vars/aws_datalake_1.yml b/...e-inventory/group_vars/aws_datalake_1.yml → ...e-inventory/group_vars/aws_datalake_1.yml
diff --git a/example-inventory/group_vars/aws_nat_1.yml → ...xample-inventory/group_vars/aws_nat_1.yml b/example-inventory/group_vars/aws_nat_1.yml → ...xample-inventory/group_vars/aws_nat_1.yml
diff --git a/...ventory/group_vars/aws_region_ireland.yml → ...ventory/group_vars/aws_region_ireland.yml b/...ventory/group_vars/aws_region_ireland.yml → ...ventory/group_vars/aws_region_ireland.yml
diff --git a/roles/datalake/meta/main.yml → ...e-cluster/platform/datalake/meta/main.yml b/roles/datalake/meta/main.yml → ...e-cluster/platform/datalake/meta/main.yml
diff --git a/roles/datalake/tasks/main.yml → ...-cluster/platform/datalake/tasks/main.yml b/roles/datalake/tasks/main.yml → ...-cluster/platform/datalake/tasks/main.yml
diff --git a/roles/routing_aws/defaults/main.yml → ...r/platform/datalake_aws/defaults/main.yml b/roles/routing_aws/defaults/main.yml → ...r/platform/datalake_aws/defaults/main.yml
diff --git a/roles/datalake_aws/meta/main.yml → ...uster/platform/datalake_aws/meta/main.yml b/roles/datalake_aws/meta/main.yml → ...uster/platform/datalake_aws/meta/main.yml
diff --git a/roles/datalake_aws/tasks/main.yml → ...ster/platform/datalake_aws/tasks/main.yml b/roles/datalake_aws/tasks/main.yml → ...ster/platform/datalake_aws/tasks/main.yml
diff --git a/...s/datalake_aws/templates/datalake.json.j2 → ...m/datalake_aws/templates/datalake.json.j2 b/...s/datalake_aws/templates/datalake.json.j2 → ...m/datalake_aws/templates/datalake.json.j2
diff --git a/roles/nat/meta/main.yml → ...cience-cluster/platform/nat/meta/main.yml b/roles/nat/meta/main.yml → ...cience-cluster/platform/nat/meta/main.yml
diff --git a/roles/nat/tasks/main.yml → ...ience-cluster/platform/nat/tasks/main.yml b/roles/nat/tasks/main.yml → ...ience-cluster/platform/nat/tasks/main.yml
diff --git a/roles/subnets_aws/defaults/main.yml → ...luster/platform/nat_aws/defaults/main.yml b/roles/subnets_aws/defaults/main.yml → ...luster/platform/nat_aws/defaults/main.yml
diff --git a/roles/nat_aws/meta/main.yml → ...ce-cluster/platform/nat_aws/meta/main.yml b/roles/nat_aws/meta/main.yml → ...ce-cluster/platform/nat_aws/meta/main.yml
diff --git a/roles/nat_aws/tasks/main.yml → ...e-cluster/platform/nat_aws/tasks/main.yml b/roles/nat_aws/tasks/main.yml → ...e-cluster/platform/nat_aws/tasks/main.yml
diff --git a/roles/nat_aws/templates/nat.json.j2 → ...er/platform/nat_aws/templates/nat.json.j2 b/roles/nat_aws/templates/nat.json.j2 → ...er/platform/nat_aws/templates/nat.json.j2
diff --git a/data-science-hub/ansible.cfg b/data-science-hub/ansible.cfg
@@ -0,0 +1,2 @@
+[defaults]
+roles_path = network:platform:../common/network:../common/platform
diff --git a/docs/2-automatic-emr-deployment/README.md b/docs/2-automatic-emr-deployment/README.md
@@ -14,9 +14,9 @@ Amazon CloudFormation is used to automate deployment of AWS infrastructure compo
 
 For the first assignment we will give some pointers and suggestions to automate [the datalake you manually provisioned here](../1-manual-emr-deployment/README.md). No step-by-step instructions are provided, but [sample solutions](../../README.md) are included in this repository for you to use as a reference.
 
-1. As a first step, try to provision a VPC using Ansible's cloud formation module. To do this follow [this guide](http://docs.ansible.com/ansible/cloudformation_module.html), a sample solution can be found in the [network_aws module](../../roles/network_aws). Reference to VPC CloudFormation template [here](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpc.html).
-2. Next, try the same with provisioning routing tables (the private and public one for the VPC), a sample solution can be found in the [routing_aws module](../../roles/routing_aws). Reference to Routing table CloudFormation template [here](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-route-table.html).
-3. To finish the networking infrastructure, also automate the provisioning of the subnets of the data lake solution (the private and public subnets). A sample solution can be found in the [subnets_aws module](../../roles/subnets_aws). Reference to subnet CloudFormation components [here](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-subnet.html), and to add them to the routing tables created earlier [here](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-subnet-route-table-assoc.html).
+1. As a first step, try to provision a VPC using Ansible's cloud formation module. To do this follow [this guide](http://docs.ansible.com/ansible/cloudformation_module.html), a sample solution can be found in the [network_aws module](../../common/network/network_aws). Reference to VPC CloudFormation template [here](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpc.html).
+2. Next, try the same with provisioning routing tables (the private and public one for the VPC), a sample solution can be found in the [routing_aws module](../../common/network/routing_aws). Reference to Routing table CloudFormation template [here](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-route-table.html).
+3. To finish the networking infrastructure, also automate the provisioning of the subnets of the data lake solution (the private and public subnets). A sample solution can be found in the [subnets_aws module](../../common/network/subnets_aws). Reference to subnet CloudFormation components [here](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-subnet.html), and to add them to the routing tables created earlier [here](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-subnet-route-table-assoc.html).
 4. Try to come up with other roles for the remaining components (OpenVPN, NAT instance, AWS EMR cluster etc.).
 
 # Designing for reusability

diff --git a/instance-roles/README.md b/instance-roles/README.md
diff --git a/roles/README.md b/roles/README.md