Autonomous Cluster for the Edge (ACE)

1. Overview

2. Architecture

3. Prerequisites

4. Quick Installation Guide

5. Production Installation Guide

Overview

Autonomous Cluster for the Edge is a ZeroConfig Technology enable edge devices to scale like Lego blocks. It is the set of open source services used to enable autonomous horizontally scaling of edge devices services to support Workload Orchestration. As application workloads demand more resources, add Intel x86 hardware as Lego building blocks to the infrastructure. Simply plug a machine into the network and walk away; the new system will auto configure itself, join the cluster and become an available resource for the application workloads. This ZeroConfig architecture can provide high availability, distributed workloads, workload affinity to specific hardware, upgrade hardware with 24/7 uptime, and more. The ZeroConfig architecture can simultaneously run Linux, Windows and Android applications across a heterogeneous infrastructure on Intel x86 Architecture from Celeron to Xeon in the same compute environment.

Key Points

• Scaling vertically can be very expensive, limiting and still be a single point of failure; instead scale horizontally and start with commodity hardware.
• To scale horizontally on heterogeneous x86 hardware from Intel Celeron to Xeon.
• To provide the ability to use existing hardware and provide a migration path to new hardware with near 100% uptime.
• To provide high availability with heterogeneous x86 hardware.
• To provide software abstraction from multiple nodes.

Architecture

ACE version components:

• Hashicorp Serf version 0.8.4
  • Device discovery and event driven orchestration
• RedHat GlusterFS version 5.10
  • Persisted file storage
• GlusterFS Plugin version 1.0
  • Docker volume plugin
• Docker Swarm version 19.03.12
• Dynamic Hardware Orchestrator
  • Dynamically discovers the state of the hardware or VM resources
• Alpine Console
  • Management Console

Prerequisites

• x86 Hardware or x86 Virtual Machine
  • At Least 5 GB of Disk Space
  • 4 GB of RAM
• Linux Distro that supports Docker
• docker 18.06.x or greater
• docker-compose v1.23.2 or greater
• bash v4.3.48 or greater

NOTE: SSH Service is not required as ACE will create one at Port 222. This value "PORT=(some alternate port number)", can be changed in /opt/ACE/compose/docker-compose.yml under the 'console:' service environment virables. The option is to change the default Port number on the host.

• For Quick Installation Guide
• For Production Installation Guide
• For detail build steps. See this

Quick Installation Guide

1. ALL the following commands must run as ROOT. Type sudo su - and the your password before proceeding.

sudo su -

2. Install Docker Compose

mkdir -p /usr/local/bin
wget -O /usr/local/bin/docker-compose "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)"
chmod a+x /usr/local/bin/docker-compose

3. Clone this project to /opt/ACE

cd /opt
git clone https://github.com/intel/RetailWorkloadOrchestrator.git ACE

4. Build the container images

cd /opt/ACE
./build.sh

5. Install Demo Keys. NOTE: These keys are published and should be used for demonstration purposes only. Please refer to the Production Installation Guide section to genereate unique keys.

./install.sh demo

6. Start ACE

systemctl start ACE

7. Confirm all services are running. (It will take 2 minutes for all services to start)

~# docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS               NAMES
d36940c44e3e        edge/console-alpine:1.0   "dumb-init /bin/init"    21 seconds ago      Up 20 seconds                           ACE_console_1
253f6246488a        edge/serf:0.8.4           "/opt/ACE/serf/docke…"   22 seconds ago      Up 21 seconds                           ACE_serf-agent_1
33034bedfce1        edge/glusterfs-rest:7     "dumb-init glusterd …"   22 seconds ago      Up 21 seconds                           ACE_gluster-server_1
a9c3989f3dda        edge/glusterfs-rest:7     "dumb-init /bin/init…"   25 seconds ago      Up 24 seconds                           ACE_gluster-rest_1
99ca764a1a5e        edge/app-docker:1.0       "app-docker.sh /usr/…"   26 seconds ago      Up 26 seconds                           ACE_app-docker_1
3a7fddeed1e7        edge/rngd:1.0             "haveged -F"             26 seconds ago      Up 26 seconds                           ACE_rngd

8. Enter the ACE Console

docker exec -it ACE_console_1 bash

9. Install Portainer.io Management. NOTE: The following command MUST run inside the ACE Console from step 8.

curl -L https://downloads.portainer.io/portainer-agent-stack.yml -o portainer-agent-stack.yml
docker stack deploy --compose-file=portainer-agent-stack.yml portainer

10. Get the IP address of this node and connect to Portainer at <IP Address>:9000. Create a username and password. Once logged in, click the "Primary" endpoint below and go to the "Swarm" link on the left navigation panel.

11. On another node repeat steps 1 through 7. From Poratiner UI refresh the Swarm page to watch an aditional node be added.

Production Installation Guide

1. Clone this project to /opt/ACE

cd /opt
git clone https://github.com/intel/RetailWorkloadOrchestrator.git ACE

2. Build the container images

cd /opt/ACE
./build.sh

3. Push all ACE images to your Docker Registry.

edge/glusterfs-plugin
edge/dxo
edge/app-docker
edge/dho
edge/serf
edge/glusterfs-rest
edge/console-alpine

4. Update compose/docker-compose.yml images to reflect your registry path.

5. Genereate ACE Certifcates and Keys. NOTE: All Certifcates and Keys must be copied to all other nodes you plan to join this node. For more details, please refer to https://github.com/intel/RetailWorkloadOrchestrator/blob/master/docs/02_Security.md

./generate_keys.sh

5. Install

./install.sh

4. Start ACE

systemctl start ACE

Serf Encryption Keys

Autonomous Cluster for the Edge uses hashicorp serf for service discovery. Encryption with AES-256 is enabled for serf v0.8.4.

The AES-256 key is to be kept in a keyring file in json format.

keyring.json

[
  "HvY8ubRZMgafUOWvrOadwOckVa1wN3QWAo46FVKbVN8="
]

This is a symmetric key which needs to be same in all the nodes of a cluster, only then serf will be able to discover them.

For Details. See this.

Security in Autonomous Cluster for the Edge

Security in Autonomous Cluster for the Edge is taken care of by using PKI architecture and root privileges. Various components of Autonomous Cluster for the Edge talk to each other either over https or they use docker-volumes/filesystem location protected by root privilege. Glusterfs is also secured using PKI.

For Details. See this.

Environment Variables

There are Environment variables defined in /opt/ACE/compose/docker-compose.yml.

1. SWARM_RESTORE_TIME_IN_SECONDS : Number of seconds ACE handlers need to wait for a swarm restore to happen after a member leaves the cluster or reboot itself.
2. MEMBER_REBOOT_TIME_IN_SECONDS : Number of seconds ACE handlers need to wait for a member to complete its reboot.
3. LOG_LEVEL : Specifies different levels at which logs needs to be printed. LOG_LEVEL=1 prints info level log statements. LOG_LEVEL=2 prints debug level log statements.

Known Limitations

• Rarely gluster cluster formation may fail and node may not be able to join the cluster, in that case, go ahead run /opt/ACE/bin/reset and reboot. Resetting ACE will wipe out all its state.
• Dynamic Hardware Orchestrator (DHO) Currently only detects system state at startup. Does not continuosly check state.