-
Notifications
You must be signed in to change notification settings - Fork 590
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs(agw): docker AGW install instructions #12259
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,114 +1,170 @@ | ||
| # Containerized AGW | ||
|
|
||
| ## Build and push images | ||
|
|
||
| ### Build | ||
| ``` | ||
| git clone https://github.com/magma/magma | ||
| cd lte/gateway/docker | ||
| docker-compose build | ||
| ``` | ||
|
|
||
| ### Push | ||
| Login to the container registry and push | ||
| ``` | ||
| docker login example.repo/magma/ | ||
| cd lte/gateway/docker | ||
| bash publish.sh example.repo/magma/ | ||
| ``` | ||
|
|
||
| ## Deploy Containerized AGW | ||
|
|
||
| ### Using cloudstrapper to create an AGW instance on AWS CF | ||
| Deploy an instance using the [agw-provision playbook](https://github.com/magma/magma/blob/master/experimental/cloudstrapper/playbooks/agw-provision.yaml) | ||
|
|
||
| ``` | ||
| cd experimental/cloudstrapper/playbooks/ | ||
| ansible-playbook agw-provision.yaml --tags createGw -e '@~/cluster.yaml' | ||
| ``` | ||
| Example cluster.yaml | ||
| ``` | ||
| # Containerization Deploy | ||
|
|
||
| * Use the Base Cloudstrapper image with an expanded disk size (64G at least) to create Build Container | ||
| * If you want to use a stand-alone Ubuntu image, [Setup](https://docs.docker.com/engine/install/ubuntu/) Docker to run [locally](https://forums.docker.com/t/couldnt-connect-to-docker-daemon-at-http-docker-localhost-is-it-running/87257). Remember to re-login after local user got added to docker so that the new shell has all the right user provisions | ||
| * fb-magma-aws has ami-03bc7ef7f3b70f77b which is a base Cloudstrapper image with extended storage | ||
| * Run Cloudstrapper AGW deploy with input file as below (SDTI network already exists) | ||
| * devsrv-tokyo:~/magma-master #cat sdti-build1.yaml | ||
|
|
||
| ``` | ||
| #Setting AGW AMI and Cloudstrapper AMI to expanded Cloudstrapper image Ubuntu to allow deploy | ||
| dirLocalInventory: ~/magma-master | ||
| awsAgwAmi: ami-03bc7ef7f3b70f77b | ||
| awsCloudstrapperAmi: ami-03bc7ef7f3b70f77b | ||
| awsAgwRegion: ap-northeast-1 | ||
| keyHost: keyMagmaHostCharlie | ||
| idSite: SDTI | ||
| idGw: sdti-build1 | ||
| awsInstanceType: t3.large | ||
|
|
||
| devsrv-tokyo:~/magma-master/magma/experimental/cloudstrapper/playbooks #ansible-playbook —tags createGw agw-provision.yaml -e '@~/magma-master/sdti-build1.yaml' | ||
| ``` | ||
|
|
||
| * Configure ssh config file to allow SSH access to CTR1 | ||
| * System information : Runs 5.4 kernel, has IP addresses configured on first and second interfaces | ||
|
|
||
| ``` | ||
| sdti-build1:~/magma/lte/gateway/docker #uname -r | ||
| 5.4.0-1045-aws | ||
| sdti-build1:~/magma/lte/gateway/docker #ip addr show | ||
| 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 | ||
| link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 | ||
| inet 127.0.0.1/8 scope host lo | ||
| valid_lft forever preferred_lft forever | ||
| inet6 ::1/128 scope host | ||
| valid_lft forever preferred_lft forever | ||
| 2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000 | ||
| link/ether 06:92:f5:05:4a:07 brd ff:ff:ff:ff:ff:ff | ||
| inet 10.23.4.220/24 brd 10.23.4.255 scope global dynamic ens5 | ||
| valid_lft 3358sec preferred_lft 3358sec | ||
| inet6 fe80::492:f5ff:fe05:4a07/64 scope link | ||
| valid_lft forever preferred_lft forever | ||
| 3: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000 | ||
| link/ether 06:a4:58:04:db:d3 brd ff:ff:ff:ff:ff:ff | ||
| inet 10.23.2.65/24 brd 10.23.2.255 scope global dynamic ens6 | ||
| valid_lft 3356sec preferred_lft 3356sec | ||
| inet6 fe80::4a4:58ff:fe04:dbd3/64 scope link | ||
| valid_lft forever preferred_lft forever | ||
| 4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default | ||
| link/ether 02:42:5c:43:5a:64 brd ff:ff:ff:ff:ff:ff | ||
| inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 | ||
| valid_lft forever preferred_lft forever | ||
| inet6 fe80::42:5cff:fe43:5a64/64 scope link | ||
| valid_lft forever preferred_lft forever | ||
| ``` | ||
|
|
||
|
|
||
| * Temporary steps | ||
| * sudo apt-get update | ||
| * Install ifupdown | ||
| * Run docker-compose build (~45 minutes) | ||
|
|
||
| ``` | ||
| Step 56/56 : RUN chmod -R +x /usr/local/bin/generate* /usr/local/bin/set_irq_affinity /usr/local/bin/checkin_cli.py && dpkg -i /var/tmp/python3-aioeventlet* && pip install jsonpointer>$JSONPOINTER_VERSION && mkdir -p /var/opt/magma/ | ||
| ---> Running in 46435f4fbad2 | ||
| Selecting previously unselected package python3-aioeventlet. | ||
| (Reading database ... 47574 files and directories currently installed.) | ||
| Preparing to unpack .../python3-aioeventlet_0.5.1-2focal_amd64.deb ... | ||
| Unpacking python3-aioeventlet (0.5.1-2) ... | ||
| Setting up python3-aioeventlet (0.5.1-2) ... | ||
| Removing intermediate container 46435f4fbad2 | ||
| ---> 08ed545b9000 | ||
|
|
||
| Successfully built 08ed545b9000 | ||
| Successfully tagged agw_gateway_python:latest | ||
| sdti-build1:~/magma/lte/gateway/docker # | ||
| ``` | ||
| * Check images on local host | ||
| * ctr-build:~/magma-ctr/magma/lte/gateway/docker #docker images | ||
|
|
||
| ``` | ||
| REPOSITORY TAG IMAGE ID CREATED SIZE | ||
| agw_gateway_c latest 755b206a9698 3 minutes ago 1.29GB | ||
| <none> <none> 5163a21390e8 6 minutes ago 4.41GB | ||
| agw_gateway_python latest d61fcb3ed86e 26 minutes ago 894MB | ||
| <none> <none> 6cb8b28381ce 29 minutes ago 1.71GB | ||
| <none> <none> 06c85a31b481 2 hours ago 4.41GB | ||
| <none> <none> b6e324a3c73a 3 hours ago 1.71GB | ||
| ubuntu focal ba6acccedd29 3 weeks ago 72.8MB | ||
| ``` | ||
|
|
||
| * Ensure repositories are created on dockerhub | ||
| * Tag and Push images to docker hub | ||
|
|
||
| ``` | ||
| ctr-build:~/magma-ctr/magma/lte/gateway/docker #docker image tag agw_gateway_python:latest arunuke/agw_gateway_python:9Nov | ||
| ctr-build:~/magma-ctr/magma/lte/gateway/docker #docker image tag agw_gateway_c:latest arunuke/agw_gateway_c:9Nov | ||
| ctr-build:~/magma-ctr/magma/lte/gateway/docker #docker image push arunuke/agw_gateway_c:9Nov | ||
| ctr-build:~/magma-ctr/magma/lte/gateway/docker #docker image push arunuke/agw_python:9Nov | ||
| ``` | ||
|
|
||
| * Use the Base Cloudstrapper image with an expanded disk size (64G at least) to create Test Container | ||
|
|
||
| ``` | ||
| ansible-playbook agw-provision.yaml --tags createGw -e '@~/magma-master/sdti-ctr1.yaml' | ||
|
|
||
| evsrv-tokyo:~/magma-master/magma/experimental/cloudstrapper/playbooks #cat ~/magma-master/sdti-ctr1.yaml | ||
| --- | ||
| dirLocalInventory: ~/cloudstrapper | ||
| awsAgwAmi: ami-0d058fe428540cd89 | ||
| buildUbuntuAmi: ami-04fade045b8da506f | ||
| awsCloudstrapperAmi: ami-0fad3311309aca4c9 | ||
| awsAgwRegion: ap-southeast-1 | ||
| keyHost: rmelero | ||
| idSite: MenloPark | ||
| idGw: devops01-agw-deploy-test | ||
| ``` | ||
|
|
||
| This is how your `dirLocalInventory` should look | ||
|
|
||
| ``` | ||
| cat ~/cloudstrapper/secrets.yaml | ||
| --- | ||
| awsAccessKey: | ||
| awsSecretKey: | ||
| ``` | ||
|
|
||
| After deploy, it might be necessary to resize ebs volume to 100 GB to accommodate the installation of docker and the build space for images. | ||
|
|
||
| Resize the disk partitions | ||
|
|
||
| ``` | ||
| growpart /dev/xvda 1; resize2fs /dev/xvda1 | ||
| ``` | ||
|
|
||
| ### Using a generic server with 2 NICs | ||
|
|
||
| Create/deploy an ubuntu 20.04 (latest version) server/instance with 2 interfaces and the appropriate resources for your use case. | ||
|
|
||
| Your interfaces should be named eth0 (SGi) and eth1 (S1). | ||
|
|
||
| Get the agw install script | ||
|
|
||
| ``` | ||
| wget https://github.com/magma/magma/raw/master/lte/gateway/deploy/agw_install_docker.sh | ||
| ``` | ||
|
|
||
| Add your rootCA.pem generated from your orc8r deployment | ||
| ``` | ||
| mkdir -p /var/opt/magma/certs | ||
| cp rootCA.pem /var/opt/magma/certs/rootCA.pem | ||
| ``` | ||
|
|
||
| Run install script to install docker, docker-compose, and bootstrap host | ||
|
|
||
| ``` | ||
| bash agw_install_docker.sh | ||
| ``` | ||
|
|
||
| After this, you can install your snowflake and gateway certs if you have them, or for new gateways, let them be generated. | ||
|
|
||
| ``` | ||
| cp snowflake /etc/snowflake | ||
| cp gateway.crt /var/opt/magma/certs | ||
| cp gateway.key /var/opt/magma/certs | ||
| ``` | ||
|
|
||
| Edit .env in /var/opt/magma/docker to have your docker registry values, and S1 and SGi interface IPs. | ||
|
|
||
| Pull images and start containers | ||
| ``` | ||
| cd /var/opt/magma/docker | ||
| ./agw_upgrade.sh | ||
| ``` | ||
|
|
||
| You can check your connection status with | ||
| ``` | ||
| docker exec -it magmad /usr/local/bin/checkin_cli.py | ||
| 1. -- Testing TCP connection to controller.orc8r-deployment.dev:8443 -- | ||
| 2. -- Testing Certificate -- | ||
| 3. -- Testing SSL -- | ||
| 4. -- Creating direct cloud checkin -- | ||
| 5. -- Creating proxy cloud checkin -- | ||
|
|
||
| Success! | ||
| ``` | ||
|
|
||
| # MME config changes | ||
|
|
||
| MME configuration is generated automatically from /var/opt/magma/configs/gateway.mconfig. | ||
| A restart of the mme container is required to apply new changes. | ||
| #Setting AGW AMI and Cloudstrapper AMI to expanded Cloudstrapper image Ubuntu to allow deploy | ||
| dirLocalInventory: ~/magma-master | ||
| awsAgwAmi: ami-03bc7ef7f3b70f77b | ||
| awsCloudstrapperAmi: ami-03bc7ef7f3b70f77b | ||
| awsAgwRegion: ap-northeast-1 | ||
| keyHost: keyMagmaHostCharlie | ||
| idSite: SDTI | ||
| idGw: sdti-ctr1 | ||
| awsInstanceType: t3.large | ||
| ``` | ||
|
|
||
| * Prepare the host | ||
| * Install ifupdown | ||
| * Unlink `/etc/resolv.conf` and create a new one with 8.8.8.8 entry | ||
| * Create `/var/opt/magma/certs `and add rootCA.pem to that folder with permissions 400 | ||
| * Copy `agw_install_docker.bash` and run script to prepare the host | ||
| * Make changes to `/var/opt/magma/docker/.env` to include the right docker information | ||
| * DOCKER_REGISTRY=[registry.hub.docker.com/arunuke/](http://registry.hub.docker.com/arunuke/) | ||
|
|
||
| ``` | ||
| DOCKER_USERNAME=arunuke | ||
| DOCKER_PASSWORD=XXX | ||
| ``` | ||
|
|
||
| * Make changes to files in `/var/opt/magma/configs` if needed | ||
| * pipelined, dnsd, enodebd, spgw, mme all will have their eth0/eth1 changed to newer values based on local interface names (eth0 and eth2 references to use the first interface for SGi and eth1 references to use the second interface for S1) | ||
| * pipelined will also set dp_router_enabled to false | ||
| * Make changes to config files and restart services by running `/var/opt/magma/docker/agw_upgrade.sh` or by running the `agw_install_docker.bash` script | ||
| * Issues | ||
| * Needs ifup on the host. Install package ifupdown (PR in progress) | ||
| * Need to resolve external IP addresses after bringing up interfaces. unlink /etc/resolv.conf, add a new entry for 8.8.8.8. Add this by creating a new role. (PR needed) | ||
| * Need to setup variables in the [variables file](https://github.com/magma/magma/tree/master/lte/gateway/deploy/roles/agw_docker/vars) under magma_root (/opt/magma) and it has to be documented in the README (PR needed) | ||
| * Need to fix externally pulled images from docker hub and/or aws ECR (works as expected, needs a README note on format) | ||
| * Target config files are set based on localized .env file. Need to fix interface name changes in [config files.](https://github.com/magma/magma/tree/master/lte/gateway/configs) (PR needed) | ||
| * Not required: Need to stop cloning into magma every time which over-writes any existing configuration, or provide a way to start/stop containers alone as a whole (clearly labelled flags) | ||
|
|
||
| ## AWS Specifics | ||
|
|
||
| * CloudFormation | ||
| * Individual stacks for EKS cluster | ||
| * EFS for shared storage (supported across fargate, managed and self-managed nodegroups) | ||
| * EKS | ||
| * Fargate compute | ||
| * Supports all Linux workloads | ||
| * Supports EFS for storage | ||
| * Private subnet only | ||
| * Managed node-groups | ||
| * Needed for GPU compute, but AL only | ||
| * Supports ARM | ||
| * Supports Bottle Rocket | ||
| * Custom AMI and CNI support | ||
| * Supports EBS and EFS | ||
| * Supports Daemonsets | ||
| * Self-managed nodes | ||
| * AWS Local Zones and Outpost can support self managed nodes only | ||
| * Can support GPU, but AL only | ||
| * Supports ARM | ||
| * Supports BottleRocket | ||
| * Supports EBS and EFS | ||
| * Supports Daemonsets | ||
| * [Nodegroups](https://docs.aws.amazon.com/eks/latest/userguide/eks-compute.html) | ||
| * Steps | ||
| * Create launch template for custom Ubuntu AMI that configures host with packages, OVS | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[misspell] reported by reviewdog 🐶
"labelled" is a misspelling of "labeled"