Deploying AGW and Orc8r On-Premises and Bare Metal

[jblakley]

TL;DR

Recounts the tips and tricks of deploying a private, on-premises, baremetal Magma Orchestrator (orc8r) and Access Gateway (AGW). No eNodeB has yet been provisioned.

What and Why?

I recently completed the bring up of AGW and Orc8r in a "closed" baremetal on-premises environment. Since baremetal is a relatively new deployment model for magma, especially the orchestrator, I thought I'd share some of my heartaches, learnings, tips & tricks, and suggestions for improvements. I'd welcome others to communicate their learning as well -- although the magma slack channel is a better place for detailed back and forth.

Why Baremetal?

l started my learning on magma with the quickstart -- it is an excellent tool to get a first experience on magma "in an afternoon". However, at this time, on magma 1.6, the recommended deployment model is baremetal AGW on Ubuntu 20.04 and AWS for the orchestrator. AGW baremetal makes sense for performance and networking reasons -- it's hard to plug an S1 interface into the cloud. Cloud-based Orc8r is a fine alternative but I chose to bring on-premises because:

• Orc8r on AWS has some account constraints that our institutional account doesn't meet.
• Being able to deploy on baremetal means we'll ultimately be able to deploy in any cloud, virtualized or baremetal environment we choose.
• Baremetal gives more transparency and simplicity to solve some of the challenges in networking, security, and DNS below.
• There's some peace of mind that comes with having the AGW and the Orc8r in the same room and on the same switch.

What Baremetal?

Our AGW is an Intel(R) Core(TM) i7-5960X CPU @ 3.00GHz with two 1Gbps NICs with 32GB RAM running Ubuntu 20.04. The Orc8r is an Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 128GB RAM running Ubuntu 20.04.

This deployment is on magma 1.6.0 and will eventually be rolled into an already operational private LTE network.

Key Challenges

The key references for deployment come from these resources:

• Install Access Gateway on Ubuntu (Bare Metal)
• Build Orchestrator
• Install Orchestrator with Ansible
• Install Orchestrator (on AWS) -- Even though this is for AWS, some of the info is useful.

The challenges I describe below arose during the course of following these directions. From the dialogs in the slack channels, I'm not alone. There are some error messages that I ran into that I don't discuss here either because they were trivial or they didn't seem to cause any problems (yet).

AGW System Constraints

The baremetal AGW requires the network interfaces be specifically named. The SG1 interface must be named eth0 and the S1 interface must be named eth1. In addition, netplan, the default Ubuntu 20.04 networking scheme, must be disabled and replaced with ifupdown. If the system is not initialy configured this way, the agw_install_ubuntu.sh script will roll back to ifupdown and rename the interfaces. However, due to constraints on our networking environment, the automated reconfiguration assigned the wrong ports to those names. That issue required that I manually convert to ifupdown and appropriately rename prior to running the install script. Instructions for this conversion can be found here. My /etc/network/interfaces file looks like this:

auto lo
iface lo inet loopback

rename eno1=eth0
auto eth0
iface eth0 inet dhcp
	hwaddress ether <SG1-MAC>
	dns-nameserver 8.8.8.8

rename en5s0=eth1
auto eth1
iface eth1 inet static
    address <S1-IP>
    hwaddress ether <S1-MAC>

Certificates

Documentation for handling certificates is somewhat unclear in the baremetal instructions. Portions of the AWS Orc8r documentation apply, however there are few things that don't:

1. The Orc8r deployment script, deploy.sh, creates the certificates for Orc8r and stores them in /etc/orc8r/certs.

2. The rootCA.pem certificate to install on the AGW is there.

3. The admin_operator certificate is also there but you still need to run:

   openssl pkcs12 -export -inkey admin_operator.key.pem -in admin_operator.pem -out admin_operator.pfx

   to generate admin_operator.pfx for browsers.

4. The AGW certificate, gateway.crt, and its key, gateway.key, are generated when the AGW checks into the Orc8r. They won't be there until after you complete the bringup process and the gateway has successfully checked in.

Helm Charts

To build and publish helm charts, I used option #2 from here. The repository needs to be named magma-charts. I made the repository public since the package script did not seem to pick up the credentials. Since I have an LTE network, my package script command line was:

${MAGMA_ROOT}/orc8r/tools/helm/package.sh -d all

A temporary(?) anomaly is that the chart versions are 1.5.23 not 1.6.0 as expected.

The Ansible Variable File

From the slack forum, many people seem to struggle with setting ansible_vars.yaml variables to get them to work. Here is an anonymized working version of mine:

---
ansible_user: ubuntu

helm_enabled: true
kube_network_plugin: calico
kube_network_plugin_multus: false

kube_apiserver_port: 6443
kube_pods_subnet: 10.233.64.0/18
kube_network_node_prefix: 24
kube_service_addresses: 10.233.0.0/18
kubeadm_control_plane: "true"
etcd_kubeadm_enabled: true

enable_nodelocaldns: true
resolvconf_mode: host_resolvconf

# Copies kubectl and kubeconfig to your ansible host
kubectl_localhost: true
kubeconfig_localhost: true

# Change these to your private nameservers if needed
nameservers:
  - <mylocaldnsIP>
  - 8.8.4.4
upstream_dns_servers:
  - 8.8.8.8
  - 8.8.4.4

# If we have a docker registry without trusted SSL or http only, add here
# docker_insecure_registries:
#   - private_registry:5000

docker_cgroup_driver: systemd
kubelet_cgroup_driver: systemd

## Kubernetes LoadBalancer
## Enabling a loadbalancer is optional, but provides HA for external access to Kubernetes API
## loadbalancer_apiserver and vrrp_nic need to be set if you want to enable this feature.

# Change to match the network of nodes where you want to run loadbalancer
#loadbalancer_apiserver:
#  address: 192.168.0.20
#  port: 8383

# Set to the NIC on which you want to run loadbalancer
#vrrp_nic: "eth0"

## Orc8r settings
## These variabless must be set
orc8r_image_repo: registry.hub.docker.com/<mylogin>
orc8r_helm_repo: https://raw.githubusercontent.com/<mygithubacct>/magma-charts/master
orc8r_domain: <mydomain>.org
orc8r_nms_admin_email: <myemail>

# Component versions (orc8r 1.3.x or 1.4.x supported)
orc8r_chart_version: 1.5.23
orc8r_image_tag: 1.6.0
orc8r_nms_image_tag: 1.6.0
orc8r_nginx_image_tag: 1.6.0

# Component passwords are randomly generated (and stored in credentials dir).
# Uncomment the following lines to explicitly set passwords
# db_root_user:
# db_root_password:
# orc8r_db_pass:
# orc8r_nms_db_pass:
# orc8r_nms_admin_pass:

# (REQUIRED): LoadBalancer settings for Magma services publicly exposed
# These address are exposed to the AGW(s) via a wireguard VPN
metallb_addresses: 192.168.1.10-192.168.1.15

# Advanced options for enabling/disabling Helm charts
deploy_nfs_server_provisioner: True
kubevirt_enabled: False
deploy_metallb: True
deploy_mariadb: True
deploy_elasticsearch: True
deploy_fluentd: True
magma_namespace: magma

IP Connections

In my environment and without using public IP addresses, it was difficult to create an IP connection between the Orc8r and the AGW. After trying a number approaches, what finally worked best was to create a wireguard VPN on 192.168.1.0, assign the metallb addresses to be in this subnet and route AGW traffic to those services through the VPN. To do this persistently, see Section 3 here.

For debugging IP and DNS, these commands were invaluable:

echo EXTERNAL-IP
kubectl -n magma get svc -ojsonpath='{range .items[?(@.spec.type=="LoadBalancer")]}{.status.loadBalancer.ingress[0].ip}{" "}{.metadata.annotations.external-dns\.alpha\.kubernetes\.io/hostname}{"\n"}{end}'
echo CLUSTER-IP
kubectl -n magma get svc -ojsonpath='{range .items[?(@.spec.type=="LoadBalancer")]}{.spec.clusterIP}{" "}{.metadata.annotations.external-dns\.alpha\.kubernetes\.io/hostname}{"\n"}{end}'

Their output looks like:

EXTERNAL-IP
192.168.1.10 fluentd.orc8r.mydomain.org
192.168.1.13 *.nms.orc8r.mydomain.org
192.168.1.12 bootstrapper-controller.orc8r.mydomain.org
192.168.1.11 controller.orc8r.mydomain.org
192.168.1.14 api.orc8r.mydomain.org
CLUSTER-IP
10.233.55.8 fluentd.orc8r.mydomain.org
10.233.31.214 *.nms.orc8r.mydomain.org
10.233.26.25 bootstrapper-controller.orc8r.mydomain.org
10.233.0.238 controller.orc8r.mydomain.org
10.233.2.157 api.orc8r.mydomain.org

DNS

Getting the DNS right from within the Orc8r k8s cluster and between the AGW and the Orc8r were the most challenging (especially with my lack of DNS experience). I wanted to use a private domain name and didn't want dependency on an external DNS server. What I did:

1. Deployed a bind9 DNS server on the Orc8r host node and updated netplan to point to its public address as a nameserver. Here's the config file:

   $TTL    604800
   @       IN      SOA     mydomain.org. root.mydomain.org. (
                                 5         ; Serial
                            604800         ; Refresh
                             86400         ; Retry
                           2419200         ; Expire
                            604800 )       ; Negative Cache TTL
   ;
   @       IN      NS      ns.mydomain.org.
   @       IN      A       <mylocaldnsIP>
   @       IN      AAAA    ::1
   ns      IN      A       <mylocaldnsIP>
   fluentd.orc8r   IN      A       192.168.1.10
   nms.orc8r       IN      A       192.168.1.13
   master.nms.orc8r       IN      A       192.168.1.13
   my-network-name.nms.orc8r       IN      A       192.168.1.13
   bootstrapper-controller.orc8r       IN      A       192.168.1.12
   controller.orc8r       IN      A       192.168.1.11
   api.orc8r       IN      A       192.168.1.14
   

2. Updated the k8s cluster coredns and localnodedns pods to use it as the only forwarding nameserver. Also having 8.8.8.8 as forwarding nameserver caused issues with internal resolution within the cluster.

   Coredns :53 Block:

       .:53 {
           errors
           health {
               lameduck 5s
           }
           ready
           kubernetes cluster.local in-addr.arpa ip6.arpa {
             pods insecure
             fallthrough in-addr.arpa ip6.arpa
           }
           prometheus :9153
           forward . <mylocaldnsIP> {
             prefer_udp
           }
           cache 30
           loop
           reload
           loadbalance
       }
   

   Nodelocaldns :53 Block:

       .:53 {
           errors
           cache 30
           reload
           loop
           bind 169.254.25.10
           forward . <mylocaldnsIP>
           prometheus :9253
       }
   

3. To be safe, I also added the load balancer IPs to the /etc/hosts of the AGW:

   192.168.1.10 fluentd.orc8r.mydomain.org
   192.168.1.13 master.nms.orc8r.mydomain.org name_of_my_network.nms.orc8r.mydomain.org
   192.168.1.12 bootstrapper-controller.orc8r.mydomain.org
   192.168.1.11 controller.orc8r.mydomain.org
   192.168.1.14 api.orc8r.mydomain.org
   

Don't forget to create an Orc8r and NMS user and import your admin_operator certificate into your browser.

Once you get this far you should be able:

1. Open NMS in your local Orc8r browser and login.
2. Create a new network in NMS.
3. Add your AGW to the network.
4. Verify that the AGW successfully checked in.

Here's the NMS dashboard for the network at this stage. I have not yet added an eNodeB although I have connected it to the S1 interface.

Suggestions for Improvement

• The major suggestion is creation of complete, consolidated documentation for this deployment model. Most of the information is there but difficult to find. Additional documentation on network and DNS setup would be very helpful. Clarification on how to find and manage certificates would be useful.
• The AGW dependency on ifupdown and eth0/eth1 naming seems unnecessary and required a workaround that in retrospect is straightforward but took a while to figure out. It also makes the solution fragile since it depends on specific hardware and operating system settings.
• There are a number of other build and deployment issues that people are having that I also had but don't mention here. Worth looking through there for more potential fixes.

Please feel free to share your comments and tips of your own!

[jdesai61ilu]

Thank you for the detailed notes. I am able to bring up the orchestrator and AGW separately. But still can't add the AGW to NMS because I can't create a 'Network' in the UI. When I try to create a new LTE Network using UI, I get a cryptic "Not Found" error. Seems like the error is stemming from the fact that lte-orc8r chart is not deployed in bare-metal-ansible instructions above. I tried to manually issue helm command to deploy lte charts, but the new pods are stuck in ContainerCreating state . This issue (#6252) has some clues but I don't know what to modify in values.yaml

sudo helm upgrade --install -n magma lte-orc8r orc8r/lte-orc8r --version 0.2.5 --wait --debug --timeout=15m

orc8r-ha-dc8c4df79-bw5g8                         0/1     ContainerCreating   0          2m8s
orc8r-lte-77746bbbf4-nsrvk                       0/1     ContainerCreating   0          2m8s
orc8r-policydb-d56f57875-zl9df                   0/1     ContainerCreating   0          2m8s
orc8r-smsd-6fd88d56cf-xm2xb                      0/1     ContainerCreating   0          2m8s
orc8r-subscriberdb-7b5cb47c7f-5hc97              0/1     ContainerCreating   0          2m8s
orc8r-subscriberdb-cache-5d8dd8d6c8-gp8jd        0/1     ContainerCreating   0          2m8s

[jblakley]

Did you run this to configure the helm charts?

${MAGMA_ROOT}/orc8r/tools/helm/package.sh -d all

Note the all. The documentation has this as fwa. If I remember, the all tells it to deploy the LTE chart.

[jdesai61ilu]

The charts are present in the repo

sudo helm search repo orc8r
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
orc8r/cwf-orc8r         0.2.1           1.0             A Helm chart for magma orchestrator's cwf module
orc8r/fbinternal-orc8r  0.2.1           1.0             A Helm chart for magma orchestrator's fbinterna...
orc8r/feg-orc8r         0.2.4           1.0             A Helm chart for magma orchestrator's feg module
orc8r/lte-orc8r         0.2.5           1.0             A Helm chart for magma orchestrator's lte module
orc8r/orc8r             1.5.23          1.0             A Helm chart for magma orchestrator
orc8r/wifi-orc8r        0.2.1           1.0             A Helm chart for magma orchestrator's wifi module

The github repo index.yaml repo shows

  lte-orc8r:
  - apiVersion: v2
    appVersion: "1.0"
    created: "2021-12-02T17:20:26.292476719Z"
    dependencies:
    - name: orc8rlib
      repository: file://../../../../orc8r/cloud/helm/orc8rlib
      version: 0.1.2
    description: A Helm chart for magma orchestrator's lte module
    digest: 778cdf0bb798ae414872faa22d819a5993c8ba9bd55efc49131794016515c042
    keywords:
    - magma
    - or8cr
    - lte-orc8r
    name: lte-orc8r
    sources:
    - https://github.com/magma/magma
    urls:
    - lte-orc8r-0.2.5.tgz
    version: 0.2.5

[jdesai61ilu]

When I look at the PODS, looks like some persistent volumes do not exist, which makes me want to think, I missed some initialization step
or my manual command to deploy lte-orc8r is missing something...

sudo helm upgrade --install -n magma lte-orc8r orc8r/lte-orc8r --version 0.2.5 --wait --debug --timeout=15m

sudo kubectl -n magma describe pod orc8r-ha-7c54b44fff-njtgs
---
    Mounts:
      /var/opt/magma/certs from certs (ro)
      /var/opt/magma/configs/orc8r from orc8r-secrets-configs-orc8r-orc8r (ro)
      /var/opt/magma/envdir from envdir (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-t2z24 (ro)

---
Events:
  Type     Reason       Age                   From     Message
  ----     ------       ----                  ----     -------
  Warning  FailedMount  47m                   kubelet  Unable to attach or mount volumes: unmounted volumes=[orc8r-secrets-configs-orc8r-orc8r certs envdir], unattached volumes=[orc8r-secrets-configs-orc8r-orc8r default-token-t2z24 certs envdir]: timed out waiting for the condition
  Warning  FailedMount  36m (x23 over 67m)    kubelet  MountVolume.SetUp failed for volume "certs" : secret "orc8r-secrets-certs" not found
  Warning  FailedMount  16m (x33 over 67m)    kubelet  MountVolume.SetUp failed for volume "envdir" : secret "orc8r-secrets-envdir" not found
  Warning  FailedMount  6m24s (x18 over 65m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[certs envdir orc8r-secrets-configs-orc8r-orc8r], unattached volumes=[certs envdir orc8r-secrets-configs-orc8r-orc8r default-token-t2z24]: timed out waiting for the condition
  Warning  FailedMount  2m9s (x40 over 67m)   kubelet  MountVolume.SetUp failed for volume "orc8r-secrets-configs-orc8r-orc8r" : secret "orc8r-secrets-configs-orc8r" not found

[jdesai61ilu]

I was able to get some help on slack and come up with following script to start extra pods (still facing errors adding network)

#!/usr/bin/env bash

HELM_REPO="orc8r"

orc8r_helm_charts=("lte" "feg" "cwf" "wifi" "fbinternal")

for orc8r_chart in "${orc8r_helm_charts[@]}"
do
    sudo helm upgrade -n magma -i ${orc8r_chart}-orc8r ${HELM_REPO}/${orc8r_chart}-orc8r \
         -f /etc/orc8r/charts/orc8r-values.yaml
done

[velosonetmao]

Hi,

Could you give me instructions how can I fill out this lines?

orc8r_image_repo: registry.hub.docker.com/
orc8r_helm_repo: https://raw.githubusercontent.com//magma-charts/master

I´ve full fill with my my credentials (I´ve publish public magma-charts) but I´m getting error when I run deploy.sh

`ERROR! Syntax Error while loading YAML.
did not find expected key

The error appears to be in '/home/magma/magma/orc8r/cloud/deploy/bare-metal-ansible/ansible_vars.yaml': line 50, column 2, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

These variabless must be set

orc8r_image_repo: hub.docker.com/u/velosonet
^ here
`

[jblakley]

I'm not sure about the /u in hub.docker.com/u/velosonet. I do not have that in my working ansible_vars.

[jdesai61ilu]

Unless you really want a specific version, or need to build your own charts, you can just use publicly available repo. Here is my entry

orc8r_image_repo: magmacore
orc8r_helm_repo: https://artifactory.magmacore.org/artifactory/helm/

But when I was using my own charts, the bare-metal-ansible scripts didn't support specifying github credentials, so I had to modify bare-metal-ansible/roles/magma/orc8r/tasks/main.yml as follows:

- name: orc8r | Ensure orc8r helm repo
  command: helm repo add --username {{ orc8r_helm_github_username }} --password {{ orc8r_helm_github_token }} orc8r {{ orc8r_helm_repo }}
  

And then specify the credentials in ansible_vars.yaml as:

orc8r_helm_github_username: MYGITHUBUSERID
orc8r_helm_github_token: MYGITHUBTOKEN
orc8r_helm_repo: https://raw.githubusercontent.com/MYGITHUBUSERID/magmaorc8r/master/
orc8r_image_repo: registry.hub.docker.com/MYDOCKERREGISTRY

[velosonetmao]

Ok guys, I´ll try this configurations.

Thanks a very much.

[maaathieu-x]

I tried this method. Everything went well, I can access master.nms.mydomain, create an organization, login as a user of this org, but then I have an error message:
Unable to communicate with magma controller: Service Unavailable

Indeed, https://organization.nms.mydomain/nms/apicontroller/magma/v1/networks answers "503 service unavailable".

I don't know what to check order to debug this. Any idea ?

Thanks

[maaathieu-x]

the "503 service unavailable" error was indeed a dns error. I can now access the create network menu, but it fails at the first step. If I choose "lte" as network type, I get a "Not Found" error. With a "third party" network type, the network appears in the list, but if I refresh the page it disappears.
In the orc8r pod logs I found this error:
REST HTTP Error: rpc error: code = Unknown desc = error creating entity upgrade_tier-default: Error 1452: Cannot add or update a child row: a foreign key constraint fails ("orc8r"."cfg_entities", CONSTRAINT "cfg_entities_ibfk_1" FOREIGN KEY ("network_id") REFERENCES "cfg_networks" ("id") ON DELETE CASCADE), Status: 500

In the nms database, my organization is well created, but I see no tenant in the orc8r database, maybe the problem comes from that.

[assyafii]

I tried this method. Everything went well, I can access master.nms.mydomain, create an organization, login as a user of this org, but then I have an error message: Unable to communicate with magma controller: Service Unavailable

Indeed, https://organization.nms.mydomain/nms/apicontroller/magma/v1/networks answers "503 service unavailable".

I don't know what to check order to debug this. Any idea ?

Thanks

Same, like my issue, im still troubleshoot for it

[maaathieu-x]

in my case it was because the lte-orc8r chart was missing. what does this command return ?

helm --namespace magma list

[assyafii]

This our helm chart

This impact, cannot manages & access networking services

[maaathieu-x]

I think you miss the lte-orc8r chart.
You can install it with this kind of command:
helm upgrade --install -n magma lte-orc8r orc8r/lte-orc8r --version 1.6 -f /etc/orc8r/charts/lte-orc8r-values.yaml --wait

or you can modify this file:
orc8r/cloud/deploy/bare-metal-ansible/roles/magma/orc8r/tasks/main.yml

by adding "lte-orc8r" in this section:

- name: orc8r | Install/upgrade helm charts
command: >-
helm upgrade --install
-n {{ magma_namespace }}
{{ item }}
orc8r/{{ item }}
{% if orc8r_chart_version is defined %} --version {{ orc8r_chart_version }}{% endif %}
-f {{ charts_dir }}/{{ item }}-values.yaml
--wait
with_items:
- orc8r
- lte-orc8r

and start the installation again

[dewadragoon]

Hi jblakley,

i am using magmacore 1.7.0. i followed the quick start guide, however i am stuck at this step

while on my actual system ( ubuntu 20.04 )

there is no "nms/packages/magmalte" directory.

the AGW build part was working, but now i am unable to connect AGW - Orchestrator because the NMS build portion failed.

[dewadragoon]

now i am stuck at this

markupsafe 2.1.1

[dewadragoon]

made another progress, but stuck again as below

[dewadragoon]

this is truly like 20 years back when installing any apps require manual compile.. :) and always have missing dependency. need extra effort and patience..

[dewadragoon]

another progress yet another roadblock

[dewadragoon]

and another roadblock

[maaathieu-x]

try to install an older version of markupsafe, like this:

pip3 install markupsafe==2.0.1

It worked for me

[dewadragoon]

yup... tried that... i even downgrade until v1.x.x. Still same issue.

[maaathieu-x]

I'm trying to install v1.8.0 and the problem with markupsafe is still present.
you need to install markupsafe 2.0.1 inside the python venv, like this:

source .venv/bin/activate
pip3 install markupsafe==2.0.1

but then I have this error:

ERROR! this task 'ansible.builtin.command' has extra params, which is only allowed in the following modules: include_tasks, shell, group_by, include, include_vars, set_fact, win_shell, import_role, script, import_tasks, include_role, command, add_host, raw, meta, win_command

it seems to be a problem with ansible v2.9.6:

ansible/ansible#71824

so I'll try with another version.

[jblakley]

Just an update. I recently completed a local bring up of the 1.8.0 Orc8r and AGW. I don't plan to produce something as detailed as above especially since the whole process was much simpler than 1.6. Here are some of the top level comments:

Even though I have dedicated servers for both AGW and Orc8r, I deployed into a KVM VM on each server. That made it easier to deal with networking and whenever I had an issue, I could spin up a new VM.

I used @ShubhamTatvamasi guide for the Orc8r. Pretty straightforward.

I used Install Docker-based Access Gateway on Ubuntu for the AGW. Note that the AGW won't check in until after the final sudo docker-compose up -d --force-recreate.

I have not yet provisioned an eNB for the network. Will do that over the next few weeks. After we do that successfully, we'll migrate our 1.6 network over to 1.8.

A few known issues as of this writing:

• The checkin_cli.py script gives an error with the docker-based AGW. This doesn't cause execution issues but it is a little discomfiting to not know for sure. I've relied on nms indication that the AGW has actually checked in.

sudo docker exec magmad checkin_cli.py
1. -- Testing TCP connection to controller.orc8r.magma18.livingedgelab.org:443 -- 
2. -- Testing Certificate -- 
3. -- Testing SSL -- 
4. -- Creating direct cloud checkin -- 

> Error: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNAVAILABLE: Socket closed"
	debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2022-09-27T16:13:14.845596181+02:00", children:[UNKNOWN:failed to connect to all addresses; last error: UNAVAILABLE: Socket closed {created_time:"2022-09-27T16:13:14.845592381+02:00", grpc_status:14}]}"
>

• I haven't found good way to get logging info (e.g., journalctl -f -u magma@mme) from the containers.

• There is a bug in @ShubhamTatvamasi's guide that can cause problems if you have multiple NICs in your Orc8r server. It can cause the k8s loadbalancer to be assigned to the wrong IP. He's working on it.

[asantos-vk]

I am getting the following errors during installation:

Unable to restart service docker: Job for docker.service failed because the control process exited with error code.\nSee "systemctl status docker.service" and "journalctl -xe" for details.

It seems that the docker.service template is failing to start/restart

how do I proceed?

[amintgan]

Hello @jblakley
Please i want to confirm if Magma supports VOLTE.

[jblakley]

@amintgan I'm not sure -- we haven't tried voice at all -- although we should.