Third Party Container Management Enhancements to SONiC Application Extensions framework

Rev 0.2

Table of Content

List of Tables
Revision
Scope
Definitions/Abbreviations
Document/References
Overview
Requirements
Architecture Design
High-Level Design
SAI API
Configuration and management
Restrictions/Limitations
Testing Requirements/Design

List of Tables

Table 1: Abbreviations
Table 2: References

Revision

Rev	Date	Author	Change Description
0.1	10/14/2022	Kalimuthu Velappan, Senthil Guruswamy, Babu Rajaram	Initial version
0.2	03/10/2023	Senthil Guruswamy	Update

This document provides High Level Design information on extending SONiC Application Extension Infrastructure to seamlessly support a Third Party Application.

Scope

There are many Third Party application dockers, that can be used in SONiC to provision, manage and monitor SONiC devices. The dockers need not be compatible with SONiC and also need not be predefined or qualified with Sonic. These are extensions to SONiC and can be dynamically installed, similar to debian apt install or pip install paradigms. To support this, we need additional capabilities to seamlessly integrate with SONiC. These are related to installation, upgrade, and configuration. SONiC Application Extension Framework already provides an infrastructure to integrate applications, and this document details the enhancements.

Definitions/Abbreviations

Table 1 Abbreviations

Abbreviation	Definition
SONiC	Software for Open Networking in Cloud
DB	Database
API	Application Programming Interface
SAI	Switch Abstraction Interface
YANG	Yet Another Next Generation
JSON	Java Script Object Notation
XML	eXtensible Markup Language
gNMI	gRPC Network Management Interface
TPC	Third Party Container
TPCM	Third Party Container Management
AEF	Application Extension Framework

Document/References

Table 2 References

Document	Location
SONiC Application Extension Infrastructure HLD	[https://github.com/stepanblyschak/SONiC/blob/8d1811f7592812bb5a2cd0b7af5023f5b6449219/doc/sonic-application-extention/sonic-application-extention-hld.md)

Overview

SONiC is an open and extensible operating system. SONiC can be extended with Third Party Containers to help with Orchestration, Monitoring, and extending any features or capabilities of SONiC NOS. These can be custom built or readily available in docker community. The rich open eco system provides with multiple docker based applications which can be used in SONiC. And these need not be pre-defined packages in SONiC. While SONiC Application Extensions framework provides the infrastructure to package and integrate a TPC into SONiC, we need more capabilities to provide a dynamic environment and capabilities:

Dynamically install TPCs from various sources without pre-packaging them in SONIC.
Ensure TPC has right system resources and privileges
Abilitiy to configure startup command arguments to TPC
Configure resource limits to TPCs, to ensure it does not starve core SONiC containers
Ability to seamlessly integrate into SONiC Services Architecture enabling start/ auto restart/ establish dependencies

Requirements

These open TPCs help extend SONiC capabilities, and thus the following requirements are outlined to integrate them seamlessly into SONiC

Dynamically install TPC from docker registry and docker image from the local file system, SCP, SFTP, URL. These remote download options are introduced now to SONiC AEF.
Upgrade TPCs from docker registry and docker image from local file system, SCP, SFTP, URL
Provide default manifest file for docker image without manifest during install.
Provide runtime install capability to pass various docker startup arguments and parameters through local custom manifest file.
Specify system resource limits for TPCs to restrict CPU, Memory usage
Provide update capability to update various TPC configurations like their memory, cpu, dependencies etc. This capability is new to SONiC AEF.

Architecture Design

There is no change to the SONiC architecture.

High-Level Design

SONiC Application Extension Infrastructure provides the framework to integrate SONiC compatible dockers. However there are many open source third party applications which can be installed on SONiC system, to extend the capabilities, and these typically are standalone or have less interaction with the SONiC system itself. So it is not necesary to pre define these docker applications in SONiC build, and provide their corresponding manifest.json. These TPCs can be installed dynamically on a SONiC device and can be managed.

This feature enables the installation and management of third-party Docker packages without manifest files through the Sonic Package Manager.
Also, it manages the TPC without affecting the sonic system resources.
This feature enables users to create a local custom manifest file (from default manifest template file) on-the-fly which can be updated or deleted as needed.
Additionally, the Sonic Package Manager provides CLI commands to create, update, and delete the local manifest file.

Figure. SONiC TPCM Support

TPC Install

The TPCM support feature will leverage the existing Sonic Package Manager framework.
Ability to download image tarballs for the Docker packages through SCP, SFTP, and URL before installing them is introduced.
If a manifest file is not found in the docker image during installation, a default local TPC manifest file is used to install the package.
A Docker image that does not contain a manifest file will be considered a Third-Party Container (TPC).
At the end of the TPC installation using sonic-package-manager install, if no manifest file is found in the image and the --name option is used, a new manifest file with the specified name is created.
The user can also create a custom local manifest file to be used during installation by specifying the "--use-local-manifest" option along with a custom name using the "--name" option in "sonic-package-manager install" command.
The custom local manifest file can be created using "sonic-package-manager manifests create" command
The custom local manifest file will be created under the directory /var/lib/sonic-package-manager/manifests/.
"--name" option takes effect only for TPC packages(that do not have a manifest file in them) and the custom local manifest file with the specified name is created.
The --name option enables the installation of multiple TPC containers using the same TPC docker image.
The default TPC manifest file will have the following properties:

{
    "version": "1.0.0",
    "package": {
        "version": "1.0.0",
        "depends": [],
        "name": "default"
    },
    "service": {
        "name": "default",
        "requires": [
            "docker"
        ],
        "after": [
            "docker"
        ],
        "before": [],
        "dependent-of": [],
        "asic-service": false,
        "host-service": false,
        "warm-shutdown": {
            "after": [],
            "before": []
        },
        "fast-shutdown": {
            "after": [],
            "before": []
        },
        "syslog": {
            "support-rate-limit": true
        }
    },
    "container": {
        "privileged": false,
        "volumes": [],
        "tmpfs": [],
        "memory": "512M",
        "cpu-quota": "20000",
        "cpu-period": "100000",
        "command": ""
    },
    "cli": {
        "config": "",
        "show": "",
        "clear": ""
    },
    "tpcm": true
}

The default manifest file contains specifications allowing TPC packages to utilize 20% of the CPU and up to 512MB of memory.

TPC Update

The manifest file properties that can be updated for a TPCM Docker package are:

'privileged', 'memory', 'cpu-quota', 'cpu-period', 'command' under container category
'requires', 'after', 'before', 'asic-service', 'host-service' under service category.

The local manifest file can be updated using "sonic-package-manager manifests update" command
The 'command' attribute is used to update the docker startup command arguments. For example, setting "command":"--path.rootfs=/host" would configure the host filesystem as the root filesystem path for the container
After updating the local manifest file, the TPC container can be updated with the "sonic-package-manager update" command. This command will reinstall the TPC container using the new manifest file.

TPC Uninstall

The uninstallation process for TPC packages using 'sonic-package-manager uninstall' follows the same procedure as regular Sonic packages, where the corresponding Docker container and service are stopped and removed, and the package image is deleted if it is not in use. The local manifest file associated with the TPC package is also removed during uninstallation.

TPC Upgrade

The upgrade process for TPC packages using 'sonic-package-manager install' follows the same procedure as regular Sonic packages.
Since the manifest package version mismatch is the trigger for package upgrade, the user must use --force option for TPC upgrade.
The new version of the TPC package is downloaded and installed, and the new manifest is created if necessary.
The old version of the TPC package is then removed if not in use by other package, and the running container is restarted with the new version.

Resource Limits for TPC

Following system resource limits shall be applicable to TPCs:

CPU
System Memory

These limits can be configured during the TPC installation.

CPU Resource

By default, for each TPC docker, the CPU resource limit is restricted to 20%.

Memory Limits

By default, for each TPC docker, the memory limit is restricted to 512M

SAI API

Not applicable

Configuration and management

CLI Enhancements

The SONiC Package Manager is another executable utility available in base SONiC OS called sonic-package-manager or abbreviated to spm. This would be extended to support these new TPCM capabilities. The command line interfaces are given bellow:

CLI

TPC Installation

This section shows the additional options that would be added for TPC installation in the sonic-package-manager CLI

admin@sonic:~$ sudo sonic-package-manager install --help

Usage: sonic-package-manager install [OPTIONS] [PACKAGE_EXPR]

  Install SONiC package.

Options:
  --from-tarball-scp                Install using the tarball from SCP source
  --from-tarball-sftp               Install using the tarball from SFTP source
  --from-tarball-url                Install using the tarball from HTTPS source
  --use-local-manifest              Use locally created manifest file
  --name                            custom name for tpcm package

Examples

admin@sonic:~$ sudo sonic-package-manager install httpd:latest --name my_httpd

Install from url:

admin@sonic:~$ sudo sonic-package-manager install --from-tarball-url https://tpc.local-server.com/home/tpcs/httpd.tar.gz --name=my_url_httpd

Install from scp:

admin@sonic:~$ sudo sonic-package-manager install --from-tarball-scp /home/tpcs/httpd.tar.gz --host 10.171.112.156 --username user1 --password pass123 --name my_scp_httpd

Install from sftp:

admin@sonic:~$ sudo sonic-package-manager install --from-tarball-sftp /home/tpcs/httpd.tar.gz --host 10.171.112.156 --username user1 --password pass123 --name my_sftp_httpd

Install from local FS:

admin@sonic:~$ sudo sonic-package-manager install --from-tarball /usb1/tpcs/httpd.tar.gz --name my_local_httpd

TPC Uninstallation

sonic-package-manager uninstall option shall be used to uninstall the TPC

Examples

admin@sonic:~$ sudo sonic-package-manager uninstall my_scp_httpd

TPC Resource limits

CPU Resource limit

As a policy, we will allocate only 20% of the CPU for TPCs.After installation, the default CPU values would be:

--cpu-period=100000 --cpu-quota=20000

These parameters are restricted to be passed as args to the TPC install.

Memory Resource limit

Default Memory limit for a TPC would be 512M

Examples

admin@sonic:~$ sudo sonic-package-manager manifests create my_node_exporter --tpcm --container memory 50m  
admin@sonic:~$ sudo sonic-package-manager install --from-repository node-exporter:latest --name my_node_exporter --use-local-manifest 

root@sonic:/home/admin# docker inspect my_node_exporter | grep Memory
            "Memory": **52428800**,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": **52428800**,
            "MemorySwappiness": null,

TPC Update CLI

Introduced a new option under sonic-package-manager CLI to support update of various parameters of manifest file for a given TPC.

Examples

Update the container section 'memory' limit and service section 'after' attribute

admin@sonic:~$ sudo sonic-package-manager manifests update my_node_exporter --tpcm --container memory 1000M --service after swss
admin@sonic:~$ sudo sonic-package-manager update my_node_exporter --use-local-manifest

TPC Upgrade CLI

No change in the sonic-package-manager CLI for upgrade which is 'sonic-package-manager install'

TPC Manifests CLI

Manifest Create

sonic-package-manager manifests create --tpcm --container --service

Examples

Create a default local tpcm manifest file

admin@sonic:~$ sudo sonic-package-manager manifests create my_httpd --tpcm

Create a local tpcm manifest file with container memory 1024M and privileged and flag set to true

admin@sonic:~$ sudo sonic-package-manager manifests create my_httpd --tpcm --container memory 1024M --container privileged true

Manifest Update

sonic-package-manager manifests update --container --service

Examples

Update a tpcm manifest file with cpu-quota 30000

admin@sonic:~$ sudo sonic-package-manager manifests update my_httpd --container cpu-quota 30000

Manifest show

sonic-package-manager manifests show

Examples

Display the contents of a local tpcm manifest file

admin@sonic:~$ sudo sonic-package-manager manifests show my_httpd

Manifest delete

sonic-package-manager manifests delete

Examples

Delete the local tpcm manifest file

admin@sonic:~$ sudo sonic-package-manager manifests delete my_httpd

Manifest list

sonic-package-manager manifests list

Examples

Display all the local tpcm manifest files from TPC manifest folder

admin@sonic:~$ sudo sonic-package-manager manifests list

Future Enhancements Proposal

Local custom Manfiest file could be used(created/updated) for SONiC packages as well.
TPC could be configured to start right after system becomes ready so that SONiC packages bootup wont be delayed.
Data preservation in TPCM migration
All manifest file properties could be updated.

Testing Requirements/Design

Unit Test cases

Installation test case:

Verify that TPCM packages can be installed successfully using the sonic package manager with a custom local manifest file.
Verify with SCP
Verify with SFTP
Verify with url

Uninstallation test case:

Verify that TPCM packages can be uninstalled using the sonic package manager without any issues.

Upgrade test case:

Verify that TPCM packages can be upgraded to newer versions using the sonic package manager without any issues.

Custom manifest file test case:

Verify that a custom manifest file can be created for TPCM packages using the sonic-package-manager manifests create command and used during TPCM installation.

Multiple TPC containers test case:

Verify that multiple TPC containers can be installed on the same system using the --name option and that they can be managed independently.

Manifest file update test case:

Verify that the local manifest file can be updated using the sonic-package-manager manifests update command and that the changes are reflected during TPCM installation.

Memory allocation test case:

Verify that the default memory allocation for TPC containers is set to 512M and that it can be adjusted using the manifest file.

CPU allocation test case:

Verify that the default CPU allocation for TPC containers is set to 20% and that it can be adjusted using the manifest file.

Error handling test case:

Verify that the sonic package manager can handle errors gracefully and provide meaningful error messages to users in case of failures during TPCM installation, upgrade, or uninstallation.

System Test cases

Verify that TPC containers can be installed and run alongside Sonic containers without conflicts or issues.
Verify that resource usage for TPC and Sonic containers is balanced and that they can coexist without resource contention.
Verify that TPC containers can be scaled up and down as needed to meet changing workload demands, and that Sonic containers can coexist with these changes

Files

tpcm_app_ext.md

Latest commit

History

tpcm_app_ext.md

File metadata and controls

Third Party Container Management Enhancements to SONiC Application Extensions framework

Rev 0.2

Table of Content

List of Tables

Revision

Scope

Definitions/Abbreviations

Table 1 Abbreviations

Document/References

Table 2 References

Overview

Requirements

Architecture Design

High-Level Design

Figure. SONiC TPCM Support

TPC Install

TPC Update

TPC Uninstall

TPC Upgrade

Resource Limits for TPC

CPU Resource

Memory Limits

SAI API

Configuration and management

CLI Enhancements

CLI

TPC Installation

Examples

TPC Uninstallation

Examples

TPC Resource limits

CPU Resource limit

Memory Resource limit

Examples

TPC Update CLI

Examples

TPC Upgrade CLI

TPC Manifests CLI

Manifest Create

Examples

Manifest Update

Examples

Manifest show

Examples

Manifest delete

Examples

Manifest list

Examples

Future Enhancements Proposal

Testing Requirements/Design

Unit Test cases

System Test cases