Converting old or creating new buffers config

Scope

This is a how-to document describing how to convert or create new configuration for port buffer pools and profiles.

Purpose

This document will be useful for migrating existing buffers configurations to the Configuration DB format or creating buffers configuration for a new platform

Converting existing JSONs to the new format

New JSON format

Old json configuration files were consumed by the swssconfig utility and saved to the Application DB(#1 in Redis DB). New jsons are processed by the sonic-cfggen and saved to the Configuration DB(#4 in Redis DB).

Old JSON example:

[
    {
        "BUFFER_POOL_TABLE:egress_lossless_pool0": {
            "size": "3637248",
            "type": "egress",
            "mode": "static"
        },
        "OP": "SET"
    },
    {
        "BUFFER_POOL_TABLE:egress_lossless_pool1": {
            "size": "3637248",
            "type": "egress",
            "mode": "static"
        },
        "OP": "SET"
    },
    {
        "BUFFER_POOL_TABLE:egress_lossy_pool": {
            "size": "5491712",
            "type": "egress",
            "mode": "dynamic"
        },
        "OP": "SET"
    },
    {
        "BUFFER_PROFILE_TABLE:ingress_lossless_profile0": {
            "pool": "[BUFFER_POOL_TABLE:ingress_lossless_pool0]",
            "xon": "18432",
            "xoff": "76800",
            "size": "113664",
            "static_th": "0"
        },
        "OP": "SET"
    },
    {
        "BUFFER_PROFILE_TABLE:ingress_lossless_profile1": {
            "pool":"[BUFFER_POOL_TABLE:ingress_lossless_pool1]",
            "xon":"18432",
            "xoff":"76800",
            "size":"113664",
            "static_th": "0"
        },
        "OP": "SET"
    }
]

Same sample converted to the new JSON format:

{
    "BUFFER_POOL": {
        "egress_lossless_pool0": {
            "size": "3637248",
            "type": "egress",
            "mode": "static"
        },
        "egress_lossless_pool1": {
            "size": "3637248",
            "type": "egress",
            "mode": "static"
        },
        "egress_lossy_pool": {
            "size": "5491712",
            "type": "egress",
            "mode": "dynamic"
        }
    },
    "BUFFER_PROFILE": {
        "ingress_lossless_profile0": {
            "pool": "[BUFFER_POOL_TABLE:ingress_lossless_pool0]",
            "xon": "18432",
            "xoff": "76800",
            "size": "113664",
            "static_th": "0"
        },
        "ingress_lossless_profile1": {
            "pool":"[BUFFER_POOL_TABLE:ingress_lossless_pool1]",
            "xon":"18432",
            "xoff":"76800",
            "size":"113664",
            "static_th": "0"
        }
    }
}

Key changes to be done:

• remove suffix "_TABLE" from the table names
• remove operations fields (i.e. "OP":"SET" or "OP":"DEL")
• tables combined into one block
• change outer square brackets to curly brackets

Configuration files location and application

Old config files

Old configuration files stored in the swss(orchagent) docker container. Config files and templates locations in swss docker:

• configs: /etc/swss/config.d/*.json
• templates: /usr/share/sonic/templates/*.j2

Configuration files generated and applied during executions of scripts /usr/bin/swssconfig.sh and /usr/bin/start.sh. To apply data from these configs swssconfig utility is used. E.g.

swssconfig some_config.json

These configs applied (some - generated and applied) every time when swss image starts.

Location in sources tree: [sonic-buildimage/src/]sonic-swss/swssconfig/sample/td2.32ports.buffers.json

New config files

New config files are stored in

/usr/share/sonic/device/$platform/$HWSKU/qos.json

where $platform is the platform name (e.g. x86_64-mlnx_msn2700-r0, see sonic-buildimage/device/ for more) and $HWSKU is the exact hardware name (e.g. ACS-MSN2700) In sources tree the full path to the QoS config is:

sonic-buildimage/device/mellanox/x86_64-mlnx_msn2700-r0/ACS-MSN2700/qos.json

These config files are processed on the first switch start and stored to the Configuration DB (this may be changed in the future).

Script which applies configs to the DB and generates them from templates if needed is /etc/rc.local (in sources tree: sonic-buildimage/files/image_config/platform/rc.local)

Buffers configuration from the scratch

When new platform/hw is added to SONiC, to create buffers configuration for it, the following tables need to be added to the corresponding buffers.json.j2: BUFFER_POOL - describes memory pools BUFFER_PROFILE - describes buffer profiles to be used on interface BUFFER_PG - links interface name with the buffer profile for given priority group(s) BUFFER_QUEUE - links interface name with the buffer profile for given queue(s) BUFFER_PORT_INGRESS_PROFILE_LIST - (optional?) specifies the list of profiles used for ingress traffic BUFFER_PORT_EGRESS_PROFILE_LIST - (optional?) specifies the list of profiles used for egress traffic

Check one of the existing configs and detailed tables schema on the SONiC wiki.

Switch buffers configuration can be completely static or have variable profiles for PG lossless queues.

Static config

Static config should describe all the tables described in the chapter above.

Variable config

If switch uses variable profiles config, BUFFER_PG tables for lossless queues can be omitted in buffers.json.j2. They will be generated in run time.

Variable buffers config feature helps to adjust the interface buffer parameters depending on speed and connected cable length. This is achieved by switching buffer profile in the interface when it speed changes.
In order to achieve this, two more pieces of configuration is needed:

• cable length - the length of the cable connected to the corresponding port.
  This information comes from the table CABLE_LENGTH. It can be statically defined in the buffers configuration file (buffers.json.j2) or generated basing on the switch role and its neighbors roles - ToR, Leaf, Spine. (see sonic-buildimage/device/mellanox/x86_64-mlnx_msn2700-r0/ACS-MSN2700/buffers.json.j2)
• profile look-up table which links interface speed and cable length to the profile settings to be used for given speed and cable

Cable length table example:

{
    "CABLE_LENGTH": {
        "AZURE": {
            "Ethernet0": "5m",
            "Ethernet4": "40m",
            "Ethernet8": "300m",
            ...
        }
    }
}

Profiles look-up table example (sonic-buildimage/device/mellanox/x86_64-mlnx_msn2700-r0/ACS-MSN2700/pg_profile_lookup.ini):

# PG lossless profiles.
# speed cable size    xon  xoff threshold
  10000  5m   34816  18432 16384  0
  25000  5m   34816  18432 16384  0
  40000  5m   34816  18432 16384  0
  50000  5m   34816  18432 16384  0
 100000  5m   36864  18432 18432  0
  10000  40m  36864  18432 18432  0
  25000  40m  39936  18432 21504  0
  40000  40m  41984  18432 23552  0
  50000  40m  41984  18432 23552  0
 100000  40m  54272  18432 35840  0
  10000  300m 49152  18432 30720  0
  25000  300m 71680  18432 53248  0
  40000  300m 94208  18432 75776  0
  50000  300m 94208  18432 75776  0
 100000  300m 184320 18432 165888 0

Currently only PG lossless profiles switching is supported.

How variable config works in run-time

See Run Time Buffers Configuration Design Document. If needed more details will be added here.