Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

syslog changes Multi NPU platforms #4738

Merged
merged 8 commits into from
Jun 30, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions files/build_templates/docker_image_ctl.j2
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,22 @@ link_namespace() {
}
{%- endif %}

function updateSyslogConf()
{
# On multiNPU platforms, change the syslog target ip to docker0 ip to allow logs from containers
# running on the namespace to reach the rsyslog service running on the host
# Also update the container name
if [[ ($NUM_ASIC -gt 1) ]]; then
TARGET_IP=$(docker network inspect bridge --format={{ "'{{(index .IPAM.Config 0).Gateway}}'" }})
CONTAINER_NAME="{{docker_container_name}}$DEV"
TMP_FILE="/tmp/rsyslog.$CONTAINER_NAME.conf"
Copy link
Collaborator

@qiluo-msft qiluo-msft Jun 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TMP_FILE="/tmp/rsyslog.$CONTAINER_NAME.conf" [](start = 8, length = 44)

Could you use Dockerfile ENV and docker create --env to pass the container_name. Then no need to create tmpfile. #Pending

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tmp file is created after rendering this template /usr/share/sonic/templates/rsyslog-container.conf.j2, which has updated syslog.conf file. I don't think the tmp file can be avoided even if we pass the ENV to the container.

In Multi-NPU platforms it is possible that multi containers can be started at the same time, to avoid container using wrong syslog.conf files I am using the container name in the tmp filename

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's create two ENV:

  1. RSYSLOG_SERVER_IP
  2. CONTAINER_NAME

Pass the ENV values by command docker create ... --env RSYSLOG_SERVER_IP=AA.BB.CC.DD ...

The benefit:

  1. not tmp file management
  2. works with multiple containers from single image

In reply to: 439583041 [](ancestors = 439583041)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@qiluo-msft created a issue to track 4778 to improvements suggested in this comment.


sonic-cfggen -t /usr/share/sonic/templates/rsyslog-container.conf.j2 -a "{\"target_ip\": \"$TARGET_IP\", \"container_name\": \"$CONTAINER_NAME\" }" > $TMP_FILE
docker cp $TMP_FILE {{docker_container_name}}$DEV:/etc/rsyslog.conf
rm -rf $TMP_FILE
fi
}

function getMountPoint()
{
echo $1 | python -c "import sys, json, os; mnts = [x for x in json.load(sys.stdin)[0]['Mounts'] if x['Destination'] == '/usr/share/sonic/hwsku']; print '' if len(mnts) == 0 else os.path.basename(mnts[0]['Source'])" 2>/dev/null
Expand Down Expand Up @@ -68,6 +84,7 @@ function preStartAction()
{%- else %}
: # nothing
{%- endif %}
updateSyslogConf
}

function postStartAction()
Expand Down
1 change: 1 addition & 0 deletions files/build_templates/sonic_debian_extension.j2
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,7 @@ echo "warmboot-finalizer.service" | sudo tee -a $GENERATED_SERVICE_FILE
sudo cp $IMAGE_CONFIGS/rsyslog/rsyslog-config.service $FILESYSTEM_ROOT_USR_LIB_SYSTEMD_SYSTEM
sudo cp $IMAGE_CONFIGS/rsyslog/rsyslog-config.sh $FILESYSTEM_ROOT/usr/bin/
sudo cp $IMAGE_CONFIGS/rsyslog/rsyslog.conf.j2 $FILESYSTEM_ROOT_USR_SHARE_SONIC_TEMPLATES/
sudo cp $IMAGE_CONFIGS/rsyslog/rsyslog-container.conf.j2 $FILESYSTEM_ROOT_USR_SHARE_SONIC_TEMPLATES/
sudo cp $IMAGE_CONFIGS/rsyslog/rsyslog.d/* $FILESYSTEM_ROOT/etc/rsyslog.d/
echo "rsyslog-config.service" | sudo tee -a $GENERATED_SERVICE_FILE

Expand Down
21 changes: 20 additions & 1 deletion files/image_config/rsyslog/rsyslog-config.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,23 @@
#!/bin/bash

sonic-cfggen -d -t /usr/share/sonic/templates/rsyslog.conf.j2 >/etc/rsyslog.conf
PLATFORM=`sonic-cfggen -H -v DEVICE_METADATA.localhost.platform`

# Parse the device specific asic conf file, if it exists
ASIC_CONF=/usr/share/sonic/device/$PLATFORM/asic.conf
if [ -f "$ASIC_CONF" ]; then
source $ASIC_CONF
fi

# On Multi NPU platforms we need to start the rsyslog server on the docker0 ip address
# for the syslogs from the containers in the namespaces to work.
# on Single NPU platforms we continue to use loopback adddres

if [[ ($NUM_ASIC -gt 1) ]]; then
udp_server_ip=$(ip -o -4 addr list docker0 | awk '{print $4}' | cut -d/ -f1)
else
udp_server_ip=$(ip -o -4 addr list lo scope host | awk '{print $4}' | cut -d/ -f1)
fi

sonic-cfggen -d -t /usr/share/sonic/templates/rsyslog.conf.j2 -a "{\"udp_server_ip\": \"$udp_server_ip\"}" >/etc/rsyslog.conf

systemctl restart rsyslog
76 changes: 76 additions & 0 deletions files/image_config/rsyslog/rsyslog-container.conf.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
#
# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html


#################
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging

#
# Set a rate limit on messages from the container
#
$SystemLogRateLimitInterval 300
$SystemLogRateLimitBurst 20000

#$ModLoad imklog # provides kernel logging support
#$ModLoad immark # provides --MARK-- message capability

# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


###########################
#### GLOBAL DIRECTIVES ####
###########################

# Set remote syslog server
template (name="ForwardFormatInContainer" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% {{container_name}}#%syslogtag%%msg:::sp-if-no-1st-sp%%msg%")
*.* action(type="omfwd" target="{{target_ip}}" port="514" protocol="udp" Template="ForwardFormatInContainer")

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Define a custom template
$template SONiCFileFormat,"%TIMESTAMP%.%timestamp:::date-subseconds% %HOSTNAME% %syslogseverity-text:::uppercase% {{container_name}}#%syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate SONiCFileFormat

#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

#
# Suppress duplicate messages and report "message repeated n times"
#
$RepeatedMsgReduction on

###############
#### RULES ####
###############
2 changes: 1 addition & 1 deletion files/image_config/rsyslog/rsyslog.conf.j2
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ $ModLoad imklog # provides kernel logging support

# provides UDP syslog reception
$ModLoad imudp
$UDPServerAddress 127.0.0.1 # bind to localhost before udp server run
$UDPServerAddress {{udp_server_ip}} #bind to localhost before udp server run
$UDPServerRun 514

# provides TCP syslog reception
Expand Down