Added techsupport data export manager and core file manager #468

Kalimuthu-Velappan · 2019-09-16T06:53:50Z

The techsupport data management functionality is divided into two main components.

Core-dump manger.
Tech-support data export manager.

1. Core-dump manager:

Added a new SONiC service named 'coredumpctl.service' to manage the core dump as follows: -

1. Support per-process core file rotation 
2. Archiving the corefile to optimize the disk space
3. Strip Core files sensitive information

2. Tech-support data export manager

Added a new SONIC service named 'export.service' to collect and export the tech-support data to a remote server for better offline debugging. The tech-support data is captured and exported under the following conditions:-

1. Collect tech-support data when a new core dump is discovered, and export it to a remote server.
2. Periodically collect and export tech-support data

Code PRs

sonic-net/sonic-buildimage#3447
sonic-net/sonic-utilities#643

renukamanavalan · 2019-09-18T23:41:20Z

doc/techsupport-manager/core_file_manager.md

+
+	 a.  Support per-process core file rotation and archiving to optimize disk space 
+
+ 	 b.  Strip Core files sensitive information 


Which module does this? Debian service or ...?
Can you please share some details on, how it works?
Is there a way to enable/disable it?

New service named 'coredumpctl.service' is added to monitor the per process core file rotation and archive managment. Through systemctl command, this can be enabled/disabled.

Can this service support user to enable core on few processes instead all at once ? Basically can we collect core per process?

renukamanavalan · 2019-09-18T23:42:27Z

doc/techsupport-manager/core_file_manager.md

+
+ 2.  Add a new SONIC service to collect and export the tech-support data as follows:-
+
+	 a.  Collect tech-support data when a new core dump is discovered, and export it to a remote server.


How big is the tech support data collected?
Can we register a app/plugin to customize the data collected?

The techsupport data size is arount 4MB. The 'generate_dump' script can be used for extending the custom data

renukamanavalan · 2019-09-18T23:44:42Z

doc/techsupport-manager/core_file_manager.md

+### Config commands
+
+>1. Config command to enable/disable the coredump generation of processes.
+>2.  Config command to store the details of exporting tech-support data to an external server which includes remote server name, path, transfer protocol type and the user credentials.


Can we register an external script to be invoked for uploading ?

This way we can extend the ability to store anywhere. For example, azure-cloud-storage

No external script. It uses the standard protocol scp/sftp to update the techsupport data to remote server.

Sure. But I think, accepting a script (with full path) as part of remote server configuration (that takes server address, user, pass,...) will make this transparently extendable. You already have the FW, so adding this can be simple, I believe.

Side note:
You don't imply "coredump generation of processes", but imply "coredump upload", correct.

renukamanavalan · 2019-09-18T23:46:02Z

doc/techsupport-manager/core_file_manager.md

+
+>1. Config command to enable/disable the coredump generation of processes.
+>2.  Config command to store the details of exporting tech-support data to an external server which includes remote server name, path, transfer protocol type and the user credentials.
+>2. Config command to enable/disable the tech-support export


To clarify:
Does disable implies "uploading core file only w/o accompanying tech support data" ?

Disable implies whole techsupport data export and not the core file generation.

Please help clarify:
Both core file & tech-support data goes to the same server. Correct?
Does the core file gets packaged inside tech-support data ?

renukamanavalan · 2019-09-18T23:50:38Z

doc/techsupport-manager/core_file_manager.md

+### Config commands
+
+>1. Config command to enable/disable the coredump generation of processes.
+>2.  Config command to store the details of exporting tech-support data to an external server which includes remote server name, path, transfer protocol type and the user credentials.


When a process cores a lot, the core-rotation would help keep the local count of core files to set value, say N.
To clarify:
What about uploading, will it upload every core file, implying that the remote server could end up getting way more than N, if that process keeps crashing.

It uploads the current snapshot of techsupport data

renukamanavalan · 2019-09-18T23:56:02Z

doc/techsupport-manager/core_file_manager.md

+
+# Tech-support export service 
+
+The tech-support data is a vital information for debugging of a system and is captured by collecting the  device configuration, system information, log files  and core files. The export service captures the tech-support data and export it to a remote server for better offline debugging.  The tech-support data is captured and exported under the following conditions 


Is this part of systemd-coredump service?
Will we be able to provide custom script to generate tech-support data?

Export service is separate service. The standard "generate_dump" script is used for generating the techsupport data

renukamanavalan · 2019-09-18T23:58:21Z

doc/techsupport-manager/core_file_manager.md

+![Tech Support export Service](images/corefilemgr.png)
+
+
+The export service is configured to monitors the coredump path for any new core file creation. Upon detection of a new core file, it triggers the tech-support data collection and export it to a remote server.   In addition, export service can be configured to capture and upload the tech-support data periodically. 


Is the export service is part of systemd-coredump service?

No, it is a separate service.

itaibaz · 2019-10-14T06:28:09Z

doc/techsupport-manager/core_file_manager.md

+![Tech Support export Service](images/corefilemgr.png)
+
+
+The export service is configured to monitors the coredump path for any new core file creation. Upon detection of a new core file, it triggers the tech-support data collection and export it to a remote server.   In addition, export service can be configured to capture and upload the tech-support data periodically. 


Can we also add to the tech support a text file with the backtrace of the crash ?
This is a quick way to see the backtrace, instead of having to open the full core and look for the correct binary version symbols
It can be generated by "gdb thread apply all bt full" on the core, to a file

Yes, It can be added.

renukamanavalan · 2019-10-15T00:34:04Z

doc/techsupport-manager/core_file_manager.md

+
+The [systemd-coredump](https://www.freedesktop.org/software/systemd/man/systemd-coredump.html) is a native systemd tool that is available in Debian o/s version 9 (stretch) and above. This tool provides an array of features to manage application core files. When it is installed, as part of base configuration it provides following functionality:
+
+1.  Configures kernel to dump a core when application performs unexpected exit. The process ID, UID, GID, signal received, time of termination, command name of the terminated process are collected. The core dump generate may be affected by ulimit settings. Care should be taken that ulimit settings do not conflict with systemd-coredump configuration.


Is there a way to add few more data to record maintained per core, like SONiC Version?

Can be added

renukamanavalan · 2019-10-15T00:41:32Z

doc/techsupport-manager/core_file_manager.md

+
+
+The export service is configured to monitors the coredump path for any new core file creation. Upon detection of a new core file, it triggers the tech-support data collection and export it to a remote server.   In addition, export service can be configured to capture and upload the tech-support data periodically. 
+


What is the cost of this feature, with respect to 'size of image' ?
Will there be a possibility to build a image w/o this feature ?
More for the sake, if this feature would be expensive.

Around 4MB. By default this service is disabled.

renukamanavalan · 2019-10-15T00:43:55Z

doc/techsupport-manager/core_file_manager.md

+
+### Config DB Schema
+
+In order to export the tech support data, remote server details have to be configured on the device. Through CLI interface, external storage server can be configured which includes server IP, path and access information like user credentials and transport protocol. This information is stored as part of config DB.


Thought that the tech-support-data accompanies the core file. If so, this remote server config is for both core & tech-support-data? Is that correct?

Remote server config is for techsupport data which includes core file as well.

Added techsupport data export manager and core file manager

71f0da1

This was referenced Sep 16, 2019

Exportmgr sonic-net/sonic-buildimage#3447

Open

Tech-support export manager sonic-net/sonic-utilities#643

Open

Kalimuthu-Velappan marked this pull request as ready for review September 16, 2019 11:15

lguohan assigned renukamanavalan Sep 16, 2019

renukamanavalan reviewed Sep 18, 2019

View reviewed changes

This was referenced Sep 20, 2019

Added 'config core' and 'show cores' CLI commands sonic-net/sonic-utilities#663

Open

SONiC core dump utility sonic-net/sonic-buildimage#3499

Open

itaibaz reviewed Oct 14, 2019

View reviewed changes

renukamanavalan reviewed Oct 15, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added techsupport data export manager and core file manager #468

Added techsupport data export manager and core file manager #468

Kalimuthu-Velappan commented Sep 16, 2019 •

edited

renukamanavalan Sep 18, 2019

Kalimuthu-Velappan Oct 15, 2019

dev-aviz Oct 16, 2019

renukamanavalan Sep 18, 2019

Kalimuthu-Velappan Oct 15, 2019

renukamanavalan Sep 18, 2019

Kalimuthu-Velappan Oct 15, 2019

renukamanavalan Oct 15, 2019

renukamanavalan Oct 15, 2019

renukamanavalan Sep 18, 2019

Kalimuthu-Velappan Oct 15, 2019

renukamanavalan Oct 15, 2019

renukamanavalan Sep 18, 2019

Kalimuthu-Velappan Oct 15, 2019

renukamanavalan Sep 18, 2019

Kalimuthu-Velappan Oct 15, 2019

renukamanavalan Sep 18, 2019

Kalimuthu-Velappan Oct 15, 2019

itaibaz Oct 14, 2019

Kalimuthu-Velappan Oct 15, 2019

renukamanavalan Oct 15, 2019

Kalimuthu-Velappan Oct 15, 2019

renukamanavalan Oct 15, 2019

Kalimuthu-Velappan Oct 15, 2019

renukamanavalan Oct 15, 2019

Kalimuthu-Velappan Oct 15, 2019


		a. Support per-process core file rotation and archiving to optimize disk space

		b. Strip Core files sensitive information


		2. Add a new SONIC service to collect and export the tech-support data as follows:-

		a. Collect tech-support data when a new core dump is discovered, and export it to a remote server.


		# Tech-support export service

		The tech-support data is a vital information for debugging of a system and is captured by collecting the device configuration, system information, log files and core files. The export service captures the tech-support data and export it to a remote server for better offline debugging. The tech-support data is captured and exported under the following conditions

		![Tech Support export Service](images/corefilemgr.png)


		The export service is configured to monitors the coredump path for any new core file creation. Upon detection of a new core file, it triggers the tech-support data collection and export it to a remote server. In addition, export service can be configured to capture and upload the tech-support data periodically.


		The [systemd-coredump](https://www.freedesktop.org/software/systemd/man/systemd-coredump.html) is a native systemd tool that is available in Debian o/s version 9 (stretch) and above. This tool provides an array of features to manage application core files. When it is installed, as part of base configuration it provides following functionality:

		1. Configures kernel to dump a core when application performs unexpected exit. The process ID, UID, GID, signal received, time of termination, command name of the terminated process are collected. The core dump generate may be affected by ulimit settings. Care should be taken that ulimit settings do not conflict with systemd-coredump configuration.


		### Config DB Schema

		In order to export the tech support data, remote server details have to be configured on the device. Through CLI interface, external storage server can be configured which includes server IP, path and access information like user credentials and transport protocol. This information is stored as part of config DB.

Added techsupport data export manager and core file manager #468

Are you sure you want to change the base?

Added techsupport data export manager and core file manager #468

Conversation

Kalimuthu-Velappan commented Sep 16, 2019 • edited

1. Core-dump manager:

2. Tech-support data export manager

Code PRs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kalimuthu-Velappan commented Sep 16, 2019 •

edited