SCM

Self Managed Configuration Management(SCM) is a simple command line app for configuration management, written in python.

Designed to run only on ubuntu operating system

This tool, currently supports below resources

• service
• directory
• file
• firewall

A resource definition in scm is directly related to the action of standard linux commands, example. service, directories, files.

pre-requisites install python 3.6 & above environment on the machine

Installation

sudo apt-get update
sudo apt-get install python3.8 python3-pip -y
mkdir -p /root/scm
cd /root/scm
pip3 install scm-config
scm -v 

Usage

Usage: scm [OPTIONS] COMMAND [ARGS]...

Options:
  -v, --version                   Display''s the application version
  --install-completion [bash|zsh|fish|powershell|pwsh]
                                  Install completion for the specified shell.
  --show-completion [bash|zsh|fish|powershell|pwsh]
                                  Show completion for the specified shell, to
                                  copy it or customize the installation.
  --help                          Show this message and exit.

Commands:
  create
  diff
  info
  init
  push
  remove
  validate

scm creates two directories in the root directory during the "init" command, for example "/root/scm", one is for storing the receipe configuration and other one is for maintaining the configuration in the hash format, this mechanism provides the scm to run any number of times without worrying about the failures.

Example workflow for Apache+PHP configuration(After the installation)

• scm init
• scm create --receipe "apache"
• Copy the contents from the apache.toml to root/scm/config/apache.toml
• scm validate --receipe "apache"
• scm diff --receipe "apache"
• scm push --receipe "apache"

Commands overview

command	command description	usage	Example
init	Used for initialization, creates necessary files and directories for the tool to store configuration.	scm init
create	Creates the receipe .toml file in the config directory based on the receipe name	scm create --receipe	scm create --receipe apache
validate	Validates the receipe file configuration based on the standard defined resources	scm validate --receipe	scm validate --receipe apache
info	Lists the configurations that are defined in the receipe file	scm info --receipe	scm info --receipe apache
diff	Lists the differences between the existing and the current configuration, if this is this is new receipe, outputs all the configurations	scm diff --receipe	scm diff --receipe apache
push	Pushes the configuration defined in the receipe file to the operating system and stores the configuration in config_hash_directory	scm push --receipe	scm push --receipe apache
remove	Removes the configuration from the hash directory and also removes the receipe file from the config directory	scm push --receipe [optional --force [optional --clean-files	scm remove --receipe apache --force --clean-files

Resource Detailed information

Service

Service resource is useful for installing and managing packages from the linux repository,

Note that currently this tool is designed to support only on Ubuntu Operating System

*Name parameter in the below section is a list format, meaning it can take multiple values and the actions and other parameters are applied to each name resource accordingly

Example

[service.setup]
name = ["apache2"] 
action= ["install", "enable"]

In the above example, "service" is the resource and "setup" is the service identifier.

• name -> Name of the service that is to be installed on the ubuntu system
• action -> Action to be done on the listed service, currently these are only supported

["install", "enable", "disable"]

[service.ops]
name = ["apache2"]
action =["restart"]

In the above example, "service" is the resource and "ops" is the service identifier. once the service is installed and service operations can be done using this tool.

• name -> Name of the service that is to be installed on the ubuntu system
• action -> Action to be done on the listed service, currently these are only supported

["stop", "start", "restart", "reload", "disable", "enable"]

Directory

Directory resource is useful for installing and managing the metadata directories on the system,

Note that currently this resource is tested only on Ubuntu Operating System, but can run on any Linux Operating System

Example

# Modify the directory permissions 
[directory.list]
name = ["/etc/apache"] 
params = {'owner'='root','group'='root','mode'= '0755'}
action = ['create']
notifies = "@format {this.service.ops}"

In the above example, "directory" is the resource and "list" is the service identifier.

• name -> Name of the directory where the metadata needs modification including creation of the directory using the action method
• params -> parameter to support the directory operations, currently ['owner', 'group' and 'mode'] are supported in the list
• action -> Action to be done on the listed service, currently for directories only ['create'] is supported
• notifies -> Parameter to notify any other resource in the same recipe file, In example, notifies the service ops resources, that would restart the apache2 service based on the modifications

["install", "enable", "disable"]

File

File resource is useful for managing the metadata and the content on the linux file system,

Note that currently this resource is tested only on Ubuntu Operating System, but can run on any Linux Operating System

Example

# Modify the input content of the file 
[file.conf]
name = ["/var/www/customers/public_html/index.php"]
action = ["create"]
override ="true"
content  = ["This is the test file"]
params = {'owner'= 'root','group'= 'root','mode' = '0755'}
notifies = "@format {this.service.ops}"

In the above example, "file" is the resource and "conf" is the service identifier.

• name -> Name of the file where the content need to be added or appended based on the configuration requirement
• action -> Action to be done on the listed service, currently for file only ['create'] is supported
• override -> This parameter will override if there is any existing file, default it will append the content to the file
• content -> Input content for the file provide in the form the double quotes. For simplicity ,large content is not tested with the current version of the code.
• params -> parameter to support the file operations, currently ['owner', 'group' and 'mode'] are supported in the list
• notifies -> Parameter to notify any other resource in the same recipe file, In example, notifies the service ops resources, that would restart the apache2 service based on the modifications

["install", "enable", "disable"]

Firewall

Firewall resource is useful for managing firewall rules using UFW linux command,

Note that currently this resource is tested only on Ubuntu Operating System, but can run on any Linux Operating System

Example

# Modify the input content of the file 
[firewall.setup]
name = ["Apache"]
action=["allow"]

In the above example, "firewall" is the resource and "setup" is the service identifier.

• name -> Name of the resource where need to allow firewall traffic
• action -> Action to be done on the listed service, currently for file only ['allow'] is supported

["allow"]

Complete overview of the example file

# Service setup for the apache instance 
[service.setup]
name = ["apache2"] 
action= ["install", "enable"]

# Service restart for the apache instance 
[service.ops]
name = ["apache2"]
action =["restart"]

[firewall.setup]
name = ["Apache"]
action=["allow"]

# Modify the directory permissions 
[directory.list]
name = ["/etc/apache"] 
params = {'owner'='root','group'='root','mode'= '0755'}
action = ['create']
notifies = "@format {this.service.ops}"

# Modify the input content of the file 
[file.conf]
name = ["/var/www/customers/public_html/index.php"]
action = ["create"]
content  = ["This is the test file"]
params = {'owner'= 'root','group'= 'root','mode' = '0755'}
notifies = "@format {this.service.ops}"

Installation and Usage

Manual clone

  $ git clone https://github.com/Sai-Repalle/scm_project
  $ cd scm_project
  $ python3 -m venv venv 
  $ source venv/bin/activate
  $ pip install -r requirements.txt 

initialization

init command is used for initialization and helpful to create the respective directories, also this command is useful for expanding future versions of the scm tool

init Initialize the tool without any parameters, this would set the configuration files required for tool to work

$ scm init

output

scm init
[INFO][04-22-2022 06:08:13]::Reading the Json configuration /root/scm/scm/settings/settings.json
[INFO][04-22-2022 06:08:13]::creating directory CONFIG_DIR
[INFO][04-22-2022 06:08:13]::creating directory CONFIG_HASH_DIR
[INFO][04-22-2022 06:08:13]::creating files CONFIG_DEF_FILE
[INFO][04-22-2022 06:08:13]::creating files CONFIG_HASH_FILE

create

create <name>

$ scm create --receipe <receipe>

Below example, will create a receipe called "apache" and apache.toml file is created in the config directory located at the root directory of the scm

output

scm create --receipe apache
[INFO][04-22-2022 06:09:56]::Reading the Json configuration /root/scm/scm/settings/settings.json
[INFO][04-22-2022 06:09:56]::creating directory CONFIG_DIR
[INFO][04-22-2022 06:09:56]::CONFIG_DIR directory already exists
[INFO][04-22-2022 06:09:56]::creating directory CONFIG_HASH_DIR
[INFO][04-22-2022 06:09:56]::CONFIG_HASH_DIR directory already exists
[INFO][04-22-2022 06:09:56]::creating files CONFIG_DEF_FILE
[INFO][04-22-2022 06:09:56]::creating files CONFIG_HASH_FIL

info

info --receipe <name>

$ scm info --receipe <receipe>

Below example, info command is listing all the resources and its actions based on the configuration defined in the receipe file.

output

scm info --receipe apache
[INFO][04-25-2022 02:00:37]::Reading the Json configuration /root/scm/settings.json
[INFO][04-25-2022 02:00:37]::creating directory CONFIG_DIR
[INFO][04-25-2022 02:00:37]::CONFIG_DIR directory already exists
[INFO][04-25-2022 02:00:37]::creating directory CONFIG_HASH_DIR
[INFO][04-25-2022 02:00:37]::CONFIG_HASH_DIR directory already exists
[INFO][04-25-2022 02:00:37]::creating files CONFIG_DEF_FILE
[INFO][04-25-2022 02:00:37]::creating files CONFIG_HASH_FILE
[INFO][04-25-2022 02:00:37]::SERVICE.setup - {'name': ['apache2'], 'action': ['install', 'enable', 'start']}
[INFO][04-25-2022 02:00:37]::SERVICE.ops - {'name': ['apache2'], 'action': ['restart']}
[INFO][04-25-2022 02:00:37]::SERVICE.setup_php - {'name': ['php', 'libapache2-mod-php'], 'action': ['install']}
[INFO][04-25-2022 02:00:37]::FIREWALL.setup - {'name': ['Apache'], 'action': ['allow']}
[INFO][04-25-2022 02:00:37]::FILE.conf - {'name': ['/var/www/html/index.php'], 'action': ['create'], 'override': 'true', 'content': ['<?php\n header("Content-Type: text/plain");\n echo "Hello, world!\n";\n'], 'params': {'owner': 'root', 'group': 'root', 'mode': '0755'}, 'notifies': "{'name': ['apache2'], 'action': ['restart']}"}
[INFO][04-25-2022 02:00:37]::FILE.phpindex - {'name': ['/etc/apache2/mods-enabled/dir.conf'], 'action': ['create'], 'override': 'true', 'content': ['<IfModule mod_dir.c>\n DirectoryIndex index.php index.html index.cgi index.pl index.xhtml index.htm </IfModule>'], 'params': {'owner': 'root', 'group': 'root', 'mode': '0755'}, 'notifies': "{'name': ['apache2'], 'action': ['restart']}"}

validate

validate -receipe <name>

$ scm validate --receipe <receipe>

Below example, validate command is validating all the resources and its actions based on the configuration defined in the receipe file, If all the resources and its configurations are in valid state, receipe would output to do next step, if there are any issues, please refere to the documentation.

output

scm validate --receipe apache
[INFO][04-25-2022 02:01:47]::Reading the Json configuration /root/scm/settings.json
[INFO][04-25-2022 02:01:47]::creating directory CONFIG_DIR
[INFO][04-25-2022 02:01:47]::CONFIG_DIR directory already exists
[INFO][04-25-2022 02:01:47]::creating directory CONFIG_HASH_DIR
[INFO][04-25-2022 02:01:47]::CONFIG_HASH_DIR directory already exists
[INFO][04-25-2022 02:01:47]::creating files CONFIG_DEF_FILE
[INFO][04-25-2022 02:01:47]::creating files CONFIG_HASH_FILE
[INFO][04-25-2022 02:01:47]::apache receipe file is valid for push, use `scm diff` to differences with the existing configuration

diff

diff -receipe <name>

$ scm diff --receipe <receipe>

In the below example, if the confiugration in the receipe file is already appplied, scm would output that the configuration is update to date with the existing configuration

[INFO][04-25-2022 02:05:00]::Reading the Json configuration /root/scm/settings.json
[INFO][04-25-2022 02:05:00]::Reading the Json configuration /root/scm/settings.json
[INFO][04-25-2022 02:05:00]::creating directory CONFIG_DIR
[INFO][04-25-2022 02:05:00]::CONFIG_DIR directory already exists
[INFO][04-25-2022 02:05:00]::creating directory CONFIG_HASH_DIR
[INFO][04-25-2022 02:05:00]::CONFIG_HASH_DIR directory already exists
[INFO][04-25-2022 02:05:00]::creating files CONFIG_DEF_FILE
[INFO][04-25-2022 02:05:00]::creating files CONFIG_HASH_FILE
[INFO][04-25-2022 02:05:00]::apache receipe file is valid for push, use `scm diff` to differences with the existing configuration
[INFO][04-25-2022 02:05:00]::Reading the Json configuration config_hash/hash_config_md5.json
[INFO][04-25-2022 02:05:00]::`apache` configuration is update to date with the existing configuration

In the below example, if the configuration in the receipe file is pushed to the operating system to install the required services

root@ip-172-31-255-140:~/scm/config# scm diff --receipe apache_remove
[INFO][04-25-2022 02:20:37]::Reading the Json configuration /root/scm/config/settings.json
[INFO][04-25-2022 02:20:37]::Reading the Json configuration /root/scm/config/settings.json
[INFO][04-25-2022 02:20:37]::creating directory CONFIG_DIR
[INFO][04-25-2022 02:20:37]::CONFIG_DIR directory already exists
[INFO][04-25-2022 02:20:37]::creating directory CONFIG_HASH_DIR
[INFO][04-25-2022 02:20:37]::CONFIG_HASH_DIR directory already exists
[INFO][04-25-2022 02:20:37]::creating files CONFIG_DEF_FILE
[INFO][04-25-2022 02:20:37]::CONFIG_DEF_FILE file already exists
[INFO][04-25-2022 02:20:37]::creating files CONFIG_HASH_FILE
[INFO][04-25-2022 02:20:37]::apache_remove receipe file is valid for push, use `scm diff` to differences with the existing configuration
[INFO][04-25-2022 02:20:37]::Reading the Json configuration config_hash/hash_config_md5.json
[INFO][04-25-2022 02:20:37]::Following resources will be applied:
[INFO][04-25-2022 02:20:37]::SERVICE.setup: ['systemctl stop apache2', 'systemctl disable apache2', 'sudo apt-get update -y', 'sudo apt-get remove apache2 -y']
[INFO][04-25-2022 02:20:37]::Following resources will be applied:
[INFO][04-25-2022 02:20:37]::SERVICE.setup_php: ['sudo apt-get update -y', 'sudo apt-get remove php -y', 'sudo apt-get update -y', 'sudo apt-get remove libapache2-mod-php -y']

push

push -receipe <name>

$ scm push --receipe <receipe>

In the below example, scm push is pushing hte receipe named "apache_remove" and the subsequent commands are being run on the operating to the remove the services

output

root@ip-172-31-255-140:~/scm/config# scm push  --receipe apache_remove
[INFO][04-25-2022 02:21:34]::Reading the Json configuration /root/scm/config/settings.json
[INFO][04-25-2022 02:21:34]::Reading the Json configuration /root/scm/config/settings.json
[INFO][04-25-2022 02:21:34]::creating directory CONFIG_DIR
[INFO][04-25-2022 02:21:34]::CONFIG_DIR directory already exists
[INFO][04-25-2022 02:21:34]::creating directory CONFIG_HASH_DIR
[INFO][04-25-2022 02:21:34]::CONFIG_HASH_DIR directory already exists
[INFO][04-25-2022 02:21:34]::creating files CONFIG_DEF_FILE
[INFO][04-25-2022 02:21:34]::CONFIG_DEF_FILE file already exists
[INFO][04-25-2022 02:21:34]::creating files CONFIG_HASH_FILE
[INFO][04-25-2022 02:21:34]::apache_remove receipe file is valid for push, use `scm diff` to differences with the existing configuration
[INFO][04-25-2022 02:21:34]::Reading the Json configuration config_hash/hash_config_md5.json
[INFO][04-25-2022 02:21:34]::Reading the Json configuration /root/scm/config/settings.json
[INFO][04-25-2022 02:21:34]::Following resources will be applied:
[INFO][04-25-2022 02:21:34]::SERVICE.setup: ['systemctl stop apache2', 'systemctl disable apache2', 'sudo apt-get update -y', 'sudo apt-get remove apache2 -y']
[INFO][04-25-2022 02:21:34]::Applying the command `systemctl stop apache2`
[INFO][04-25-2022 02:21:35]::Applying the command `systemctl disable apache2`
Synchronizing state of apache2.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable apache2
[INFO][04-25-2022 02:21:35]::Applying the command `sudo apt-get update -y`
Hit:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:4 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Fetched 252 kB in 0s (666 kB/s)
Reading package lists... Done
[INFO][04-25-2022 02:21:42]::Applying the command `sudo apt-get remove apache2 -y`
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  apache2-data apache2-utils ssl-cert
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  apache2
0 upgraded, 0 newly installed, 1 to remove and 2 not upgraded.
After this operation, 537 kB disk space will be freed.
(Reading database ... 94525 files and directories currently installed.)
Removing apache2 (2.4.29-1ubuntu4.22) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ufw (0.36-0ubuntu0.18.04.2) ...
Rules updated for profile 'Apache'

remove

remove --receipe <name> --force <optional> --clean_files <optional>

$ scm push --receipe <receipe>

In the below example, scm is removing the apache_remove receipe from the configuration hash directory,

output

scm remove --receipe apache_remove
[INFO][04-25-2022 02:33:38]::Reading the Json configuration /root/scm/config/settings.json
[INFO][04-25-2022 02:33:38]::Reading the Json configuration /root/scm/config/settings.json
[INFO][04-25-2022 02:33:38]::creating directory CONFIG_DIR
[INFO][04-25-2022 02:33:38]::CONFIG_DIR directory already exists
[INFO][04-25-2022 02:33:38]::creating directory CONFIG_HASH_DIR
[INFO][04-25-2022 02:33:38]::CONFIG_HASH_DIR directory already exists
[INFO][04-25-2022 02:33:38]::creating files CONFIG_DEF_FILE
[INFO][04-25-2022 02:33:38]::CONFIG_DEF_FILE file already exists
[INFO][04-25-2022 02:33:38]::creating files CONFIG_HASH_FILE
[INFO][04-25-2022 02:33:38]::apache_remove receipe file is valid for push, use `scm diff` to differences with the existing configuration
[INFO][04-25-2022 02:33:38]::Configuration doesn't remove the receipe file, please clean up manually