Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixes #15023: Adapt system techniques to HTTPS reporting #1451

Merged
merged 2 commits into from
Jul 15, 2019
Merged

Fixes #15023: Adapt system techniques to HTTPS reporting #1451

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c56e889
Fixes #15023: Adapt system techniques to HTTPS reporting
amousset Jun 5, 2019
ef8e915
Fixes #15023: Adapt system techniques to HTTPS reporting
amousset Jun 14, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions techniques/system/common/1.0/metadata.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,9 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
<FILE name="properties.cf">
<INCLUDED>true</INCLUDED>
</FILE>
<FILE name="reporting-http.cf">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could it be reporting-https.cf ?

<INCLUDED>true</INCLUDED>
</FILE>
<FILE name="rudder-agent-community-cron">
<OUTPATH>common/cron/rudder-agent-community-cron</OUTPATH>
<INCLUDED>false</INCLUDED>
Expand Down Expand Up @@ -125,6 +128,7 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
<NAME>RUDDER_SYSLOG_PROTOCOL</NAME>
<NAME>RUDDER_NODE_GROUPS_VARS</NAME>
<NAME>RUDDER_NODE_GROUPS_CLASSES</NAME>
<NAME>REPORTING_PROTOCOL</NAME>
</SYSTEMVARS>

<SECTIONS>
Expand Down
47 changes: 17 additions & 30 deletions techniques/system/common/1.0/promises.st
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,21 +31,21 @@ bundle common rudder_roles
classes:

# on changes only, we only reports on repaired or error message
"changes_only_mode" expression => strcmp("&RUDDER_REPORT_MODE&", "changes-only");
"reports_disabled_mode" expression => strcmp("&RUDDER_REPORT_MODE&", "reports-disabled");
"changes_only" expression => "changes_only_mode.!force_full_compliance";
"reports_disabled" expression => "reports_disabled_mode.!force_full_compliance";
"changes_only_mode" expression => strcmp("&RUDDER_REPORT_MODE&", "changes-only");
"reports_disabled_mode" expression => strcmp("&RUDDER_REPORT_MODE&", "reports-disabled");
"changes_only" expression => "changes_only_mode.!force_full_compliance";
"reports_disabled" expression => "reports_disabled_mode.!force_full_compliance";
"rudder_reporting_https" expression => strcmp("&REPORTING_PROTOCOL&", "HTTPS");

# full compliance is the default mode
"full_compliance" not => "changes_only|reports_disabled";
"rudder_reporting_syslog" not => "rudder_reporting_https";
}

body common control
{
any::
output_prefix => "rudder";

protocol_version => "2";
protocol_version => "2";

inputs => {
@{va.inputs_list},
Expand Down Expand Up @@ -157,7 +157,7 @@ bundle common va
"common_input_list" slist => { "common/1.0/common.cf", "common/1.0/cf-serverd.cf", "common/1.0/rudder-groups.cf", "common/1.0/hooks.cf", "common/1.0/cron-setup.cf", "common/1.0/site.cf", "common/1.0/update.cf", "common/1.0/monitoring.cf", "common/1.0/restart-services.cf", "common/1.0/internal-security.cf", "common/1.0/environment-variables.cf", "common/1.0/properties.cf" };
"inventory_input_list" slist => { "inventory/1.0/fusionAgent.cf" };
"distribute_policy_input_list" slist => { "distributePolicy/1.0/common.cf", "distributePolicy/1.0/rsyslogConf.cf", "distributePolicy/1.0/propagatePromises.cf", "distributePolicy/1.0/apache-acl.cf" };
"server_roles_input_list" slist => { "server-roles/1.0/common.cf", "server-roles/1.0/component-check.cf", "server-roles/1.0/alive-check.cf", "server-roles/1.0/service-check.cf", "server-roles/1.0/integrity-check.cf", "server-roles/1.0/network-check.cf", "server-roles/1.0/password-check.cf", "server-roles/1.0/postgres-check.cf", "server-roles/1.0/logrotate-check.cf", "server-roles/1.0/technique-reload.cf", "server-roles/1.0/compress-webapp-log.cf", "server-roles/1.0/compress-ldap-backups.cf", "server-roles/1.0/servers-by-role.cf" };
"server_roles_input_list" slist => { "server-roles/1.0/common.cf", "server-roles/1.0/relayd.cf", "server-roles/1.0/component-check.cf", "server-roles/1.0/alive-check.cf", "server-roles/1.0/service-check.cf", "server-roles/1.0/integrity-check.cf", "server-roles/1.0/network-check.cf", "server-roles/1.0/password-check.cf", "server-roles/1.0/postgres-check.cf", "server-roles/1.0/logrotate-check.cf", "server-roles/1.0/technique-reload.cf", "server-roles/1.0/compress-webapp-log.cf", "server-roles/1.0/compress-ldap-backups.cf", "server-roles/1.0/servers-by-role.cf" };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is server-roles/1.0/relayd.cf included now ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because it was missing from a previous PR


policy_server::
"inputs_list" slist => { @{ncf_inputs}, @{common_input_list}, @{inventory_input_list}, @{distribute_policy_input_list}, @{server_roles_input_list} };
Expand All @@ -179,7 +179,6 @@ bundle common va
"policy_server" expression => strcmp("root","${rudder_roles.uuid}");
# Root Server is the top policy server machine
"root_server" expression => strcmp("root","${rudder_roles.uuid}");

# static definition of the machine roles
&NODEROLE&

Expand Down Expand Up @@ -262,29 +261,29 @@ bundle agent initialize_ncf()
"report about ncf init" usebundle => rudder_common_reports_generic("Common", "ncf_init", "${system_common.directiveId}", "ncf Initialization", "None", "Configuration library initialization");
}

# Log and reporting configuration not valid in initial promises
&if(!INITIAL)&
bundle agent configure_rudder_reporting_system {
methods:
!reports_disabled::
rudder_reporting_syslog.!reports_disabled::
"configure_syslog" usebundle => check_log_system;
"configure_reporting" usebundle => check_rsyslog_version;
reports_disabled::

(rudder_reporting_syslog.reports_disabled)|rudder_reporting_https::
"remove_reporting" usebundle => remove_rudder_syslog_configuration;

rudder_reporting_https::
"make http reports" usebundle => send_rudder_reports;
}

#######################################################
# Check the log system, and configure it accordingly
# This only works with UNIX flavoured systems
#
# Note: if any configurations are changed here, corresponding services must be
# restarted in the restart_services bundle

bundle agent check_log_system
{

vars:

debian::
"syslog_ng_source" string => "s_src";

Expand All @@ -304,12 +303,11 @@ bundle agent check_log_system
rsyslogd_conffile_present.rsyslog_version_5_6_4_ok::

"rsyslogd_message_reduction" string => "$RepeatedMsgReduction off${const.n}";

rsyslogd_conffile_present.rsyslog_version_5_6_4_not_ok::

"rsyslogd_message_reduction" string => "";


any::

"syslog_conf_comment" string => "# Autogenerated by rudder, do not edit${const.n}";
Expand Down Expand Up @@ -402,7 +400,7 @@ bundle agent check_log_system
"any" usebundle => rudder_common_report("Common", "result_success", "${system_common.directiveId}", "Log system for reports", "None", "Logging system for report centralization is already correctly configured");

commands:

rsyslogd_conffile_present.check_rsyslog_version_present::
# 5.6.4 is the first version mentionning RepeatedMsgReduction in http://www.rsyslog.com/change-log/
# The option is probably older though.
Expand Down Expand Up @@ -471,9 +469,7 @@ bundle agent check_rsyslog_version {
# restarted in the restart_services bundle

bundle agent remove_rudder_syslog_configuration {

vars:

"syslogng_delete_lines_patterns" slist => {
"\s*# Autogenerated by rudder, do not edit.*",
".*destination\(rudder_loghost\);.*",
Expand All @@ -485,7 +481,6 @@ bundle agent remove_rudder_syslog_configuration {
};

classes:

"syslogng" expression => fileexists("${check_log_system.syslogng_conffile}");
"syslogd" expression => fileexists("${check_log_system.syslog_conffile}");

Expand Down Expand Up @@ -658,7 +653,7 @@ body executor control
schedule => { &AGENT_RUN_SCHEDULE& };
executorfacility => "LOG_DAEMON";

exec_command => "${g.rudder_command} agent run -uR";
exec_command => "${g.rudder_command} agent run -uRN";
}

########################################################
Expand Down Expand Up @@ -691,7 +686,6 @@ bundle edit_line fix_syslogd(syslogd)

}


bundle edit_line edit_syslog_conf_file(line_to_add, pattern_to_remove)
{

Expand All @@ -706,26 +700,19 @@ bundle edit_line edit_syslog_conf_file(line_to_add, pattern_to_remove)
# This was used in Rudder pre-3.0. We need to keep this in all versions that support upgrading from pre-3.0 versions (up to and including 2.11)
# See http://www.rudder-project.org/redmine/issues/5773
"filter\ f\_local\_rudder\{facility\(local6\)\ and\ program\(\"rudder\.\*\"\)\;\}\;destination\ loghost\ \{(tcp|udp)\(\"[^\"]+\".*";

insert_lines:
"${line_to_add}"
location => syslogng_log_part;

}

body location syslogng_log_part
{

select_line_matching => "^\s*log\s*\{.*";
before_after => "before";
first_last => "first";

}

bundle edit_line ensure_rsyslogd_on_suse
{
field_edits:

# match a line starting like 'SYSLOG_DAEMON=something'
"^SYSLOG_DAEMON=.*$"
edit_field => col("=","2","\"rsyslogd\"","set"),
Expand Down
57 changes: 57 additions & 0 deletions techniques/system/common/1.0/reporting-http.cf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
#####################################################################################
# Copyright 2019 Normation SAS
#####################################################################################
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, Version 3.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
#####################################################################################

# TODO purge too old reports

bundle agent send_rudder_report(path) {
vars:
"endpoint" string => "https://${server_info.policy_server}/reports/";
"command" string => "${g.rudder_curl} --tlsv1.2 --location --insecure --fail --silent --proxy '' --user ${g.davuser}:${g.davpw} --upload-file";

files:
"${path}"
delete => tidy,
if => "send_report_${path}_ok",
classes => classes_generic("tidy_report_${path}");

methods:
"any" usebundle => rudder_common_report("Common", "log_warn", "${system_common.directiveId}", "Log system for reports", "None", "Could not send report ${path}"),
if => "send_report_${path}_error";
# May be a another done by another agent run
"any" usebundle => rudder_common_report("Common", "log_warn", "${system_common.directiveId}", "Log system for reports", "None", "Could not remove report ${path}"),
if => "tidy_report_${path}_error";

commands:
"${command} ${path} ${endpoint}"
classes => classes_generic_two("send_report_${path}", "send_rudder_reports");
}

bundle agent send_rudder_reports
{
vars:
"raw_reports" slist => lsdir("${rudder_reports}/ready/", ".*", "false");
# Max 50 reports by run to avoid blocking the agent too long after a long
# disconnection
"reports" slist => sublist("reports", "head", 50);

methods:
"send" usebundle => send_rudder_report("${reports}");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you may want to use @{reports} (not sure if it's relevant, but could be helpful)

"cleanup" usebundle => _classes_sanitize("send_rudder_reports");
"report" usebundle => rudder_common_reports_generic("Common", "send_rudder_reports", "${system_common.directiveId}", "Log system for reports", "None", "Reports forwarding to policy server");
}

2 changes: 1 addition & 1 deletion techniques/system/common/1.0/restart-services.cf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ bundle agent restart_services

# Ensure at least one syslog is running
# We cannot detect which one is used for Rudder
!aix.!solaris::
rudder_reporting_syslog.!aix.!solaris::
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't it dealt with in the calling bundle ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could rename the bundle to restart_syslog and only call it when syslog is used for reporting indeed, good idea.

"run_syslog" usebundle => disable_reporting;
"run_syslog" usebundle => _method_reporting_context("Make sure syslog service runs", "None");
"run_syslog" usebundle => service_ensure_running("syslog"),
Expand Down
1 change: 1 addition & 0 deletions techniques/system/common/1.0/site.cf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ bundle common g
"rudder_var_tmp" string => "${rudder_var}/tmp"; # tmp generated data
"rudder_base_sbin" string => "${rudder_base}/sbin"; #folder where tools are installed
"rudder_inventories" string => "${rudder_var}/inventories";
"rudder_reports" string => "${rudder_var}/reports";
"rudder_disable_agent_file" string => "${rudder_base}/etc/disable-agent";
"rudder_disable_server_file" string => "${rudder_base}/etc/disable-policy-distribution";
"rudder_disable_inventory_file" string => "${rudder_base}/etc/disable-inventory";
Expand Down
29 changes: 17 additions & 12 deletions techniques/system/distributePolicy/1.0/rsyslogConf.cf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,23 @@
#
#####################################################################################

bundle agent rudder_postgresql {
root_server|role_rudder_relay_top::
"db_name" string => "${distribute_policy_common.report_db_name}";
"db_user" string => "${distribute_policy_common.report_db_user}";
"db_pass" string => "${p.psql_password[2]}";

# Define the where to send the syslog messages
# default is localhost if the rudder_db is unknown
# or if the role rudder-server-root is installed

root_server.(!role_rudder_db_server_defined|role_rudder_server_root)::
"host" string => "localhost";

(root_server|role_rudder_relay_top).role_rudder_db_server_defined.!role_rudder_server_root::
"host" string => "${rudder_servers_by_role.roles[rudder-db]}";
}

# Configure rsyslog on the root/relay servers

bundle agent install_rsyslogd
Expand All @@ -31,18 +48,6 @@ bundle agent install_rsyslogd

root_server|role_rudder_relay_top::
"rsyslog_source_file" string => "rudder-rsyslog-root.conf";
"rudder_reports_db_name" string => "${distribute_policy_common.report_db_name}";
"rudder_reports_db_user" string => "${distribute_policy_common.report_db_user}";

# Define the where to send the syslog messages
# default is localhost if the rudder_db is unknown
# or if the role rudder-server-root is installed

root_server.(!role_rudder_db_server_defined|role_rudder_server_root)::
"rudder_postgres_server" string => "localhost";

(root_server|role_rudder_relay_top).role_rudder_db_server_defined.!role_rudder_server_root::
"rudder_postgres_server" string => "${rudder_servers_by_role.roles[rudder-db]}";

policy_server.!(root_server|role_rudder_relay_top)::
"rsyslog_source_file" string => "rudder-rsyslog-relay.conf";
Expand Down
2 changes: 1 addition & 1 deletion techniques/system/distributePolicy/1.0/rudder-rsyslog-root.st
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ $template RudderReportsFormat,"insert into RudderSysEvents (executionDate, nodeI
# The report format is @@Policy@@State@@RuleId@@DirectiveId@@0@@Component@@Key@@ExecutionTimeStamp##NodeId@#HumanReadableMessage
#
# 1 - Send every matching report in the database...
:msg, ereregex, "(R: )?@@[ a-zA-Z0-9_\-]+?@@[a-zA-Z0-9_\-]+?@@[a-zA-Z0-9\-]+@@[a-zA-Z0-9\-]+?@@0@@.*?@@.*?@@[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}[+-][0-9]{1,2}:[0-9]{2}##[a-zA-Z0-9\-]+?@#.*" :ompgsql:${install_rsyslogd.rudder_postgres_server},${install_rsyslogd.rudder_reports_db_name},${install_rsyslogd.rudder_reports_db_user},${p.psql_password[2]};RudderReportsFormat
:msg, ereregex, "(R: )?@@[ a-zA-Z0-9_\-]+?@@[a-zA-Z0-9_\-]+?@@[a-zA-Z0-9\-]+@@[a-zA-Z0-9\-]+?@@0@@.*?@@.*?@@[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}[+-][0-9]{1,2}:[0-9]{2}##[a-zA-Z0-9\-]+?@#.*" :ompgsql:${rudder_postgresql.host},${rudder_postgresql.db_name},${rudder_postgresql.db_user},${rudder_postgresql.db_pass};RudderReportsFormat

# 2 - Drop the remaining rudder logs to prevent local storage cluttering

Expand Down
2 changes: 1 addition & 1 deletion techniques/system/inventory/1.0/fusionAgent.st
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
# If fusion agent is not installed,
# the installation is done
#
# Then the inventory file is improved with external informations (UUID, cf-key, VMs)
# Then the inventory file is improved with external information (UUID, cf-key, VMs)

# Compute the inventory time
bundle agent computeInventoryTime
Expand Down
4 changes: 4 additions & 0 deletions techniques/system/server-roles/1.0/metadata.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,10 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
<FILE name="relayd.cf">
<INCLUDED>true</INCLUDED>
</FILE>
<FILE name="relayd.conf.tpl">
<OUTPATH>server-roles/1.0/relayd.conf.tpl</OUTPATH>
<INCLUDED>false</INCLUDED>
</FILE>
<FILE name="rudder-logrotate">
<OUTPATH>server-roles/logrotate.conf/rudder</OUTPATH>
<INCLUDED>false</INCLUDED>
Expand Down
8 changes: 4 additions & 4 deletions techniques/system/server-roles/1.0/password-check.cf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,11 +77,11 @@ bundle agent root_password_check_file

vars:

"pgpass[1]" string => "${install_rsyslogd.rudder_postgres_server}";
"pgpass[1]" string => "${rudder_postgresql.host}";
"pgpass[2]" string => "5432";
"pgpass[3]" string => "${install_rsyslogd.rudder_reports_db_user}";
"pgpass[4]" string => "${install_rsyslogd.rudder_reports_db_name}";
"pgpass[5]" string => "${p.psql_password[2]}";
"pgpass[3]" string => "${rudder_postgresql.db_user}";
"pgpass[4]" string => "${rudder_postgresql.db_name}";
"pgpass[5]" string => "${rudder_postgresql.db_pass}";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

<3


files:

Expand Down
31 changes: 6 additions & 25 deletions techniques/system/server-roles/1.0/relayd.cf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,38 +18,19 @@

bundle agent rudder_relayd_configuration {
vars:
"config_file" string => "/opt/rudder/etc/rudder-relayd.conf";
"service" string => "rudder-relayd";
"component" string => "Relayd service configuration";

"file_class_prefix" string => canonify("file_key_value_present_in_ini_section_${config_file}");
"config_file" string => "/opt/rudder/etc/relayd/main.conf";
"service" string => "rudder-relayd";
"component" string => "Relayd service configuration";
"file_class_prefix" string => canonify("file_from_template_${config_file}");

methods:
"any" usebundle => disable_reporting;
"any" usebundle => _method_reporting_context("${component}", "None");

"any" usebundle => file_key_value_present_in_ini_section("${config_file}", "general", "nodes_list_file", "\"${g.rudder_base}/etc/nodeslist.json\"");
"any" usebundle => file_key_value_present_in_ini_section("${config_file}", "general", "node_id", "\"${g.uuid}\"");
"any" usebundle => file_key_value_present_in_ini_section("${config_file}", "processing.inventory", "directory", "\"/var/rudder/inventories\"");
"any" usebundle => file_key_value_present_in_ini_section("${config_file}", "processing.reporting", "directory", "\"/var/rudder/reports\"");

root_server::
"any" usebundle => file_key_value_present_in_ini_section("${config_file}", "processing.inventory", "output", "\"disabled\"");
"any" usebundle => file_key_value_present_in_ini_section("${config_file}", "processing.reporting", "output", "\"database\"");
"any" usebundle => file_key_value_present_in_ini_section("${config_file}", "output.database", "url", "\"postgres://rudder:${p.psql_password[2]}@localhost/rudder\"");

policy_server.!root_server::
"any" usebundle => file_key_value_present_in_ini_section("${config_file}", "processing.inventory", "output", "\"upstream\"");
"any" usebundle => file_key_value_present_in_ini_section("${config_file}", "processing.reporting", "output", "\"upstream\"");
"any" usebundle => file_key_value_present_in_ini_section("${config_file}", "output.upstream", "url", "\"https://${server_info.policy_server}\"");
"any" usebundle => file_key_value_present_in_ini_section("${config_file}", "output.upstream", "user", "\"${g.davuser}\"");
"any" usebundle => file_key_value_present_in_ini_section("${config_file}", "output.upstream", "password", "\"${g.davpw}\"");

any::
"any" usebundle => file_from_template_mustache("${this.promise_dir}/relayd.conf.tpl", "${config_file}");
"any" usebundle => service_restart("rudder-relayd"),
if => "${file_class_prefix}_repaired";

"any" usebundle => enable_reporting;

"any" usebundle => rudder_common_reports_generic("server-roles", "${file_class_prefix}", "${server_roles_common.directiveId}", "${component}", "None", "rudder-relayd configration");
"any" usebundle => rudder_common_report("server-roles", "log_repaired", "${server_roles_common.directiveId}", "${component}", "None", "rudder-relayd service has been restarted"),
if => "service_restart_rudder_relayd_repaired";
Expand Down
Loading