Investigate and enhance external modules logging messages

[fdalmaup]

Affected integration
AWS, GCloud, Azure

Description

As mentioned in #14535, the debug mode (wazuh_modules.debug=2) is disabled by default, therefore when not enabled, the logging of the different modules does not give enough information if there are warning or error messages, making harder the troubleshooting.

For example, in case of a Throttling Exception in the AWS module, the user would only see the debug message if the debug level is 2 due to the following section of code:

wazuh/wodles/aws/aws_s3.py

Lines 1059 to 1064 in 78bbd9f

	except botocore.exceptions.ClientError as err:
	if err.response['Error']['Code'] == 'ThrottlingException':
	debug('Error: The "iter_files_in_bucket" request was denied due to request throttling. If the problem '
	'persists check the following link to learn how to use the Retry configuration to avoid it: '
	f'{RETRY_CONFIGURATION_URL}', 2)
	sys.exit(16)

Those types of errors are difficult to reproduce because they depend on several variables such as environment congestion, number of requests to the service, etc.

Another example is regarding the different allowed regions for the AWS module. In some AWS services, in case a non-existent region is used in the module's execution, there is no warning message:

Fake region defined in Config bucket

/var/ossec/wodles/aws/aws-s3 --bucket wazuh-aws-wodle-config --aws_profile dev --only_logs_after 2022-NOV-20 --type config --debug 2 --region us-fake
DEBUG: +++ Debug mode on - Level: 2
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: +++ Table does not exist; create
DEBUG: +++ Working on 123456789123 - us-fake
DEBUG: +++ Marker: AWSLogs/123456789123/Config/us-fake/2022/11/20
DEBUG: +++ No logs to process in bucket: 123456789123/us-fake
DEBUG: +++ DB Maintenance

We should review the different messages and debug levels used in order to make error troubleshooting simpler so that the solution can be carried out quickly. As well, if it is a module error itself, its reproduction should be easier to carry out.

Tasks

• Investigate and determine the minimum necessary logging for the modules' troubleshooting.
• Test in a manager.
• Test in an agent.
• Unit tests without failures. Updated if there are any relevant changes.
• Integration tests without failures. Updated if there are any relevant changes.
• Update the documentation if necessary.
• Add entry to the changelog if necessary.

[RamosFe]

Update

Started to read Azure wodle and Azure SDK for checking which errors are not warned by the logging.

[RamosFe]

Update - Azure Wodle

Azure Log Analytics

Error handling

• Errors related to configuration credentials are already handled:
  

  wazuh/wodles/azure/azure-logs.py

  Lines 294 to 302 in e5d9fd4

  	if args.la_auth_path and args.la_tenant_domain:
  	client, secret = read_auth_file(auth_path=args.la_auth_path, fields=("application_id", "application_key"))
  	elif args.la_id and args.la_key and args.la_tenant_domain:
  	logging.warning(DEPRECATED_MESSAGE.format(name="la_id and la_key", release="4.4", url=CREDENTIALS_URL))
  	client = args.la_id
  	secret = args.la_key
  	else:
  	logging.error("Log Analytics: No parameters have been provided for authentication.")
  	sys.exit(1)

• Errors related to tokens are already handled:
  

  wazuh/wodles/azure/azure-logs.py

  Lines 799 to 814 in e5d9fd4

  	try:
  	token_response = post(auth_url, data=body).json()
  	return token_response['access_token']
  	except (ValueError, KeyError):
  	if token_response['error'] == 'unauthorized_client':
  	err_msg = "The application id provided is not valid."
  	elif token_response['error'] == 'invalid_client':
  	err_msg = "The application key provided is not valid."
  	elif token_response['error'] == 'invalid_request' and 90002 in token_response['error_codes']:
  	err_msg = f"The '{domain}' tenant domain was not found."
  	else:
  	err_msg = "Couldn't get the token for authentication."
  	logging.error(f"Error: {err_msg}")
  	except RequestException as e:
  	logging.error(f"Error: An error occurred while trying to obtain the authentication token: {e}")

• There should be logged the contents of the error response before the raise_for_status:
  

  wazuh/wodles/azure/azure-logs.py

  Line 413 in e5d9fd4

  response.raise_for_status()

  Minimum fields to be logged:

  • Status Code (code field)
  • Error msg (message field)
  • Details (details field)
    Documentation: https://learn.microsoft.com/en-us/rest/api/loganalytics/dataaccess/query/get?source=recommendations&tabs=HTTP#errorinfo

Debug

• The information of the query should only be logged if the debug flag is activated:
  

  wazuh/wodles/azure/azure-logs.py

  Line 370 in e5d9fd4

  logging.info(f"Log Analytics: The search starts for query: '{query}'")

• The URL, header and body content should be logged if the debug flag is activated:
  

  wazuh/wodles/azure/azure-logs.py

  Lines 393 to 394 in e5d9fd4

  	logging.info("Log Analytics: Sending a request to the Log Analytics API.")
  	response = get(url, params=body, headers=headers)

• The information send to the socket should be logged if the debug flag is activated:
  

  wazuh/wodles/azure/azure-logs.py

  Lines 456 to 459 in e5d9fd4

  	event = {}
  	for c in range(0, len(columns)):
  	event[columns[c]['name']] = row[c]
  	logging.info("Log Analytics: Sending event by socket.")

Azure Graph

Error handling

• Errors related to configuration credentials are already handled:
  

  wazuh/wodles/azure/azure-logs.py

  Lines 470 to 479 in e5d9fd4

  	# Read credentials
  	if args.graph_auth_path and args.graph_tenant_domain:
  	client, secret = read_auth_file(auth_path=args.graph_auth_path, fields=("application_id", "application_key"))
  	elif args.graph_id and args.graph_key and args.graph_tenant_domain:
  	logging.warning(DEPRECATED_MESSAGE.format(name="graph_id and graph_key", release="4.4", url=CREDENTIALS_URL))
  	client = args.graph_id
  	secret = args.graph_key
  	else:
  	logging.error("Graph: No parameters have been provided for authentication.")
  	sys.exit(1)

• Errors related to tokens are already handled:
  

  wazuh/wodles/azure/azure-logs.py

  Lines 799 to 814 in e5d9fd4

  	try:
  	token_response = post(auth_url, data=body).json()
  	return token_response['access_token']
  	except (ValueError, KeyError):
  	if token_response['error'] == 'unauthorized_client':
  	err_msg = "The application id provided is not valid."
  	elif token_response['error'] == 'invalid_client':
  	err_msg = "The application key provided is not valid."
  	elif token_response['error'] == 'invalid_request' and 90002 in token_response['error_codes']:
  	err_msg = f"The '{domain}' tenant domain was not found."
  	else:
  	err_msg = "Couldn't get the token for authentication."
  	logging.error(f"Error: {err_msg}")
  	except RequestException as e:
  	logging.error(f"Error: An error occurred while trying to obtain the authentication token: {e}")

• There should be logged the contents of the error response before the raise_for_status:
  

  wazuh/wodles/azure/azure-logs.py

  Lines 590 to 594 in e5d9fd4

  	elif response.status_code == 400:
  	logging.error(f"Bad Request for url: {response.url}")
  	logging.error(f"Ensure the URL is valid and there is data available for the specified datetime.")
  	else:
  	response.raise_for_status()

  Minimum fields to be logged:

  • Status Code (code field)
  • Error msg (message field)
  • Details (details field)
    Documentation: https://learn.microsoft.com/en-us/rest/api/loganalytics/dataaccess/query/get?source=recommendations&tabs=HTTP#errorinfo

Debug

• The URL and headers should be logged if the debug flag is activated:
  

  wazuh/wodles/azure/azure-logs.py

  Line 566 in e5d9fd4

  response = get(url=url, headers=headers)

• The response of the request should be logged if the debug flag is activated:
  

  wazuh/wodles/azure/azure-logs.py

  Line 566 in e5d9fd4

  response = get(url=url, headers=headers)

• There should be logged when it iterates to the next resource using the field next_url:
  

  wazuh/wodles/azure/azure-logs.py

  Line 588 in e5d9fd4

  if next_url:

Azure Storage

Error handling

• Errors related to configuration credentials are already handled:
  

  wazuh/wodles/azure/azure-logs.py

  Lines 603 to 613 in e5d9fd4

  	# Read credentials
  	logging.info("Storage: Authenticating.")
  	if args.storage_auth_path:
  	name, key = read_auth_file(auth_path=args.storage_auth_path, fields=("account_name", "account_key"))
  	elif args.account_name and args.account_key:
  	logging.warning(DEPRECATED_MESSAGE.format(name="account_name and account_key", release="4.4", url=CREDENTIALS_URL))
  	name = args.account_name
  	key = args.account_key
  	else:
  	logging.error("Storage: No parameters have been provided for authentication.")
  	sys.exit(1)

• Errors related to tokens are already handled:
  

  wazuh/wodles/azure/azure-logs.py

  Lines 799 to 814 in e5d9fd4

  	try:
  	token_response = post(auth_url, data=body).json()
  	return token_response['access_token']
  	except (ValueError, KeyError):
  	if token_response['error'] == 'unauthorized_client':
  	err_msg = "The application id provided is not valid."
  	elif token_response['error'] == 'invalid_client':
  	err_msg = "The application key provided is not valid."
  	elif token_response['error'] == 'invalid_request' and 90002 in token_response['error_codes']:
  	err_msg = f"The '{domain}' tenant domain was not found."
  	else:
  	err_msg = "Couldn't get the token for authentication."
  	logging.error(f"Error: {err_msg}")
  	except RequestException as e:
  	logging.error(f"Error: An error occurred while trying to obtain the authentication token: {e}")

• Errors related to container existence/permission handled but it should be logged the msg property from AzureException:
  

  wazuh/wodles/azure/azure-logs.py

  Lines 622 to 640 in e5d9fd4

  	if args.container != '*':
  	try:
  	if not block_blob_service.exists(args.container):
  	logging.error(f"Storage: The '{args.container}' container does not exists.")
  	sys.exit(1)
  	containers = [args.container]
  	except AzureException:
  	logging.error(f"Storage: Invalid credentials for accessing the '{args.container}' container.")
  	sys.exit(1)
  	else:
  	try:
  	logging.info("Storage: Getting containers.")
  	containers = [container.name for container in block_blob_service.list_containers()]
  	except AzureSigningError:
  	logging.error("Storage: Unable to list the containers. Invalid credentials.")
  	sys.exit(1)
  	except AzureException as e:
  	logging.error(f"Storage: The containers could not be listed: '{e}'.")
  	sys.exit(1)

• Errors related to reading from a blob handled:
  

  wazuh/wodles/azure/azure-logs.py

  Lines 724 to 725 in e5d9fd4

  	except (ValueError, AzureException, AzureHttpError) as e:
  	logging.error(f"Storage: Error reading the blob data: '{e}'.")

Debug

• It should log the containers if the debug flag is activated:
  

  wazuh/wodles/azure/azure-logs.py

  Line 634 in e5d9fd4

  containers = [container.name for container in block_blob_service.list_containers()]

• It should log the data send to the socket if the debug flag is activated:
  

  wazuh/wodles/azure/azure-logs.py

  Lines 745 to 746 in e5d9fd4

  	logging.info("Storage: Sending event by socket.")
  	send_message(dumps(log_record))

• There should be logged when it iterates to the next resource using the field next_marker:
  

  wazuh/wodles/azure/azure-logs.py

  Line 767 in e5d9fd4

  if blobs.next_marker:

[RamosFe]

Update

AWS Wodle

Info

• Should be info:
  

  wazuh/wodles/aws/aws_s3.py

  Line 3659 in e5d9fd4

  debug("+++ Warning: No regions were specified, trying to get events from all regions", 1)

  
  

  wazuh/wodles/aws/aws_s3.py

  Line 3665 in e5d9fd4

  debug('+++ Getting alerts from "{}" region.'.format(region), 1)

  
  

  wazuh/wodles/aws/aws_s3.py

  Lines 2912 to 2915 in e5d9fd4

  	if self.sent_events:
  	debug(f"+++ {self.sent_events} events collected and processed in {self.inspector_region}", 1)
  	else:
  	debug(f'+++ There are no new events in the "{self.inspector_region}" region', 1)

Debug

• Should log the initialized class if the debug option is set:
  

  wazuh/wodles/aws/aws_s3.py

  Lines 3595 to 3618 in e5d9fd4

  	try:
  	if options.logBucket:
  	if options.type.lower() == 'cloudtrail':
  	bucket_type = AWSCloudTrailBucket
  	elif options.type.lower() == 'vpcflow':
  	bucket_type = AWSVPCFlowBucket
  	elif options.type.lower() == 'config':
  	bucket_type = AWSConfigBucket
  	elif options.type.lower() == 'custom':
  	bucket_type = AWSCustomBucket
  	elif options.type.lower() == 'guardduty':
  	bucket_type = AWSGuardDutyBucket
  	elif options.type.lower() == 'cisco_umbrella':
  	bucket_type = CiscoUmbrella
  	elif options.type.lower() == 'waf':
  	bucket_type = AWSWAFBucket
  	elif options.type.lower() == 'alb':
  	bucket_type = AWSALBBucket
  	elif options.type.lower() == 'clb':
  	bucket_type = AWSCLBBucket
  	elif options.type.lower() == 'nlb':
  	bucket_type = AWSNLBBucket
  	elif options.type.lower() == 'server_access':
  	bucket_type = AWSServerAccess

• Should log the initialized class if the debug option is set:
  

  wazuh/wodles/aws/aws_s3.py

  Lines 3643 to 3647 in e5d9fd4

  	elif options.service:
  	if options.service.lower() == 'inspector':
  	service_type = AWSInspector
  	elif options.service.lower() == 'cloudwatchlogs':
  	service_type = AWSCloudWatchLogs

Error Handling

• Should log the passed LogBucket option:
  

  wazuh/wodles/aws/aws_s3.py

  Lines 3619 to 3620 in e5d9fd4

  	else:
  	raise Exception("Invalid type of bucket")

• Should log the error code of the request:
  

  wazuh/wodles/aws/aws_s3.py

  Lines 1088 to 1107 in e5d9fd4

  	except botocore.exceptions.ClientError as error:
  	error_message = "Unknown"
  	exit_number = 1
  	error_code = error.response.get("Error", {}).get("Code")
  	if error_code == THROTTLING_EXCEPTION_ERROR_CODE:
  	error_message = f"{THROTTLING_EXCEPTION_ERROR_MESSAGE.format(name='check_bucket')}: {error}"
  	exit_number = 16
  	elif error_code == INVALID_CREDENTIALS_ERROR_CODE:
  	error_message = INVALID_CREDENTIALS_ERROR_MESSAGE
  	exit_number = 3
  	elif error_code == INVALID_REQUEST_TIME_ERROR_CODE:
  	error_message = INVALID_REQUEST_TIME_ERROR_MESSAGE
  	exit_number = 19
  	print(f"ERROR: {error_message}")
  	exit(exit_number)
  	except botocore.exceptions.EndpointConnectionError as e:
  	print(f"ERROR: {str(e)}")
  	exit(15)

• Should log and raise an error if an invalid region is passed:
  

  wazuh/wodles/aws/aws_s3.py

  Lines 950 to 963 in e5d9fd4

  	def iter_regions_and_accounts(self, account_id, regions):
  	if not account_id:
  	# No accounts provided, so find which exist in s3 bucket
  	account_id = self.find_account_ids()
  	for aws_account_id in account_id:
  	# No regions provided, so find which exist for this AWS account
  	if not regions:
  	regions = self.find_regions(aws_account_id)
  	if not regions:
  	continue
  	for aws_region in regions:
  	debug("+++ Working on {} - {}".format(aws_account_id, aws_region), 1)
  	self.iter_files_in_bucket(aws_account_id, aws_region)
  	self.db_maintenance(aws_account_id=aws_account_id, aws_region=aws_region)

• There should not be debug and should be error (It should be logged even if the debug flag is not activated):
  

  wazuh/wodles/aws/aws_s3.py

  Lines 1058 to 1074 in e5d9fd4

  	except botocore.exceptions.ClientError as err:
  	if err.response['Error']['Code'] == 'ThrottlingException':
  	debug('Error: The "iter_files_in_bucket" request was denied due to request throttling. If the problem '
  	'persists check the following link to learn how to use the Retry configuration to avoid it: '
  	f'{RETRY_CONFIGURATION_URL}', 2)
  	sys.exit(16)
  	else:
  	debug(f'ERROR: The "iter_files_in_bucket" request failed: {err}', 1)
  	sys.exit(1)
  	except Exception as err:
  	if hasattr(err, 'message'):
  	debug(f"+++ Unexpected error: {err.message}", 2)
  	else:
  	debug(f"+++ Unexpected error: {err}", 2)
  	print(f"ERROR: Unexpected error querying/working with objects in S3: {err}")
  	sys.exit(7)

• Should log the passed service option:
  

  wazuh/wodles/aws/aws_s3.py

  Lines 3643 to 3649 in e5d9fd4

  	elif options.service:
  	if options.service.lower() == 'inspector':
  	service_type = AWSInspector
  	elif options.service.lower() == 'cloudwatchlogs':
  	service_type = AWSCloudWatchLogs
  	else:
  	raise Exception("Invalid type of service")

Logging Format:

• Debug:
  Is printed with print and uses the function debug:
  

  wazuh/wodles/aws/aws_s3.py

  Lines 3427 to 3429 in e5d9fd4

  	def debug(msg, msg_level):
  	if debug_level >= msg_level:
  	print('DEBUG: {debug_msg}'.format(debug_msg=msg))

  The format of a debug msg is DEBUG: +++ Debug mode on - Level: 1 (example of use).

• Error:
  Sometimes it uses the function debug and other times it uses print.
  The format of an error msg is:

  • DEBUG: ERROR: The "get_log_streams" request failed: Code 404' (example of msg with debug).
  • ERROR: Access error: 404 (Example of msg with print).

• Info:
  There is no info level, which is replaced by debug.

[RamosFe]

Update

Started developing the changes mentioned above, starting with the AWS Wodle.

[RamosFe]

Update

Google Cloud

Error Handling

For handling errors the wodle uses aBase Exception class that defines the format of the logs

wazuh/wodles/gcloud/exceptions.py

Lines 16 to 44 in e5d9fd4

	class WazuhIntegrationException(Exception):
	"""Class that represents an exception for the Wazuh external integrations.
	Parameters
	----------
	error : int
	Error key.
	kwargs : str
	Values of the error message that should be substituted.
	"""
	def __init__(self, errcode: int, **kwargs):
	self._errcode = errcode
	info = self.__class__.ERRORS[errcode]
	self._message = info['message'].format(**kwargs) if kwargs else \
	info['message']
	self._key = info['key']
	super().__init__(f'{self.key}: {self.message}')
	@property
	def errcode(self):
	return self._errcode
	@property
	def key(self):
	return self._key
	@property
	def message(self):
	return self._message

All the subclasses handle the multiple error cases the API could give, so there is no improvement to make in the structure of how the errors are handled.

Log Format

The format of the logs is the same for all the errors, as shown below:

wazuh/wodles/gcloud/gcloud.py

Lines 92 to 107 in e5d9fd4

	except exceptions.WazuhIntegrationException as gcloud_exception:
	logging_func = logger.critical if \
	isinstance(gcloud_exception, exceptions.WazuhIntegrationInternalError) else \
	logger.error
	logging_func(f'An exception happened while running the wodle: {gcloud_exception}', exc_info=log_level == 1)
	exit(gcloud_exception.errcode)
	except Exception as e:
	logger.critical(f'Unknown error: {e}', exc_info=True)
	exit(exceptions.UNKNOWN_ERROR_ERRCODE)
	else:
	logger.info(f'Received {"and acknowledged " if arguments.integration_type == "pubsub" else ""}'
	f'{num_processed_messages} message{"s" if num_processed_messages != 1 else ""}')
	exit(0)

[RamosFe]

Update

AWS

Created an error function for errors that must be logged no matter the level of logging passed and to have the same format for all the errors. Added the requested changes marked in the comments above.

Azure

Added the changes described in earlier comments.

[RamosFe]

Update

Added more logs for the logging level info. The list of events that are now logged is shown below:

Azure

Azure Analytics

• When it uses the auth file for authentication.
• When it uses the key and domain for authentication.
• Time intervals of the Analytics request query.
• When it requests the API.

Azure Graph

• When it uses the auth file for authentication.
• When it uses the key and domain for authentication.
• Time intervals of the Graph request filter.
• When it requests the API.
• When it iterates to the next page.

Azure Storage

• When it uses the auth file for authentication.
• When it uses the key and account name for authentication.
• When it gets all the containers of the account.
• When it gets the specified containers of the account.
• When it requests data for a container.
• When it requests data for a blob.
• When it skips a blob due to a specific
  condition (Prefix not set, not expected format, empty blob).

[RamosFe]

Update

Added more logs for the logging level info. The list of events that are now logged is shown below:

AWS

• What type of service is working with.
• What type of bucket is working with.
• When the account id is not specified and the id is searched in the bucket.
• When the region is not specified and the regions of the account are searched in the account data.
• Region in which the bucket is being processed.
• File from which information is being obtained.

Google Cloud

• When it checks the credentials before starting.
• When it works with Pub/Sub or Access Logs.

[RamosFe]

Update

The Issue is blocked until the 3 Issues mentioned here related to the design of the log are completed. After these issues are completed, we need to adapt the logs created in this Issue to the new format.

[RamosFe]

Update

After multiple changes in the external modules, especially azure and aws, it was decided that it was best to create a new branch and add the changes to this new branch than rebasing the existing one with 4.9.0. The PR #16748 was replaced with PR #23790.

UT Results

(venv) federamos@pop-os:~/Documents/Wazuh/Repositories/wazuh/wodles$ pytest .
========================================================================================================== test session starts ==========================================================================================================
platform linux -- Python 3.9.18, pytest-7.3.1, pluggy-1.5.0
rootdir: /home/federamos/Documents/Wazuh/Repositories/wazuh/wodles
configfile: pytest.ini
plugins: asyncio-0.18.1, html-2.1.1, metadata-3.1.1, tavern-1.23.5, aiohttp-1.0.4, trio-0.8.0
asyncio: mode=auto
collected 889 items                                                                                                                                                                                                                     

aws/tests/test_aws_bucket.py .................................................................................................................................................................................................... [ 22%]
.............                                                                                                                                                                                                                     [ 23%]
aws/tests/test_aws_s3.py ......................                                                                                                                                                                                   [ 25%]
aws/tests/test_aws_service.py ....                                                                                                                                                                                                [ 26%]
aws/tests/test_cloudtrail.py ..                                                                                                                                                                                                   [ 26%]
aws/tests/test_cloudwatchlogs.py .....................................................                                                                                                                                            [ 32%]
aws/tests/test_config.py ..............................................................................                                                                                                                           [ 41%]
aws/tests/test_guardduty.py .................                                                                                                                                                                                     [ 43%]
aws/tests/test_inspector.py ......                                                                                                                                                                                                [ 43%]
aws/tests/test_load_balancers.py ............                                                                                                                                                                                     [ 45%]
aws/tests/test_s3_log_handler.py .......................                                                                                                                                                                          [ 47%]
aws/tests/test_server_access.py .................................                                                                                                                                                                 [ 51%]
aws/tests/test_sqs_message_processor.py ........                                                                                                                                                                                  [ 52%]
aws/tests/test_sqs_queue.py .......                                                                                                                                                                                               [ 53%]
aws/tests/test_tools.py ..................................                                                                                                                                                                        [ 57%]
aws/tests/test_umbrella.py ......                                                                                                                                                                                                 [ 57%]
aws/tests/test_vpcflow.py ...........................                                                                                                                                                                             [ 60%]
aws/tests/test_waf.py .......                                                                                                                                                                                                     [ 61%]
aws/tests/test_wazuh_integration.py ......................................................................................................                                                                                        [ 73%]
azure/tests/test_azure_utils.py ...................................................                                                                                                                                               [ 78%]
azure/tests/azure_services/test_analytics.py ..................                                                                                                                                                                   [ 80%]
azure/tests/azure_services/test_graph.py ............                                                                                                                                                                             [ 82%]
azure/tests/azure_services/test_storage.py ...........................                                                                                                                                                            [ 85%]
azure/tests/db/test_db_utils.py .........                                                                                                                                                                                         [ 86%]
azure/tests/db/test_orm.py ...............................                                                                                                                                                                        [ 89%]
docker-listener/tests/test_docker_listener.py ...................                                                                                                                                                                 [ 91%]
gcloud/tests/test_bucket.py .................................                                                                                                                                                                     [ 95%]
gcloud/tests/test_gcloud.py .........                                                                                                                                                                                             [ 96%]
gcloud/tests/test_integration.py ........                                                                                                                                                                                         [ 97%]
gcloud/tests/test_subscriber.py ..............                                                                                                                                                                                    [ 99%]
gcloud/tests/test_tools.py ........                                                                                                                                                                                               [100%]

========================================================================================================== 889 passed in 4.53s ==========================================================================================================

Comments about AWS

Some of the debug messages were left untouched because they were created in the Refactor Issue and their correct operation was validated therein.