Repository navigation
Developer script for anonymizing logs #562
chmtc94
started this conversation in
Show and tell
Replies: 1 comment
1) Fix issue in script
|
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
For anyone interested, I am sharing here the Python script I wrote and use to anonymize integration logs when debug mode is enabled.
Hide secrets in log files.py
This script utilizes a
PythonJSONDecoderclass—which I also wrote—to directly parse a JSON-formatted string (as found in the logs) into a Python object. The log format uses Python syntax (single quotes instead of double quotes,Trueinstead oftrue,Falseinstead offalse, etc.).PythonJSON.py
The script can then simply use the standard
loadscommand from thejsonpackage to decode the log string (python_string) into a Python object (python_object):python_object = json.loads(python_string, cls=PythonJSONDecoder)This makes all standard Python object manipulation functions available for use on
python_object.I should note that this script could certainly be improved; it was originally designed solely for my personal use to save time when anonymizing logs. When @andreadegiovine added this feature to the integration, I encountered severe CPU overload issues that forced me to abandon it (see issue #414 ).
I have continued using my script, recently making minor tweaks to automatically add line breaks for better log readability. It seems more logical to me not to consume HA CPU resources for a function that is only useful for debugging—which is not the integration's primary purpose😉
I am providing this script merely as a potential starting point for anyone interested; it is up to them to adapt it to their specific needs, potentially for use in other projects. Since use cases vary widely, I do not plan to maintain the script. However, I remain available to answer any questions or consider suggestions.
The script processes the file to be anonymized in two passes:
The first pass identifies all occurrences of variable names to be anonymized along with the values assigned to them, but only within JSON strings found in the log lines. During this pass, all occurrences of the recorded values are replaced with the anonymized name
<VARIABLE_NAME_i>, whereiis an integer that increments each time the variable is encountered with a value different from any previously seen for that variable.The list of variable names to be anonymized is defined in a separate file (which should be stored securely). An example is provided here:
secrets_values_example.json
In this file, the
SECRET_KEYSlist contains the keys whose values are to be anonymized.In the second pass, all values encountered during the first pass are replaced with the anonymized name stored during that first pass. This second pass is necessary to replace any values requiring anonymization that appear outside of JSON strings in the log (for example, in URLs).
To ensure the most complete anonymization possible, the
secret_values_example.jsonfile includes an initial set of data to be anonymized—data that might not be identified during the first pass (e.g., if not present in JSON strings or if the log excerpt is insufficient to capture an occurrence). This is the purpose of theSECRET_VALUESdictionary insecret_values_example.json, which is used to initialize the list of values to be anonymized before the first pass begins.To use the script as-is, place the
hide_secret_in_log_files.pyfile in a personal working directory, along with thePythonJSON.pyfile.The
secret_values_example.jsonfile should be customized, renamed, and stored in a private location. I placed this data in a separate file—rather than inside the script itself—to avoid issues in case I were to share the script by putting it in a public folder.The script is launched from a terminal within the script's directory using the command:
> Python "Hide secrets in log files.py"The script first asks you to select the JSON file to use for
secret_values—specifically, thesecret_values_example.jsonfile you have previously customized (the minimalist interface isn't great, but then again, this is a development tool, not really a polished application).Next, it asks you to select the files to be anonymized (multiple files can be selected).
Once confirmed, all selected files are processed:
_anonymisedadded to their filenames.May this sharing be useful.
All reactions