IMPROVEMENT: Rename file_documentation into configuration_steps

Misc grammar improvements in documentation
Refractored two classes out of the LocalFilesystem class

ARCHITECTURE.md

@@ -4,16 +4,16 @@ Before we decided on a software architecture or programming language or toolkit
 
 ## Software requirements
 
-The goal of this software is to automate some of the tasks involved in configuring and tuning an ArduPilot based vehicle.
+The goal of this software is to automate some of the tasks involved in configuring and tuning an ArduPilot-based vehicle.
 The method it follows is explained in the [How to methodically tune (almost) any multicopter using ArduCopter forum Blog post](https://discuss.ardupilot.org/t/how-to-methodically-tune-almost-any-multicopter-using-arducopter-4-4-x/110842/1). 
 This list of functionalities provides a comprehensive overview of the software's capabilities and can serve as a starting point for further development and refinement.
 
 ### 1. Parameter Configuration Management
 - The software must allow users to view and manage drone parameters.
 - Users should be able to select an intermediate parameter file from a list of available files.
-- The software must display a table of parameters with columns for parameter name, current value, new value, unit, write to flight controller, and change reason.
+- The software must display a table of parameters with columns for the parameter name, current value, new value, unit, write to flight controller, and change reason.
 - The software must validate the new parameter values and handle out-of-bounds values gracefully, reverting to the old value if the user chooses not to use the new value.
-- The software must save parameter changes to both the flight controller and to the intermediate parameter files
+- The software must save parameter changes to both the flight controller and the intermediate parameter files
 
 ### 2. Communication Protocols
 - The software must support communication with the drone's flight controller using MAVlink and FTP over MAVLink protocols.
@@ -30,7 +30,7 @@ This list of functionalities provides a comprehensive overview of the software's
 - The software must provide feedback to the user, such as success or error messages, when performing actions like writing parameters to the flight controller.
 - Users should be able to skip to the next parameter file without writing changes.
 - The software must ensure that all changes made to entry widgets are processed before proceeding with other actions, such as writing parameters to the flight controller.
-- Read-only parameters are displayed in red, Sensor Calibrations are displayed in yellow, non existing parameters in blue
+- Read-only parameters are displayed in red, Sensor Calibrations are displayed in yellow and non-existing parameters in blue
 - Users should be able to edit the new value for each parameter directly in the table.
 - Users should be able to edit the reason changed for each parameter directly in the table.
 - The software must provide tooltips for each parameter to explain their purpose and usage.
@@ -50,19 +50,19 @@ This list of functionalities provides a comprehensive overview of the software's
 ### 8. Parameter File Management
 - The software must support the loading and parsing of parameter files.
 - Users should be able to navigate through different parameter files and select the one to be used.
-- Comments are first class citizens and are preserved when reading/writing files
-- The software must write in the end of the configuration following summary files:
+- Comments are first-class citizens and are preserved when reading/writing files
+- The software must write at the end of the configuration the following summary files:
   - Complete flight controller "reason changed" annotated parameters in "complete.param" file
-  - Non default, read-only "reason changed" annotated parameters in, "non-default_read-only.param" file
-  - Non default, writable calibrations "reason changed" annotated parameters in "non-default_writable_calibrations.param" file
-  - Non default, writable non-calibrations "reason changed" annotated parameters in "non-default_writable_non-calibrations.param" file
+  - Non-default, read-only "reason changed" annotated parameters in, "non-default_read-only.param" file
+  - Non-default, writable calibrations "reason changed" annotated parameters in "non-default_writable_calibrations.param" file
+  - Non-default, writable non-calibrations "reason changed" annotated parameters in "non-default_writable_non-calibrations.param" file
 
 
 ### 9. Customization and Extensibility
 - The software must be extensible to support new drone models and parameter configurations.
-- Users should be able to customize the software's behavior through configuration files: file_documentation.json and *.param.
-- Development should use industry best-practices:
-  - [Test driven development](https://en.wikipedia.org/wiki/Test-driven_development) (TDD)
+- Users should be able to customize the software's behavior through configuration files: ArduCopter_configuration_steps.json and *.param.
+- Development should use industry best practices:
+  - [Test-driven development](https://en.wikipedia.org/wiki/Test-driven_development) (TDD)
   - [DevOps](https://en.wikipedia.org/wiki/DevOps)
 
 ### 10. Performance and Efficiency
@@ -76,20 +76,21 @@ To satisfy the software requirements described above the following software arch
 
 It consists of four main components:
 
-1. the application itself, does the command line parsing and starts the other processes
-2. the local filesystem backend, does file I/O on the local file system. Operates mostly on parameter files and metadata/documentation files
-3. the flight controller backend, does communication with the flight controller
-4. the tkinter frontend, this is the GUI the user interacts with
+1. the application itself does the command line parsing and starts the other processes
+2. the local filesystem backend does file I/O on the local file system. Operates mostly on parameter files and metadata/documentation files
+3. the flight controller backend communicates with the flight controller
+4. the tkinter frontend, which is the GUI the user interacts with
   1. `frontend_tkinter_base.py`
   2. `frontend_tkinter_connection_selection.py`
   3. `frontend_tkinter_directory_selection.py`
   4. `frontend_tkinter_component_editor.py`
   5. `frontend_tkinter_parameter_editor.py`
+  6. `frontend_tkinter_parameter_editor_table.py`
 
 ![Software Architecture diagram](images/Architecture.drawio.png)
 
 The parts can be individually tested, and do have unit tests.
-They can also be exchanged, for instance tkinter-frontend can be replaced with vxwidgets or pyQt.
+They can also be exchanged, for instance, tkinter-frontend can be replaced with vxwidgets or pyQt.
 
-In the future we might port the entire application into a client-based web-application.
+In the future, we might port the entire application into a client-based web application.
 That way the users would not need to install the software and will always use the latest version.

MethodicConfigurator/backend_filesystem.py

@@ -30,7 +30,6 @@
 
 from json import load as json_load
 from json import dump as json_dump
-from json import JSONDecodeError
 
 from platform import system as platform_system
 
@@ -50,6 +49,9 @@
 from annotate_params import split_into_lines
 from annotate_params import update_parameter_documentation
 
+from backend_filesystem_vehicle_components import VehicleComponents
+from backend_filesystem_configuration_steps import ConfigurationSteps
+
 
 TOOLTIP_MAX_LENGTH = 105
 
@@ -74,109 +76,49 @@ def is_within_tolerance(x: float, y: float, atol: float = 1e-08, rtol: float = 1
     return abs(x - y) <= atol + (rtol * abs(y))
 
 
-class LocalFilesystem:  # pylint: disable=too-many-instance-attributes, too-many-public-methods
+class LocalFilesystem(VehicleComponents, ConfigurationSteps):  # pylint: disable=too-many-public-methods
     """
     A class to manage local filesystem operations for the ArduPilot methodic configurator.
 
     This class provides methods for initializing and re-initializing the filesystem context,
-    reading parameters from files, and handling file documentation. It is designed to simplify
+    reading parameters from files, and handling configuration steps. It is designed to simplify
     the interaction with the local filesystem for managing ArduPilot configuration files.
 
     Attributes:
         vehicle_dir (str): The directory path where the vehicle configuration files are stored.
         vehicle_type (str): The type of the vehicle (e.g., "ArduCopter", "Rover").
-        file_documentation_filename (str): The name of the file containing documentation for the configuration files.
-        file_documentation (dict): A dictionary containing the file documentation.
         file_parameters (dict): A dictionary of parameters read from intermediate parameter files.
         param_default_dict (dict): A dictionary of default parameter values.
         doc_dict (dict): A dictionary containing documentation for each parameter.
     """
     def __init__(self, vehicle_dir: str, vehicle_type: str):
-        self.file_documentation_filename = "file_documentation.json"
-        self.file_documentation = {}
-        self.vehicle_components_json_filename = "vehicle_components.json"
-        self.vehicle_components = {}
+        self.file_parameters = None
+        VehicleComponents.__init__(self)
+        ConfigurationSteps.__init__(self, vehicle_dir, vehicle_type)
         if vehicle_dir is not None:
             self.re_init(vehicle_dir, vehicle_type)
 
     def re_init(self, vehicle_dir: str, vehicle_type: str):
         self.vehicle_dir = vehicle_dir
         self.vehicle_type = vehicle_type
-        self.file_documentation = {}
         self.param_default_dict = {}
         self.doc_dict = {}
-        self.vehicle_components = {}
 
         # Read intermediate parameters from files
         self.file_parameters = self.read_params_from_files()
         if not self.file_parameters:
             return # No files intermediate parameters files found, no need to continue, the rest needs them
 
-        # Define a list of directories to search for the file_documentation_filename file
-        search_directories = [self.vehicle_dir, os_path.dirname(os_path.abspath(__file__))]
-        file_found = False
-        for i, directory in enumerate(search_directories):
-            try:
-                with open(os_path.join(directory, self.file_documentation_filename), 'r', encoding='utf-8') as file:
-                    self.file_documentation = json_load(file)
-                    file_found = True
-                    if i == 0:
-                        logging_warning("File documentation '%s' loaded from %s (overwriting default file documentation).",
-                                         self.file_documentation_filename, directory)
-                    if i == 1:
-                        logging_info("File documentation '%s' loaded from %s.", self.file_documentation_filename, directory)
-                    break
-            except FileNotFoundError:
-                pass
-        if file_found:
-            self.validate_forced_parameters_in_file_documentation()
-            self.validate_derived_parameters_in_file_documentation()
-        else:
-            logging_warning("No file documentation will be available.")
-
         # Read ArduPilot parameter documentation
         xml_dir = vehicle_dir if os_path.isdir(vehicle_dir) else os_path.dirname(os_path.realpath(vehicle_dir))
         xml_root, self.param_default_dict = get_xml_data(BASE_URL + vehicle_type + "/", xml_dir, PARAM_DEFINITION_XML_FILE)
         self.doc_dict = create_doc_dict(xml_root, vehicle_type, TOOLTIP_MAX_LENGTH)
 
-        self.extend_and_reformat_parameter_documentation_metadata()
-        self.load_vehicle_components_json_data()
-
-    def validate_forced_parameters_in_file_documentation(self):
-        for filename, file_info in self.file_documentation.items():
-            if 'forced_parameters' in file_info and filename in self.file_parameters:
-                if not isinstance(file_info['forced_parameters'], dict):
-                    logging_error("Error in file '%s': '%s' forced parameter is not a dictionary",
-                                        self.file_documentation_filename, filename)
-                    continue
-                for parameter, parameter_info in file_info['forced_parameters'].items():
-                    if "New Value" not in parameter_info:
-                        logging_error("Error in file '%s': '%s' forced parameter '%s'"
-                                          " 'New Value' attribute not found.",
-                                          self.file_documentation_filename, filename, parameter)
-                    if "Change Reason" not in parameter_info:
-                        logging_error("Error in file '%s': '%s' forced parameter '%s'"
-                                          " 'Change Reason' attribute not found.",
-                                          self.file_documentation_filename, filename, parameter)
-
-    def validate_derived_parameters_in_file_documentation(self):
-        for filename, file_info in self.file_documentation.items():
-            if 'derived_parameters' in file_info and filename in self.file_parameters:
-                if not isinstance(file_info['derived_parameters'], dict):
-                    logging_error("Error in file '%s': '%s' derived parameter is not a dictionary",
-                                        self.file_documentation_filename, filename)
-                    continue
-                for parameter, parameter_info in file_info['derived_parameters'].items():
-                    if "New Value" not in parameter_info:
-                        logging_error("Error in file '%s': '%s' derived parameter '%s'"
-                                          " 'New Value' attribute not found.",
-                                          self.file_documentation_filename, filename, parameter)
-                    if "Change Reason" not in parameter_info:
-                        logging_error("Error in file '%s': '%s' derived parameter '%s'"
-                                          " 'Change Reason' attribute not found.",
-                                          self.file_documentation_filename, filename, parameter)
-
-    def extend_and_reformat_parameter_documentation_metadata(self):
+        self.__extend_and_reformat_parameter_documentation_metadata()
+        self.load_vehicle_components_json_data(vehicle_dir)
+
+
+    def __extend_and_reformat_parameter_documentation_metadata(self):
         for param_name, param_info in self.doc_dict.items():
             if 'fields' in param_info:
                 if 'Units' in param_info['fields']:
@@ -278,7 +220,7 @@ def export_to_param(self, params: Dict[str, 'Par'], filename_out: str, annotate_
 
     def intermediate_parameter_file_exists(self, filename: str) -> bool:
         """
-        Checks if an intermediate parameter file exists in the vehicle directory.
+        Check if an intermediate parameter file exists in the vehicle directory.
 
         Parameters:
         - filename (str): The name of the file to check.
@@ -289,7 +231,7 @@ def intermediate_parameter_file_exists(self, filename: str) -> bool:
         return os_path.exists(os_path.join(self.vehicle_dir, filename)) and \
             os_path.isfile(os_path.join(self.vehicle_dir, filename))
 
-    def all_intermediate_parameter_file_comments(self) -> Dict[str, str]:
+    def __all_intermediate_parameter_file_comments(self) -> Dict[str, str]:
         """
         Retrieves all comments associated with parameters from intermediate parameter files.
 
@@ -322,14 +264,14 @@ def annotate_intermediate_comments_to_param_dict(self, param_dict: Dict[str, flo
         - Dict[str, 'Par']: A dictionary of parameters with intermediate parameter file comments.
         """
         ret = {}
-        ip_comments = self.all_intermediate_parameter_file_comments()
+        ip_comments = self.__all_intermediate_parameter_file_comments()
         for param, value in param_dict.items():
             ret[param] = Par(float(value), ip_comments.get(param, ''))
         return ret
 
     def categorize_parameters(self, param: Dict[str, 'Par']) -> List[Dict[str, 'Par']]:
         """
-        Categorizes parameters into three categories based on their default values and documentation attributes.
+        Categorize parameters into three categories based on their default values and documentation attributes.
 
         This method iterates through the provided dictionary of parameters and categorizes them into three groups:
         - Non-default, read-only parameters
@@ -349,7 +291,7 @@ def categorize_parameters(self, param: Dict[str, 'Par']) -> List[Dict[str, 'Par'
         for param_name, param_info in param.items():
             if param_name in self.param_default_dict and is_within_tolerance(param_info.value,
                                                                              self.param_default_dict[param_name].value):
-                continue     # parameter has default value, ignore it
+                continue     # parameter has a default value, ignore it
 
             if param_name in self.doc_dict and self.doc_dict[param_name].get('ReadOnly', False):
                 non_default__read_only_params[param_name] = param_info
@@ -400,7 +342,7 @@ def zip_files(self, files_to_zip: List[Tuple[bool, str]]):
                 zipf.write(os_path.join(self.vehicle_dir, file_name), arcname=file_name)
 
             # Check for and add specific files if they exist
-            specific_files = ["00_default.param", "apm.pdef.xml", "file_documentation.json",
+            specific_files = ["00_default.param", "apm.pdef.xml", "ArduCopter_configuration_steps.json",
                               "vehicle_components.json", "vehicle.jpg"]
             for file_name in specific_files:
                 file_path = os_path.join(self.vehicle_dir, file_name)
@@ -431,29 +373,6 @@ def vehicle_image_filepath(self):
     def vehicle_image_exists(self):
         return os_path.exists(self.vehicle_image_filepath())
 
-    def load_vehicle_components_json_data(self):
-        data = {}
-        try:
-            filepath = os_path.join(self.vehicle_dir, self.vehicle_components_json_filename)
-            with open(filepath, 'r', encoding='utf-8') as file:
-                data = json_load(file)
-        except FileNotFoundError:
-            logging_warning("File '%s' not found in %s.", self.vehicle_components_json_filename, self.vehicle_dir)
-        except JSONDecodeError:
-            logging_error("Error decoding JSON data from file '%s'.", filepath)
-        self.vehicle_components = data
-        return data
-
-    def save_vehicle_components_json_data(self, data) -> bool:
-        filepath = os_path.join(self.vehicle_dir, self.vehicle_components_json_filename)
-        try:
-            with open(filepath, 'w', encoding='utf-8') as file:
-                json_dump(data, file, indent=4)
-        except Exception as e:  # pylint: disable=broad-except
-            logging_error("Error saving JSON data to file '%s': %s", filepath, e)
-            return True
-        return False
-
     def new_vehicle_dir(self, base_dir: str, new_dir: str):
         return os_path.join(base_dir, new_dir)
 
@@ -487,7 +406,7 @@ def getcwd():
     @staticmethod
     def valid_directory_name(dir_name: str):
         """
-        Checks if a given directory name contains only alphanumeric characters, underscores, hyphens,
+        Check if a given directory name contains only alphanumeric characters, underscores, hyphens,
         and the OS directory separator.
 
         This function is designed to ensure that the directory name does not contain characters that are
@@ -508,11 +427,6 @@ def tempcal_imu_result_param_tuple(self):
         tempcal_imu_result_param_filename = "03_imu_temperature_calibration_results.param"
         return [tempcal_imu_result_param_filename, os_path.join(self.vehicle_dir, tempcal_imu_result_param_filename)]
 
-    def auto_changed_by(self, selected_file: str):
-        if selected_file in self.file_documentation:
-            return self.file_documentation[selected_file].get('auto_changed_by', '')
-        return ''
-
     def copy_fc_values_to_file(self, selected_file: str, params: Dict[str, float]):
         ret = 0
         if selected_file in self.file_parameters: