Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
V2 - General Bug fixes and updates (#51)
* Update AwsClientHub to accept keyword arguments The AwsClientHub class has been modified to accept additional keyword arguments. These arguments are passed to the configuration manager during its instantiation. This feature will enable more flexible environment configurations as all options can now be supplied as arguments during the class initialization or configuration reload. * Refactor RdsServiceGateway for improved robustness This commit simplifies the get_data and update_database methods in the RdsServiceGateway class, making them more robust. It removes unused imports, introduces psycopg2.extensions, and tweaks how queries are formatted and logged. The function signatures are also updated to simplify and clarify their usage. Additional helper methods get_cursor and get_connection are also introduced for easy access to database cursor and connection. * Refactor upload methods in S3ServiceGateway Split the upload method in S3ServiceGateway into upload_file and upload_object, enhancing functionality and usability. The upload_file method handles the uploading of files specifically, while upload_object caters to uploading any type of byte object. * Update .env path detection and set default SSH port Enhancements were made to the ConfigurationManager class. Specifically, the logic for locating the .env file was updated to handle scenarios where the filename is directly provided. Moreover, the default SSH port was changed from 0 to 22, taking into account the standard default SSH port. * Refactor and improve error handling in Lambda function This commit refactors the error handling function in the Lambda codebase to provide clearer, more readable code. It replaces the usage of the DictValidator for type checking with TypeChecker. Further, it updates the format of function parameter comments to conform with the standard style. The error handling mechanism has been improved for clarity and maintainability. * Refactor `trigger_dataflow_metrics.py` for improved readability The `trigger_dataflow_metrics.py` script has been cleaned up. Redundant or unnecessary comments have been removed, the function `trigger_minimum_dataflow_metrics` has been restructured, and arguments have been simplified. Updates have also been made to ensure that values are pulled more effectively from event and context arguments within the `trigger_dataflow_metrics` function. * Add MultiThreaded decorator for concurrent execution This commit includes a new MultiThreaded decorator which can help running multiple threads concurrently. It provides options for the number of threads to spawn and the level of logging to use. The decorator is designed in such a way as to also handle and log any exceptions occurring within each thread. * Enhance Retryable decorator with additional error-handling and logging features The Retryable decorator now not only retries a function call on exceptions, but also when a HTTP request yields a non-200 status code. Exceptions have also been expanded to include `JSONDecodeError`, specific `requests` exceptions, `botocore` exceptions and general exceptions. Logging functionality has been extended to provide verbosity control and customizable logging levels. The function's state is logged at different verbosity levels. * Add logging level parameter to TimedMethod decorator Enhanced the TimedMethod decorator to accept a 'level' parameter which determines the logging level for time tracking messages. This lets the user define the log's verbosity, establishing a finer control over the monitorization of performance. * Added MultiThreaded decorator to __init__.py An import line was added for the MultiThreaded decorator in the decorator's __init__.py file of the WrenchCL project. This decorator was also added to the __all__ list, making it officially a part of the public API. * Update methods in OpenAIFactory for enhanced functionality The methods for 'audio_to_text_to_embeddings' and 'image_to_text_to_embeddings' in OpenAIFactory are updated. The changes include a new optional 'dimensions' parameter for returning the vector of specific size for both methods. In addition, the 'image_to_text_to_embeddings' method has been enhanced to return both the generated embeddings and the vision query response. * Updated method parameters and functionality in OpenAIGaterway In the OpenAIGateway module, several method parameters have been added or modified to improve functionality and flexibility. Major changes include the addition of assistant_id and format parameters in the text_response method, optimizations and refinements in image_to_text method, and modifications in get_embeddings method parameters. Removed the convert_image_to_url_or_base64 function as it's redundant with the updated image_to_text function. * Add support for Base64 strings to file type determination This update modifies the `get_file_type` function in `FileTyper.py`, allowing it to handle Base64 strings in addition to URLs and file paths. The method can now determine file type using the details embedded in a Base64 string, improving the tool's flexibility and functionality in handling different data sources. * Add SHA-1 hashing and Base64 validation to Image2B64 tool The updated Image2B64 tool now includes a SHA-1 hashing function for provided data, and supports optional return of this hash when converting images to Base64. This update also * Add single quote decoder to JsonSerializer This commit introduces a new decoder class, `single_quote_decoder`, to the `JsonSerializer` module. The class is for pre-processing JSON strings that use single quotes for keys and values or include Markdown block markers. This function is supposed to fix issues with decoding JSON strings from certain sources that do not strictly follow the syntax rules. * Rename DictValidator to TypeChecker and enhance functionality The DictValidator.py file was renamed to TypeChecker.py. The validation function was enhanced to handle a list of dictionaries, multiple expected types per parameter with a new "none_is_ok" option for accepting None values. It retains the error handling strategy, raising TypeError or issuing warning and returning False when the input data is invalid. These changes make type checking more versatile and robust. * Update styling and handling in WrenchLogger module Improved the text styling and handling in the WrenchLogger module. The colors and the brightness of the printout have been adjusted. Handling of string type data, edge cases for different formatting situations and empty starts have been properly addressed. A visual enhancement to application headers was added when colorama is imported. * Update imports in Tools __init__ file Updated the import script on Tools __init__ script. New methods namely validate_base64, get_hash from Image2B64 and typechecker from TypeChecker, along with single_quote_decoder from JsonSerializer have been added. Removed validate_input_dict from DictValidator. * Update and expand the README file The README has been updated and is now more informative. Crucial areas such as the ways to use AwsClientHub, RdsServiceGateway, S3ServiceGateway, OpenAIFactory, OpenAIGateway, utility tools, and decorators are now covered. Detailed instructions regarding keyword arguments for AwsClientHub and _ConfigurationManager have also been added. * Add pillow and hashlib to requirements Additions have been made to the project's dependencies. The Python packages 'pillow' and 'hashlib' have been added to the requirements.txt file, which are required for image processing and providing secure hashes and message digest algorithms respectively. * Add Documentation Pages * Remove hashlib from requirements This commit removes the unnecessary hashlib package from the project's dependencies. Now, there are only the required and used packages in the requirements file. * Update imports in trigger_dataflow_metrics.py The typing imports in the trigger_dataflow_metrics.py file have been updated. 'Optional' was added to the list of imported items from the typing module. * Remove unnecessary import in test file An unused import, validate_input_dict, has been removed from the test_imports.py file. This tidy-up helps to make the codebase cleaner, improving its maintainability and readability. * No jekyll processing * Moved file * Remove outdated HTML files from WrenchCL DataFlow documentation and swapped theme to RTD * Remove outdated HTML files from WrenchCL DataFlow documentation and swapped theme to RTD * Refactor MultiThreaded decorator code The unnecessary 'import logging' statement has been removed from the MultiThreaded.py file. In addition, cosmetic changes were made with whitespace added for better code readability. * Remove unused Literal import from TypeChecker.py A cleanup task was carried out on the TypeChecker.py file. The import statement was updated to remove "Literal" since it was not being used anywhere in the code. This change contributes to maintaining the cleanliness of the code.
- Loading branch information
1 parent
606b4e5
commit 4108044