diff --git a/.gitignore b/.gitignore index 83bd9e2d..043ad5ea 100644 --- a/.gitignore +++ b/.gitignore @@ -189,3 +189,5 @@ manifest.bak .spellignore .project-planning .master-planning +output-shell.txt +output-script.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index f8b9b7eb..7b7c8fe6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,149 @@ # Changelog +## 2025-02-10 - Documentation Overhaul and Process Improvements + +### Documentation Structure and Organization +- Established standardized documentation format across all components +- Enhanced core library documentation with detailed API references +- Improved cross-referencing between related documentation files +- Added comprehensive examples and use cases for all major features + +### Process Improvements +- Implemented new periodic review process with clear guidelines +- Added structured approach for tracking and documenting changes +- Enhanced changelog management with better categorization +- Improved worklog organization for technical details + +### Tool Documentation +- Enhanced documentation for all Python and shell tools: + - `chunkfile.py`: Complete rewrite with detailed API docs + - `warehouse.sh`: Added comprehensive usage guide + - `generate_manifest.sh`: Enhanced with git integration details +- Added visual guides for complex operations +- Updated all command-line help messages + +### Technical Documentation +- Added detailed architecture documentation +- Enhanced troubleshooting guides +- Improved installation and setup instructions +- Added performance considerations and best practices + +## 2025-02-06 - Major Release: Library Reorganization and New Tools + +### Core Library Reorganization +- Renamed all shell library files to use `_lib.sh` suffix for better organization +- Added new core libraries and new functionality: + - `config_lib.sh`: Configuration management functions + - `type_lib.sh`: Type checking and validation + - `string_lib.sh`: String manipulation utilities + - `errno_lib.sh`: Enhanced error handling + - `helpsys_lib.sh`: Improved help system + - `init_lib.sh`: Initialization routines + - `venvutil_lib.sh`: Virtual environment utilities + - `wrapper_lib.sh`: Command wrapping functions + +### Virtual Environment Enhancements +- Enhanced `lenv` function with new capabilities: + - Added column headers for better readability + - Added Python version display for each environment + - Improved sorting by time or name (forward/reverse) + - Enhanced long format time display with date and time + - Added environment status indicators +- Improved `ccln` (clone) function: + - Added support for cloning without sequence number + - Enhanced error handling during clone operations + - Added automatic environment activation after cloning +- Added virtual environment renaming capability: + - New `renv` function for renaming environments + - Preserves all packages and configurations + - Handles both sequenced and non-sequenced environments + - Updates environment references in configuration + - Maintains environment history and logs +- Added new environment management features: + - Better environment variable handling + - Improved error recovery mechanisms + - Enhanced logging for all venv operations + +### New Tools and Utilities +- Added performance testing tools: + - `torch_torture.py`: PyTorch stress testing + - `numpy_torture.py`: NumPy stress testing + - `numpy-comp` and `numpy-comp.sh`: NumPy compilation tools + - `compare_test`: General performance comparison framework +- Added chat management utilities: + - `extract-chat` and `extract_chat.py`: Chat extraction tools + - `rename-chat` and `rename-chat.py`: Chat renaming utilities + +### Core System Improvements +- Enhanced `setup.sh` with improved Conda integration and error handling +- Updated `setup.cf` with new configuration options +- Modified core utilities for better performance: + - `purgevenv`: Enhanced cleanup operations + - `genmd`: Better documentation generation +- Enhanced `requirements.txt` with latest dependencies + +### Documentation Enhancements +- Added new documentation: + - `Metrics_Layout.md`: Performance metrics documentation +- Updated existing documentation: + - `Standards.md`: Updated coding standards + - `filetree.md`: New file structure documentation + - `installer-manifest.md`: Updated installation guide +- Comprehensive updates to function documentation in `docs/shdoc/` + +### Technical Improvements +- Enhanced error handling across all utilities: + - Better POSIX errno code handling + - Improved error message formatting + - Enhanced error context for debugging +- Improved virtual environment management: + - Better state tracking + - Enhanced environment switching + - Improved environment cleanup +- Better type checking and validation: + - Added robust type checking functions + - Enhanced input validation + - Improved error reporting +- Enhanced string manipulation functions: + - Added new string cleaning utilities + - Improved pattern matching + - Better text formatting +- Improved configuration management: + - Enhanced config file handling + - Better default value management + - Improved configuration validation +- Better help system organization: + - Enhanced function documentation + - Improved command-line help + - Better cross-referencing + +## 2025-01-09 - Script Sourcing and Directory Handling Improvements + +### Shell Include Files Enhancements + +- Implemented consistent script sourcing tracking across all function include scripts +- Fixed bug in script sourcing prevention mechanism +- Improved directory creation handling +- Streamlined directory operations with optimized `mkdir -p` usage + +### Core Script Updates + +- `init_env.sh`: Enhanced script sourcing control with improved tracking +- `help_sys.sh`: Improved directory handling and path management +- `errno.sh`: Cleaned up code organization and removed redundant sections +- `util_funcs.sh`: Enhanced directory operations and error handling +- `venv_funcs.sh`: Updated script sourcing mechanism for better reliability +- `wrapper_funcs.sh`: Improved function loading control and consistency +- `buildvenvs`: Updated directory handling for better efficiency (experimental) +- `modules/conda-install.sh`: Enhanced installation process and error handling (experimental) + +### Technical Improvements + +- Removed redundant directory existence checks before mkdir operations +- Standardized script sourcing prevention across all include files +- Enhanced error handling for directory operations +- Improved code organization and cleanup + ## 2024-12-30 - Help System and Documentation Enhancements ### Help System Improvements @@ -357,5 +501,3 @@ This collection includes build scripts, benchmarking tools, and regression testi If you find any of my work here helpful, please reach out. I would like to have a dialog with anyone else interested. Watch this spot, more to come, and you can always [buy me a coffee.](https://www.buymeacoffee.com/venvutil) - -# diff --git a/README.md b/README.md index a6985e77..a44f913c 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,26 @@ # venvutil - Manage Conda and Pip VENV's with some simple functions and scripts -(*Still under development*) This project is continuously evolving, becoming a catch-all for useful tools and shell functions that facilitate working with Python VENV's and LLM's. +This is release 20250210_01-rel.This project is continuously evolving, becoming a catch-all for useful tools and shell functions that facilitate working with Python VENV's and LLM's. + +## What's New in This Major Release (20250210_01-rel) + +- Comprehensive documentation overhaul with standardized structure, improved API references, and enhanced cross-referencing across components. +- Library reorganization: All core libraries are now loaded via 'venvutil_lib.sh' with enhanced error handling and type checking. +- Enhanced virtual environment management: Updated 'lenv' now displays column headers and Python versions, improved 'ccln' cloning functionality, and a new 'renv' command for renaming environments. +- Improved logging and process management: Robust error handling and logging across pip, conda, and shell functions, ensuring reliability on both macOS and Linux. +- Thoroughly tested in four separate, fresh user environments, ensuring cross-platform stability and performance. +- Migration Notes: Update any direct library sourcing to use 'venvutil_lib.sh' and revise configuration files to align with the new pkg-config format. + - For full details, please refer to CHANGELOG.md for the complete list of changes. + +## Table of Contents - [venvutil - Manage Conda and Pip VENV's with some simple functions and scripts](#venvutil---manage-conda-and-pip-venvs-with-some-simple-functions-and-scripts) + - [What's New in This Major Release (20250210\_01-rel)](#whats-new-in-this-major-release-20250210_01-rel) + - [Table of Contents](#table-of-contents) - [Project Overview](#project-overview) - [Key Features](#key-features) - [Why Use Venvutil?](#why-use-venvutil) + - [Tested on the following systems](#tested-on-the-following-systems) - [Installation Instructions](#installation-instructions) - [Prerequisites](#prerequisites) - [Running the installer](#running-the-installer) @@ -14,7 +29,8 @@ - [Tools Overview](#tools-overview) - [Shell Functions](#shell-functions) - [Conda and Pip Logging](#conda-and-pip-logging) - - [~~C++, G++, and~~ LD Pass-Through](#c-g-and-ld-pass-through) + - [LD Pass-Through and NumPy builds](#ld-pass-through-and-numpy-builds) + - [Recipe for building NumPy with Accelerate Framework optimizations on Apple Silicon](#recipe-for-building-numpy-with-accelerate-framework-optimizations-on-apple-silicon) - [Purpose](#purpose) - [Explanation](#explanation) - [NLTK and Token Count](#nltk-and-token-count) @@ -23,9 +39,14 @@ - [Support My Work](#support-my-work) - [License](#license) - [Future Improvements](#future-improvements) + - [High Priority](#high-priority) + - [Performance and Security](#performance-and-security) + - [Tools and Integration](#tools-and-integration) - [Recent Improvements](#recent-improvements) - - [Setup Script Enhancements](#setup-script-enhancements-1) - - [Shell Function Improvements](#shell-function-improvements) + - [Core Functionality](#core-functionality) + - [Library Organization](#library-organization) + - [Performance Tools](#performance-tools) + - [Documentation](#documentation) ## Project Overview @@ -51,6 +72,13 @@ Venvutil is a versatile toolset designed to simplify the management of Python vi - **Ensure Reproducibility**: By freezing the state of environments, Venvutil helps ensure that your setups are consistent across different machines and setups. - **Transparent Tool Wrapping**: Provides pass-through wrappers Pip, Conda, and LD to ensure compatibility without compromising security or functionality. +### Tested on the following systems + +- macOS 15.3 (Sequoia) +- macOS 13.4 (Monterey) +- Red Hat Enterprise Linus 8 +- Redhat Enterprise Linux 9 + ## Installation Instructions ### Prerequisites @@ -73,8 +101,6 @@ bash ./setup.sh install By default this installs in $HOME/local/venvutil. You can override this with the -d flag. To any location you wish. The installer will download and update Conda if necessary, along with the python packages listed above. NLTK needs data and that will be downloaded into your home directory into the nltk_data directory. -More updates will come in the next few days. - Thanks for using Venvutil! ## Setup Script Enhancements @@ -88,6 +114,10 @@ Thanks for using Venvutil! ### Tools Overview +- **extract-chat** extracts ChatGPT JSON chatlogs, works with my [Safari extension](https://github.com/unixwzrd/chatgpt-chatlog-export), to extract chat history. + - Extract in either Markdown or HTML format. + - Retains code and references where possible along with some internal metadata. + - May be broken into chunks and fed into a fresh GPT context for continuity. - **tokencount**: [Detailed Documentation](docs/tokencount.md) *TODO* - A tool designed to count tokens in text files, useful for analyzing text data and preparing it for processing with language models. - **chunkfile**: [Detailed Documentation](docs/chunkfile.md) @@ -105,7 +135,7 @@ Thanks for using Venvutil! - A script that generates markdown documentation from project files, facilitating easy sharing and collaboration. - **filetree**: [Detailed Documentation](docs/filetree.md) - will produce file hierarchy structure based on file and directories to exclude and include.. -- **core functions provided by init_env.sh**: [Detailed Documentation](docs/shdoc/README.md) +- **core functions provided by venvutil_lib.sh**: [Detailed Documentation](docs/shdoc/README.md) - Provides a number of useful shell functions for managing aVirtual Environments along with some utility function, such as `ptree` - **compile wrappers for C++, G++, and LD**: [Detailed Documentation](docs/compile_wrappers.md) - To help compile many things in the macOS Environment which incorrectly pass the linker the --version flag. @@ -114,6 +144,8 @@ Thanks for using Venvutil! These are a few of the shell functions provided by venvutil which I find useful. There is more documentation on the functions in the README of the [venvutil Tools](docs/shdoc/README.md). +To use the functions and tools, simply source in the venvutil_lib.sh file in your .bashrc. The setup.sh script will handle adding the necessary checks and source statements to your .bashrc file. + - **venvutil Tools**: [Detailed Documentation](docs/shdoc/README.md) - A collection of shell functions and scripts for managing Python virtual environments and LLMs. - **vhelp**: [Detailed Documentation](docs/shdoc/bin/shinclude/functions/vhelp.md) @@ -163,15 +195,19 @@ This logging combined with the frozen environments can be used to ensure that yo Configuration options, logs and freezes are found in the `$HOME` -## ~~C++, G++, and~~ LD Pass-Through +## LD Pass-Through and NumPy builds Meson was fixed which gave me troubles tracking this down, so I am removing the hard links for c++ and g++, but leaving in the ld script pass-through just in case something else tries to invoke it using the wrong flag for `--version` when it needs to be `-v`, here are the instructions for building NumPy with the optimizations turned on. It also seems that after I built GCC, it conflicted with the Xcode c++ compiler, installing another c++ in /usr/local/bin which was simply a herd link to g++. +### Recipe for building NumPy with Accelerate Framework optimizations on Apple Silicon + +This has also been placed in the `numpy-comp` script, just specify version of NumPy you want to build. + ```bash CFLAGS="-I/System/Library/Frameworks/vecLib.framework/Headers -Wl,-framework -Wl,Accelerate -framework Accelerate" pip install numpy==1.26.* --force-reinstall --no-deps --no-cache --no-binary :all: --no-build-isolation --compile -Csetup-args=-Dblas=accelerate -Csetup-args=-Dlapack=accelerate -Csetup-args=-Duse-ilp64=true ``` -This will build and install NumPy 1.26 into your Python virtual environment. With the Accelerate Framework optimizations on, you can now use NumPy with Apple Silicon. +This will build and install NumPy 1.26 into your Python virtual environment. With the Accelerate Framework optimizations on, you can now use NumPy with Apple Silicon. The `numpy-comp` script will take of all teh details. There are several test scripts for NumPy and PyTorch which may be used to compare different builds for performance, these van run on multiple virtual environments for varies size NumPy arrays, PyTorch tensors and varying iterations. These are useful for seeing what combinations of packages will give the best performance. ### Purpose @@ -188,12 +224,12 @@ This will build and install NumPy 1.26 into your Python virtual environment. Wit ## Recent Changes -- **Logging Enhancements**: Improved logging with dynamic program names and lazy formatting. +- **Logging Enhancements**: (more) Improved logging with dynamic program names and lazy formatting. - **Configuration Management**: Introduced global variable declarations and robust configuration handling in `genmd`. ## Project Status -The project is actively maintained and continuously evolving with new features and improvements. +The project is actively maintained and continuously evolving with new features and improvements. Check out the CHANGELOG.md and teh project TODO.md list for more details. ## Support My Work @@ -207,7 +243,7 @@ Thank you for your support! This project is licensed under the Apache License Version 2.0, License. - Copyright 2024 Michael P. Sullivan - unixwzrd@unixwzrd.ai + Copyright 2025 Michael P. Sullivan - unixwzrd@unixwzrd.ai Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -226,36 +262,47 @@ This project is licensed under the Apache License ## Future Improvements -- **Chunkfile Enhancements**: - - Add support for custom chunk naming patterns - - Add compression support for output chunks - - Add support for automatic chunk size calculation based on available memory -- **l processing of chunks Virtual Environment Tools**: Continue to expand the collection of tools for managing virtual environments. - - High on the list is `vinfo`, `venvdiff`, `vlog` -- **Additional Documentation**: Expand the documentation to include more examples and examples of using the tools. -- **Overall Enhancements**: Additional improvements and documentation are needed, but focus is shifting to other projects for now. +### High Priority +- **Testing Framework**: Comprehensive testing suite for shell functions, including unit tests and integration tests +- **Documentation**: Enhanced function reference, troubleshooting guides, and architecture documentation +- **Core Functionality**: Standard package sets for new Virtual Environments and improved package management + +### Performance and Security +- **Security Enhancements**: Improved permission handling and secure configuration options +- **Performance Testing**: Enhanced NumPy/PyTorch testing tools with visualization and metrics +- **Optimization**: Improved file handling and parallel processing capabilities + +### Tools and Integration +- **Chat Tools**: Enhanced conversation analytics and metadata extraction +- **User Interface**: Interactive modes and improved progress reporting +- **Integration**: Enhanced container support and remote environment management + +For a complete list of planned improvements, see our [TODO.md](TODO.md). ## Recent Improvements -### Setup Script Enhancements - -- **Error Handling**: Enhanced error handling with proper exit codes and validation -- **Configuration**: Improved package configuration management and logging -- **Hard Links**: Added support for hard link creation in manifest -- **Rollback**: Added framework for installation rollback capability -- **Manifest**: Enhanced manifest handling and validation - -### Shell Function Improvements - -- **Virtual Environment Management**: - - Enhanced environment variable handling - - Improved error recovery mechanisms - - Better logging for venv operations -- **Error Handling**: - - Improved POSIX errno codes with better formatting - - Enhanced error message categorization - - Added detailed debugging context -- **Help System**: - - Added new documentation path handling - - Updated function naming conventions - - Enhanced help message formatting +### Core Functionality +- Enhanced virtual environment management with `lenv` Python version display +- Added environment renaming capability with `renv` +- Improved cloning functionality in `ccln` +- Enhanced logging and configuration management + +### Library Organization +- Renamed shell libraries to use `_lib.sh` suffix for better clarity +- Created specialized libraries for different functionalities +- Enhanced error handling and type checking +- Improved help system and initialization routines + +### Performance Tools +- Added PyTorch and NumPy stress testing tools +- Implemented compilation tools and benchmarking +- Enhanced chat management utilities +- Improved documentation generation + +### Documentation +- Added comprehensive performance metrics documentation +- Updated coding standards and file structure documentation +- Enhanced installation guide and function documentation +- Added migration guide for version 20250206-00_R1 + +For a complete list of changes, see our [CHANGELOG.md](CHANGELOG.md). diff --git a/TODO.md b/TODO.md new file mode 100644 index 00000000..ebaa4448 --- /dev/null +++ b/TODO.md @@ -0,0 +1,143 @@ +# Consolidated Development TODO List + +## High Priority + +### Documentation +- [ ] Implement automated documentation validation +- [ ] Create documentation test suite +- [ ] Add automated cross-reference checking + +### Process Improvements +- [ ] Create automated changelog generation from git commits +- [ ] Implement documentation linting +- [ ] Create documentation coverage reports + +### Testing Framework +- [ ] Create comprehensive testing framework for shell functions + - [ ] Implement unit tests for core utilities + - [ ] Test string manipulation functions + - [ ] Test type checking and validation + - [ ] Test error handling functions + - [ ] Test configuration management + - [ ] Test virtual environment management functions + - [ ] Test environment creation/deletion + - [ ] Test cloning operations + - [ ] Test package installation/removal + - [ ] Test environment activation/deactivation + - [ ] Test file chunking operations + - [ ] Test different chunk sizes + - [ ] Test overlap functionality + - [ ] Test error conditions + - [ ] Test manifest generation + - [ ] Test file type detection + - [ ] Test permission handling + - [ ] Test checksum generation + - [ ] Add integration tests for tool interactions + - [ ] Test pip wrapper functionality + - [ ] Test conda wrapper functionality + - [ ] Test environment variable handling + - [ ] Implement regression test suite + - [ ] Create baseline test cases for all core functions + - [ ] Test backward compatibility + - [ ] Test cross-platform functionality + - [ ] Automated test runs before commits + - [ ] Test result comparison and reporting + - [ ] Create test data generators + - [ ] Generate sample virtual environments + - [ ] Create test package sets + - [ ] Generate test configuration files + +### Core Functionality +- [ ] Add standard package sets for new Virtual Environments + - [ ] Implement in Pip and Conda wrappers + - [ ] Allow user-defined package sets +- [ ] Enhance Virtual Environment Management + - [ ] Implement comprehensive venv comparison (vdiff): + - [ ] Compare two different venvs + - [ ] Track changes in a venv over time + - [ ] Compare by date/timestamp + - [ ] Generate detailed diff reports + - [ ] Improve rollback and recovery: + - [ ] Implement point-in-time recovery + - [ ] Add snapshot functionality + - [ ] Track package state changes + - [ ] Provide rollback preview + - [ ] Add Python version management: + - [ ] Support upgrading Python version in existing venv + - [ ] Clone venv with different Python version + - [ ] Validate package compatibility during upgrade + - [ ] Migration testing support + - [ ] Requirements management: + - [ ] Generate project-specific requirements + - [ ] Track dependency changes over time + - [ ] Support different requirement formats (pip, conda) + - [ ] Dependency conflict detection +- [ ] Implement integrity checking using checksums from manifest +- [ ] Develop repair function for permissions/ownership +- [ ] Package removal functionality + - [ ] Remove package files, configuration and logs + - [ ] Option for complete removal including dependencies + +## Medium Priority + +### Security Enhancements +- [ ] Add checksum verification for file operations +- [ ] Implement better permission handling +- [ ] Add secure configuration options +- [ ] Add validation for external tool inputs + +### Performance Testing and Optimization +- [ ] Enhance NumPy/PyTorch testing tools + - [ ] Add more performance metrics + - [ ] Create visualization tools for test results +- [ ] Optimize large file handling in chunkfile + - [ ] Implement streaming for large files + - [ ] Add memory usage monitoring +- [ ] Improve manifest generation performance +- [ ] Profile and optimize core functions +- [ ] Reduce startup time + +### Chat Tools +- [ ] Add search functionality +- [ ] Add tag-based organization +- [ ] Enhance metadata extraction +- [ ] Implement token-based file splitting + +## Low Priority + +### User Interface +- [ ] Add command completion for shells +- [ ] Add dry-run mode for destructive operations + +### Integration Features +- [x] Implement CI/CD pipeline with pre-commit actions +- [ ] Add package manager integration beyond pip/conda +- [ ] Add support for project-specific configurations +- [ ] Set up automated dependency updates + +## Completed Tasks ✓ + +- [x] Create shell script to automate Conda environment setup and package installation +- [x] Develop documentation generation system +- [x] Enhance error handling and logging across all scripts +- [x] Implement more extensive logging mechanisms +- [x] Implement git integration for manifest generation +- [x] Add support for automatic detection of deleted files +- [x] Improve documentation for manifest file format +- [x] Add changelog entries for recent updates +- [x] Generate manifests based on specifying branch against working tree +- [x] Change `ccln` to clone current without sequence number +- [x] Add/change `nenv`/`benv` to create VENV without sequence number +- [x] Add support for upgrades using Git +- [x] Document manifest layout and parsing logic +- [x] Establish standardized documentation format +- [x] Create periodic review process +- [x] Implement documentation workflow +- [x] Add comprehensive cross-referencing +- [x] Create visual guides for complex operations + +## Notes +- Focus on core virtual environment management functionality +- Prioritize stability and reliability over new features +- Keep documentation current and accurate +- Regular testing and validation is essential \ No newline at end of file diff --git a/bin/buildvenvs b/bin/buildvenvs index 6d557549..1a4c3a66 100755 --- a/bin/buildvenvs +++ b/bin/buildvenvs @@ -11,11 +11,13 @@ MY_INCLUDE="${MY_BIN}/shinclude" MY_CONF_DIR="${MY_DIR}/conf" MY_CONFIG=$1 # Source in some useful shortcut functions for VENV management. -[ -f "${MY_INCLUDE}/init_env.sh" ] && . "${MY_INCLUDE}/init_env.sh" \ - || ( echo "${MY_NAME}: Could not find init_env.sh in INCLUDEDIR: ${MY_INCLUDE}"; exit 1 ) +# shellcheck source=/dev/null +[ -f "${MY_INCLUDE}/venvutil_lib.sh" ] && . "${MY_INCLUDE}/venvutil_lib.sh" \ + || ( echo "${MY_NAME}: Could not find venvutil_lib.sh in INCLUDEDIR: ${MY_INCLUDE}"; exit 1 ) umask 022 +# shellcheck disable=SC2206 __VENV_INTERNAL_FUNCTIONS=( ${__VENV_INTERNAL_FUNCTIONS[@]} "__BLIS" @@ -39,21 +41,21 @@ CONDA_ROOT=${_CONDA_ROOT:-$(conda info -s | grep CONDA_ROOT | cut -d " " -f 2)} APP_CODE="bld" # Set the build order and the packages to install/reinstall -#ACKAGE_INSTALL=( +#PACKAGE_INSTALL=( # "numpy" # "pytorch" # "numpy" # "webui" -# "rnumpy" +# "numpy" # "llama-cpp-python" -# "rnumpy" +# "numpy" # ) PI_PACKAGE=0 -PI_INSTYPE=1 +#PI_INSTYPE=1 -# Set up th edefault build sequence and configuration for the VENV's. +# Set up the default build sequence and configuration for the VENV's. # -#ACKAGE_CONFIG=( +#PACKAGE_CONFIG=( # CONFIG | DESCRIPTION | PACKAGE | ALT_NAME | PRE_FLAGS | METHOD| POST_FLAGS # "base | Standard Pip install | numpy | | | pip | --force-reinstall" # "conda | Standard Conda package install | numpy | | | conda | -y" @@ -62,10 +64,10 @@ PI_INSTYPE=1 # "oblas | Pip Install recompile using OpenBLAS | numpy | | NPY_BLAS_ORDER='openblas' NPY_LAPACK_ORDER='openblas' | pip | --force-reinstall --no-cache --no-binary :all: --compile" # "blis | Pip Install recompile using BLIS | numpy | | NPY_BLAS_ORDER='blis' | pip | --force-reinstall --no-cache --no-binary :all: --compile" # "blisblas | Pip Install recompile using BLIS for OpenBLAS and BLAS for LAPACK | numpy | | NPY_BLAS_ORDER='blis' NPY_LAPACK_ORDER='openblas' | pip | --force-reinstall --no-cache --no-binary :all: --compile" -# "base | Stangard conda install for PyTorch | pytorch | | | conda | torchvision -c pytorch nstal" -# "pip | Stangard pip install for Torch | pytorch | torch | | pip | torchvision --force-reinstall --nodeps" +# "base | Standard conda install for PyTorch | pytorch | | | conda | torchvision -c pytorch install" +# "pip | Standard pip install for Torch | pytorch | torch | | pip | torchvision --force-reinstall --no-deps" # "base | Oobabooga install | webui | | | git | clone https://github.com/oobabooga/text-generation-webui.git webui" -# "base | Stangard pip install for Torch | llama-cpp-python | | CMAKE_ARGS='--fresh -DLLAMA_METAL=ON' FORCE_CMAKE=1 | pip | --force-reinstall --no-cache --no-binary :all: --compile llama-cpp-python" +# "base | Standard pip install for Torch | llama-cpp-python | | CMAKE_ARGS='--fresh -DLLAMA_METAL=ON' FORCE_CMAKE=1 | pip | --force-reinstall --no-cache --no-binary :all: --compile llama-cpp-python" # ) PC_CONFIG=0 PC_DESC=1 @@ -89,9 +91,11 @@ __BUILD_BASE="${BUILD_BASE:=${PWD}}" echo "${MY_NAME}: Argiuments passed '${MY_ARGS}'" echo "${MY_NAME}: Sourcing in user config- '${MY_CONF_DIR}/config-${MY_CONFIG}.sh'" +# shellcheck disable=SC1090 [ -e "${MY_CONF_DIR}/config-${MY_CONFIG}.sh" ] && . "${MY_CONF_DIR}/config-${MY_CONFIG}.sh" \ || echo "${MY_NAME}: File not found - '${MY_CONF_DIR}/config-${MY_CONFIG}.sh'" +# shellcheck disable=SC2206 __VENV_INTERNAL_FUNCTIONS=( ${__VENV_INTERNAL_FUNCTIONS[@]} "__CONDA" @@ -109,9 +113,7 @@ __VENV_INTERNAL_FUNCTIONS=( __BUILD_DIR="${BUILD_DIR:=${__BUILD_BASE}/build}" __BENCHMARK_DIR="${__BUILD_DIR}/benchmark" -[ -e "${__BUILD_BASE}" ] || mkdir -p "${__BUILD_BASE}" -[ -e "${__BUILD_DIR}" ] || mkdir -p "${__BUILD_DIR}" -# [ -e "${__BENCHMARK_DIR}" ] || mkdir -p "${__BENCHMARK_DIR}" +mkdir -p "${__BUILD_BASE}" "${__BUILD_DIR}" "${__BENCHMARK_DIR}" echo "${MY_NAME}: Setting build Directory to: ${__BUILD_DIR}" __LASTENV_FILE="${__BUILD_DIR}/.lastvenv" @@ -125,18 +127,16 @@ __INTERNAL_STEPS=( "__CREATE_APP_BASE" ) - - - # Conda install +# Conda install __CONDA() { - cd ${__BUILD_DIR} - [ -e ./tmp ] || mkdir tmp + cd "${__BUILD_DIR}" + mkdir -p tmp cd tmp curl https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh -o miniconda.sh - # Do a non-destructive Conda install whcih will preserve existing VENV's + # Do a non-destructive Conda install which will preserve existing VENV's sh miniconda.sh -b -u . "${HOME}/miniconda3/bin/activate" - conda init $(basename "${SHELL}") + conda init "$(basename "${SHELL}")" conda update -n base -c defaults conda -y # Replace the shell name below with your preferred shell. The -l switch gives you a login shell # and, contrary to what you may heard, you don't have to log out or exit the terminal. Simply @@ -144,10 +144,11 @@ __CONDA() { # set. This also works in Linux and most other Unix-like POSIX operating systems. export _CONDA_ROOT cd "${__BUILD_BASE}" + rm -rf tmp # Since the code is re-entrant, we want to make sure we don't re-enter this function. # this function will never return so it will never get back to the case statement and # write the step number out. I've put that part in the script, but it will never reach that - # part. If we don't this will spawn more bash shells, ythan you want to imagine. I'm considering + # part. If we don't this will spawn more bash shells, than you want to imagine. I'm considering # setting ulimit to something like 500 echo "1" > "${__STEP_FILE}" echo "${MY_NAME}: Re-running in new Conda environment" @@ -197,13 +198,13 @@ __BLIS() { git clone "${URL_BLIS}" mkdir -p blis/build-blis cd blis/build-blis - # The ARMv1.2 ThunderX2 (NEON) seems to be supported, though this is missing soem of the features + # The ARMv1.2 ThunderX2 (NEON) seems to be supported, though this is missing some of the features # in the M1 and M2 families. # # M1 ARMv8.4-A 64-bit "Firestorm+Icestorm" (NEON) # M2 ARMv8.6-A 64-bit "Avalanche+Blizzard" (NEON) # - # ARMv8.1 ThunderX2 (NEON) seens to work. + # ARMv8.1 ThunderX2 (NEON) seems to work. # # The configuration for ARMv8.1AFX (SVE) is not supported and fails testing. I could not find out # much more information on this architecture. @@ -242,12 +243,13 @@ __PKG_BUILD() { __PREV_VENV=$(cat "${__LASTENV_FILE}") # Set the current active environment. cact "${__PREV_VENV}" - # Current environment clone using the PACKAGE name passed and amke it active. + # Current environment clone using the PACKAGE name passed and make it active. ccln "${__BUILD_PACKAGE}" # Preserve the name of teh new current environment. last_build_env=${CONDA_DEFAULT_ENV} # Loop through the keys and values in PACKAGE_CONFIG and if teh match the BUILD_PACKAGE, - # build them using the attributes in the arary. + # build them using the attributes in the array. + # shellcheck disable=SC2068 for i in ${!PACKAGE_CONFIG[@]}; do IFS='|' read -ra DETAILS <<< "${PACKAGE_CONFIG[$i]}" # Strip leading and trailing whitespace as this causes a problem with the name check @@ -256,7 +258,7 @@ __PKG_BUILD() { echo "${MY_NAME}: PACKAGE TO BUILD: ${__PACKAGE}" echo "${MY_NAME}: PACKAGE TO MATCH: ${__PACKAGE} <=> ${__BUILD_PACKAGE}" if [ "${__PACKAGE}" == "${__BUILD_PACKAGE}" ]; then - # Set current active VENV to the alst VENV built as it will be th estart of + # Set current active VENV to the last VENV built as it will be the start of # the branch each time. cact "${last_build_env}" echo "${MY_NAME}: BUILDING PACKAGE: ${__PACKAGE} BUILD_PACKAGE: ${__BUILD_PACKAGE}" @@ -273,13 +275,13 @@ __PKG_BUILD() { case "${__METHOD}" in pip ) echo "${MY_NAME}: PIP INSTALL SELECTED" - # Create a clone of the current envirnment, this will also increment the sequence number. + # Create a clone of the current environment, this will also increment the sequence number. ccln "${__VENV_BUILD}" __COMMAND="${__PRE_FLAGS} ${__METHOD} install ${__POST_FLAGS} ${__PACKAGE}" ;; conda ) echo "${MY_NAME}: CONDA INSTALL SELECTED" - # Create a clone of the current envirnment, this will also increment the sequence number. + # Create a clone of the current environment, this will also increment the sequence number. ccln "${__VENV_BUILD}" __COMMAND="${__PRE_FLAGS} ${__METHOD} install ${__PACKAGE} ${__POST_FLAGS}" ;; @@ -301,10 +303,11 @@ __PKG_BUILD() { echo "${MY_NAME}: Current Conda default VENV: ${CONDA_DEFAULT_ENV}" echo "${MY_NAME}: DESCRIPTION: ${__DESCRIPTION}" echo "${MY_NAME}: COMMAND: ${__COMMAND}" - eval "${__COMMAND}" || { echo "${MY_NAME}: Abnormal terminaion: ${__COMMAND}"; - echo "${MY_NAME}: Fix and restart, recoverable and restartable."; - exit 6 - } + eval "${__COMMAND}" || { + echo "${MY_NAME}: Abnormal termination: ${__COMMAND}"; + echo "${MY_NAME}: Fix and restart, recoverable and restartable."; + exit 6 + } fi done # Write the base checkpoint VENV name to a state file so we are restartable. @@ -320,10 +323,11 @@ __CLEANUP() { # MAIN SCRIPT # Combine internal steps with package builds __BUILD_STEPS=("${__INTERNAL_STEPS[@]}") +# shellcheck disable=SC2068 for pkg in ${!PACKAGE_INSTALL[@]}; do IFS='|' read -ra DETAILS <<< "${PACKAGE_INSTALL[$pkg]}" # Strip leading and trailing whitespace as this causes a problem with the name check - __PI_PACKAGE=$(strip_space ${DETAILS[${PI_PACKAGE}]}) + __PI_PACKAGE=$(strip_space "${DETAILS[${PI_PACKAGE}]}") echo "${MY_NAME}: ADDING PACKAGE: ${pkg} -- ${__PI_PACKAGE}" __BUILD_STEPS=("${__BUILD_STEPS[@]}" "__PKG_BUILD ${__PI_PACKAGE} ${DETAILS[${PI_INSTYP}]}") done @@ -335,7 +339,7 @@ for (( step_num=${stepnum}; step_num<${#__BUILD_STEPS[@]}; step_num++)); do step=${__BUILD_STEPS[$step_num]} echo "${MY_NAME}: ENTERING BUILD STEP FOR - ${step}" echo "${MY_NAME}: Calling package installer for: ${step}" - # Ensure we are always in the build directory befotre doing any step - SAFETY + # Ensure we are always in the build directory before doing any step - SAFETY cd "${__BUILD_DIR}" eval "${step}" || exit 1 echo $((i + step_num)) > "${__STEP_FILE}" diff --git a/bin/compare_test b/bin/compare_test new file mode 100755 index 00000000..98ec3527 --- /dev/null +++ b/bin/compare_test @@ -0,0 +1,48 @@ +#!/usr/bin/env bash + +# Example: run both scripts with different matrix sizes +# and iteration counts, logging to unique files. + +. $HOME/.bashrc + +run_tests() { + size=$1 + iter=$2 + venv=$3 + + StartTime=$(date +%Y-%m-%d\ %H:%M:%S) + + #echo "===== Numpy Benchmark: size=$size, iter=$iter =====" + # numpy_torture --size "$size" --iterations "$iter" --output "numpy_s${size}_i${iter}.log" + numpy_torture --size "$size" --iterations "$iter" + + #echo "===== Torch Benchmark CPU: size=$size, iter=$iter =====" + # torch_torture --device cpu --size "$size" --iterations "$iter" --output "torch_cpu_s${size}_i${iter}.log" + # torch_torture --device cpu --size "$size" --iterations "$iter" + + #echo "===== Torch Benchmark GPU (if available): size=$size, iter=$iter =====" + # torch_torture --device cuda --size "$size" --iterations "$iter" --output "torch_gpu_s${size}_i${iter}.log" + torch_torture --device auto --size "$size" --iterations "$iter" + + EndTime=$(date +%Y-%m-%d\ %H:%M:%S) + echo "Start Time: ${StartTime}, End Time: ${EndTime}" + +} + +export -f run_tests + +for size in 500 1000 1500 2000 2500; do + #for size in 500 1000 ; do + for iter in 5 10 15 20 25; do + #for iter in 5 10 ; do + for venv in comp-numpy-daily-pytorch adv-numpy-daily-pytorch; do + if [ "${venv}" != "${prev_venv}" ]; then + cact ${venv} + fi + + /usr/bin/time -a bash -c "run_tests ${size} ${iter} ${venv}" + + prev_venv=${venv} + done + done +done diff --git a/bin/extract-chat b/bin/extract-chat new file mode 120000 index 00000000..05c99e10 --- /dev/null +++ b/bin/extract-chat @@ -0,0 +1 @@ +extract_chat.py \ No newline at end of file diff --git a/bin/extract_chat.py b/bin/extract_chat.py new file mode 100755 index 00000000..a8e21d98 --- /dev/null +++ b/bin/extract_chat.py @@ -0,0 +1,1027 @@ +#!/usr/bin/env python +""" +This script extracts conversation logs from JSON files and saves them as Markdown or HTML files. + +The script processes JSON files containing chat conversations and converts them to either Markdown +or HTML format while preserving the conversation structure, timestamps, and special message types. + +Key features: +- Supports both Markdown and HTML output formats +- Handles system, user, assistant and tool messages +- Preserves message timestamps and ordering +- Processes code blocks and embedded markdown +- Generates unique filenames based on conversation metadata +- Supports batch processing of multiple files + +Example usage: + python extract_chat.py input.json --format html --output-dir ./output/ + python extract_chat.py ./chats/*.json --format markdown +""" + +import argparse +import glob +import html +import json +import os +import re +import unicodedata +from datetime import datetime +from typing import Dict, List, Optional, Tuple + +import ftfy +import mistune + +# Date/time formats +FILENAME_DATE_FORMAT = "%Y-%m-%d-%H%M%S" +DATETIME_FORMAT = "%Y-%m-%d %H:%M:%S" +VALID_MESSAGE_ROLES = {"assistant", "system", "user"} +HTML_TEMPLATE_HEADER = """ + + + + {title} + + + +

{title}

+

Starting: {start_time}
Ending: {end_time}

+""" +HTML_TEMPLATE_FOOTER = """""" + + +def fix_timestamp(ts: Optional[float]) -> Optional[float]: + """ + Fix overly large timestamps by moving the decimal point to get a valid Unix timestamp. + + Args: + ts: Input timestamp that may need fixing + + Returns: + Fixed timestamp in Unix timestamp range (~10 digits), or None if input was None + + Example: + >>> fix_timestamp(1234567890) # Already valid + 1234567890 + >>> fix_timestamp(1234567890123) # Too large + 1234567.890123 + """ + if ts is None: + return None + # Convert to string to count digits before decimal + ts_str = str(float(ts)) + whole_digits = ts_str.split('.', maxsplit=1)[0] + # If more than 10 digits before decimal, adjust + if len(whole_digits) > 10: + power = len(whole_digits) - 10 + return float(ts) / (10**power) + return float(ts) + + +def format_timestamp(ts: Optional[float]) -> str: + """ + Format a numeric timestamp into a human-readable date/time string. + + Args: + ts: Unix timestamp to format + + Returns: + Formatted string in YYYY-MM-DD HH:MM:SS format, or empty string if invalid + + Example: + >>> format_timestamp(1234567890) + '2009-02-13 23:31:30' + >>> format_timestamp(None) + '' + """ + ts_fixed = fix_timestamp(ts) + if ts_fixed is None: + return "" + dt = datetime.fromtimestamp(ts_fixed) + return dt.strftime(DATETIME_FORMAT) + + +def parse_datetime_string(dt_str: str) -> datetime: + """ + Parse a datetime string in YYYY-MM-DD HH:MM:SS format. + + Args: + dt_str: Datetime string to parse + + Returns: + Parsed datetime object, or current time if parsing fails + + Example: + >>> parse_datetime_string('2023-01-01 12:00:00') + datetime.datetime(2023, 1, 1, 12, 0) + """ + try: + return datetime.strptime(dt_str, DATETIME_FORMAT) + except ValueError: + return datetime.now() + + +def sanitize_title(title: str) -> str: + """ + Sanitize a title string for use in filenames. + + Args: + title: Input title string + + Returns: + Sanitized title with problematic characters replaced by underscores + + Example: + >>> sanitize_title('Hello, World!') + 'Hello_World' + >>> sanitize_title('') + 'untitled' + """ + s = re.sub(r"[^\w\-.]", "_", title) + s = re.sub(r"_+", "_", s).strip("._-") + return s or "untitled" + + +def generate_unique_filename( + input_file: str, + title: str, + create_time: Optional[float], + update_time: Optional[float], + extension: str = "json", + out_dir: Optional[str] = None +) -> str: + """ + Generate a unique filename for the output file. + + Args: + input_file: Original input filename + title: Title for the output file + create_time: Creation timestamp + update_time: Last update timestamp + extension: File extension (default: 'json') + out_dir: Optional output directory + + Returns: + Generated unique filename in format: YYYY-MM-DD-HHMMSS_YYYY-MM-DD-HHMMSS-TITLE.ext + + Raises: + ValueError: If create_time is invalid or too many filename collisions occur + + Example: + >>> generate_unique_filename('input.json', 'Test', 1234567890, 1234567890) + '2009-02-13-233130_2009-02-13-233130-Test.json' + """ + # Fix both timestamps + ctime_fixed = fix_timestamp(create_time) + utime_fixed = fix_timestamp(update_time) + + if ctime_fixed is None: + raise ValueError("Invalid or missing create_time") + + # If update_time is missing, use create_time + if utime_fixed is None: + utime_fixed = ctime_fixed + + # Convert each to string + ctime_str = datetime.fromtimestamp(ctime_fixed).strftime(FILENAME_DATE_FORMAT) + utime_str = datetime.fromtimestamp(utime_fixed).strftime(FILENAME_DATE_FORMAT) + cleaned_title = sanitize_title(title) + + dir_path = out_dir if out_dir else os.path.dirname(input_file) + os.makedirs(dir_path, exist_ok=True) + + base_name = f"{ctime_str}_{utime_str}-{cleaned_title}" + filename = os.path.join(dir_path, f"{base_name}.{extension}") + + counter = 0 + while os.path.exists(filename): + counter += 1 + if counter > 99: + raise ValueError(f"Too many duplicates for {filename}") + filename = os.path.join(dir_path, f"{base_name}-{counter:02d}.{extension}") + return filename + + +def load_json_file(file_path: str) -> Optional[Dict]: + """ + Load and parse a JSON file. + + Args: + file_path: Path to JSON file to load + + Returns: + Parsed JSON data as dict, or None if loading/parsing fails + + Example: + >>> data = load_json_file('valid.json') + >>> type(data) + + >>> load_json_file('invalid.json') + None + """ + try: + with open(file_path, "r", encoding="utf-8") as f: + data = json.load(f) + if isinstance(data, dict): + return data + print(f"Error: {file_path} did not contain a JSON object.") + except Exception as e: + print(f"Error reading JSON from {file_path}: {e}") + return None + + +def is_tool_message(message_role: str) -> bool: + """ + Check if a message role represents a tool message. + + Args: + message_role: Message role string to check + + Returns: + True if role is not one of the standard roles (user/system/assistant) + + Example: + >>> is_tool_message('user') + False + >>> is_tool_message('code_interpreter') + True + """ + return message_role not in VALID_MESSAGE_ROLES + + +def normalize_newlines(input_text: str) -> str: + """ + Normalize multiple consecutive newlines to single newlines. + + Args: + input_text: Input text to normalize + + Returns: + Text with consecutive newlines collapsed to single newlines + + Example: + >>> normalize_newlines('line1\\n\\n\\nline2') + 'line1\\nline2' + """ + return re.sub(r"\n{2,}", "\n", input_text) + + +# ----------------------------------------------------------------------------- +# Markdown Utilities +# ----------------------------------------------------------------------------- +def detect_markdown(input_text: str) -> List[Tuple[int, int, str]]: + """ + Detect markdown segments within text using Mistune parser. + + Args: + input_text: Input text to analyze for markdown + + Returns: + List of tuples containing: + - Start position in text + - End position in tex + - Raw markdown content + + Example: + >>> detect_markdown('Normal text **bold** more text') + [(12, 18, '**bold**')] + """ + markdown_segments: List[Tuple[int, int, str]] = [] + current_position = 0 + + try: + markdown_parser = mistune.create_markdown() + abstract_syntax_tree = markdown_parser.parse(input_text) + + def process_token(token): + nonlocal current_position + if not isinstance(token, dict): + return + + # If 'raw' is present, we can attempt to locate it in the original text. + if token.get("type") != "text" and "raw" in token: + raw_markdown = token["raw"] + start_index = input_text.find(raw_markdown, current_position) + if start_index != -1: + end_index = start_index + len(raw_markdown) + markdown_segments.append((start_index, end_index, raw_markdown)) + current_position = end_index + + # Recurse over children + for child_token in token.get("children", []): + process_token(child_token) + + for token in abstract_syntax_tree: + process_token(token) + + return sorted(markdown_segments, key=lambda x: x[0]) + except Exception as error: + print(f"Warning: Error parsing markdown: {error}") + return [] + + +def format_text_block(text_content: str, output_format: str) -> List[str]: + """ + Format a text block for output in HTML or Markdown. + + Args: + text_content: Text content to format + output_format: Output format ('html' or 'markdown') + + Returns: + List of formatted text lines + + Example: + >>> format_text_block('Hello **world**', 'html') + ['

Hello world

\\n', '\\n\\n'] + """ + formatted_lines = [] + if output_format == "html": + html_content = mistune.create_markdown( + plugins=["strikethrough", "footnotes", "table"], escape=True + )(text_content) + formatted_lines.append(html_content) + else: + formatted_lines.append(text_content) + formatted_lines.append("\n\n") + return formatted_lines + + +def format_code_block( + code_content: str, programming_language: str, output_format: str +) -> List[str]: + """ + Format a code block for output in HTML or Markdown. + + Args: + code_content: Code content to format + programming_language: Programming language for syntax highlighting + output_format: Output format ('html' or 'markdown') + + Returns: + List of formatted code block lines + + Example: + >>> format_code_block('print("hello")', 'python', 'markdown') + ['```python\\nprint("hello")\\n```\\n\\n'] + """ + formatted_lines = [] + if output_format == "html": + escaped_code = html.escape(code_content) + formatted_lines.append( + f'
{escaped_code}
\n' + ) + else: + formatted_lines.append(f"```{programming_language}\n{code_content}\n```\n\n") + return formatted_lines + + +# ----------------------------------------------------------------------------- +# Heading and Tool Message Helpers +# ----------------------------------------------------------------------------- +def generate_heading( + message_role: str, timestamp: str, output_format: str +) -> List[str]: + """ + Generate appropriate heading for a message based on role. + + Args: + message_role: Message role (user/system/assistant/tool) + timestamp: Message timestamp string + output_format: Output format ('html' or 'markdown') + + Returns: + List of formatted heading lines + + Example: + >>> generate_heading('user', '2023-01-01 12:00:00', 'markdown') + ['## **USER**\\n\\n', '2023-01-01 12:00:00\\n\\n'] + """ + heading_lines: List[str] = [] + if is_tool_message(message_role): + if output_format == "html": + heading_lines.append( + f'
\nTool Message: {message_role}\n' + ) + if timestamp: + heading_lines.append(f'{timestamp}\n') + else: + heading_lines.append( + f"## **TOOL - {message_role.replace('_', ' ').title()}**\n" + ) + if timestamp: + heading_lines.append(f"{timestamp}\n") + heading_lines.append("
\n\nContents:\n\n\n") + else: + # Standard role headings + if output_format == "html": + heading_lines.append(f"

{message_role.upper()}

\n") + if timestamp: + heading_lines.append(f'{timestamp}\n') + else: + heading_lines.append(f"## **{message_role.upper()}**\n\n") + if timestamp: + heading_lines.append(f"{timestamp}\n\n") + return heading_lines + + +def close_tool_message_block(message_role: str, output_format: str) -> str: + """ + Generate closing tags for tool message blocks if needed. + + Args: + message_role: Message role + output_format: Output format ('html' or 'markdown') + + Returns: + Closing tags string if role is tool message, empty string otherwise + + Example: + >>> close_tool_message_block('code_interpreter', 'html') + '
\\n' + """ + if not is_tool_message(message_role): + return "" + return "
\n" if output_format == "html" else "\n\n" + + +# ----------------------------------------------------------------------------- +# Handling Tool Message Content +# ----------------------------------------------------------------------------- +def process_file_listing(listing_content: str, timestamp: str) -> List[str]: + """ + Process file listing content into details blocks. + + Args: + listing_content: File listing text content + timestamp: Message timestamp string + + Returns: + List of formatted lines with details blocks for each file + + Example: + >>> process_file_listing('- **file.txt**\\ncontents', '2023-01-01') + ['
\\n', '\\n', '- **file.txt**\\n', '2023-01-01\\n', + '\\n\\n', 'contents\\n', '
\\n\\n'] + """ + formatted_lines: List[str] = [] + currently_in_file_block = False + for line in listing_content.split("\n"): + line_stripped = line.strip() + if not line_stripped: + continue + + if line_stripped.startswith("- **") and line_stripped.endswith("**"): + # Start a new details block if needed + if currently_in_file_block: + formatted_lines.append("\n\n") + formatted_lines.append("
\n\n") + formatted_lines.append(f"{line_stripped}\n") + if timestamp: + formatted_lines.append(f"{timestamp}\n") + formatted_lines.append("\n\n") + currently_in_file_block = True + elif currently_in_file_block: + # Continue the listing + formatted_lines.append(f"{line_stripped}\n") + + if currently_in_file_block: + formatted_lines.append("
\n\n") + return formatted_lines + + +def process_tool_content(content_data: Dict, timestamp: str) -> List[str]: + """ + Process specialized tool message content. + + Args: + content_data: Tool message content dictionary + timestamp: Message timestamp string + + Returns: + List of formatted content lines + + The function handles different content types: + - tether_browsing_display: Search results + - tether_quote: Quoted content with title/URL + - Generic tool content in parts + """ + formatted_lines: List[str] = [] + content_type = content_data.get("content_type", "") + + if content_type == "tether_browsing_display": + search_result = content_data.get("result", "") + if search_result: + formatted_lines.append("
\nSearch Results\n\n") + formatted_lines.append(f"{normalize_newlines(search_result)}\n") + formatted_lines.append("
\n\n") + return formatted_lines + + if content_type == "tether_quote": + quote_title = content_data.get("title", "") + quote_url = content_data.get("url", "") + quote_text = content_data.get("text", "") + + if not any([quote_title, quote_url, quote_text]): + return formatted_lines + + formatted_lines.append("
\n\n") + if quote_title: + formatted_lines.append(f"**{quote_title}**\n") + if timestamp: + formatted_lines.append(f"{timestamp}\n") + formatted_lines.append("\n\n") + + if quote_url: + formatted_lines.append(f"Source: {quote_url}\n\n") + if quote_text: + formatted_lines.append(f"{normalize_newlines(quote_text)}\n") + + formatted_lines.append("
\n\n") + return formatted_lines + + # Default: generic tool content in 'parts' + for content_part in content_data.get("parts", []): + if not isinstance(content_part, str) or not content_part.strip(): + continue + formatted_lines.extend( + process_file_listing(normalize_newlines(content_part), timestamp) + ) + + return formatted_lines + + +# ----------------------------------------------------------------------------- +# Main Conversation Processing +# ----------------------------------------------------------------------------- +def build_message_sequence(message_mapping: Dict[str, Dict]) -> List[str]: + """ + Create ordered list of message IDs using stack-based traversal. + + Args: + message_mapping: Dictionary of message data keyed by message ID + + Returns: + List of message IDs in conversation order + + Example: + >>> build_message_sequence({'msg1': {'children': ['msg2']}, 'msg2': {}}) + ['msg1', 'msg2'] + """ + message_sequence: List[str] = [] + visited_messages = set() + + # Root messages have no 'parent' + root_messages = [ + msg_id + for msg_id, msg_data in message_mapping.items() + if not msg_data.get("parent") + ] + message_stack = root_messages[::-1] + + while message_stack: + current_message_id = message_stack.pop() + if current_message_id in visited_messages: + continue + visited_messages.add(current_message_id) + message_sequence.append(current_message_id) + + child_messages = message_mapping[current_message_id].get("children", []) + # Reverse them so they appear in correct order when popped + for child_id in reversed(child_messages): + message_stack.append(child_id) + + return message_sequence + + +def clean_text(text: str) -> str: + """ + Clean text by normalizing using NFC and removing problematic sequences + that optionally start with '0x', followed by 'EE88' and between 2 to 6 hexadecimal digits. + + Args: + text: The input text to clean. + + Returns: + The cleaned text with problematic sequences removed. + """ + # Let ftfy handle known weirdness + text = ftfy.fix_text(text) + + # Normalize to NFC + text = unicodedata.normalize('NFC', text) + + # Optionally remove control chars, private-use chars, etc. + cleaned = [] + for ch in text: + cat = unicodedata.category(ch) + # Keep standard controls like newlines, tabs, and otherwise discard control chars + # Keep everything else unless it’s a private-use area or other undesired code point + if cat.startswith('C'): + # Keep linefeed, carriage return, and tab if you want them + if ch not in ('\n', '\r', '\t'): + continue + # You could add further checks if you don’t want private-use areas, etc. + cleaned.append(ch) + + return ''.join(cleaned) + + +def handle_regular_message(message_content: Dict, output_format: str) -> List[str]: + """ + Process standard message content (code or text). + + Args: + message_content: Message content dictionary + output_format: Output format ('html' or 'markdown') + + Returns: + List of formatted content lines + + The function handles different content types: + - code: Programming code with language specification + - multimodal_text: Text content with multiple parts + - text: Plain text content + """ + formatted_lines: List[str] = [] + content_type = message_content.get("content_type") + + if content_type == "code": + code_text = message_content.get("text", "") + programming_language = message_content.get("language", "python") + formatted_lines.extend( + format_code_block(code_text, programming_language, output_format) + ) + return formatted_lines + + if content_type in ("multimodal_text", "text", "model_editable_context"): + # Get text parts based on content type + text_parts = message_content.get("parts", []) + if not text_parts and content_type == "text": + text_parts = [message_content.get("text", "")] + + for text_segment in text_parts: + # Skip empty or dictionary segments + if not text_segment or isinstance(text_segment, dict): + continue + + # Clean the text segment + text_segment = clean_text(text_segment) + + if output_format == "markdown": + formatted_lines.append(text_segment + "\n\n") + else: + # For HTML, let Mistune handle the conversion + markdown_parser = mistune.create_markdown( + plugins=['strikethrough', 'footnotes', 'table'] + ) + html_content = markdown_parser(text_segment) + formatted_lines.append(html_content + "\n") + + return formatted_lines + + +def process_text_with_markdown(text_segment: str, output_format: str) -> List[str]: + """ + Process text content with embedded markdown. + + Args: + text_segment: Text content to process + output_format: Output format ('html' or 'markdown') + + Returns: + List of formatted lines with markdown properly handled + + Example: + >>> process_text_with_markdown('Text with **bold**', 'markdown') + ['Text with **bold**'] + """ + formatted_lines: List[str] = [] + processed_text_chunks: List[str] = [] + last_segment_end = 0 + + try: + markdown_segments = detect_markdown(text_segment) + if not markdown_segments: + # No embedded markdown found, just format normally + formatted_lines.extend(format_text_block(text_segment, output_format)) + return formatted_lines + + # Build a raw string that has the embedded segments fenced + for segment_start, segment_end, raw_markdown in markdown_segments: + if segment_start > last_segment_end: + processed_text_chunks.append( + text_segment[last_segment_end:segment_start] + ) + processed_text_chunks.append("```markdown\n") + processed_text_chunks.append(raw_markdown) + processed_text_chunks.append("\n```\n") + last_segment_end = segment_end + + # Remainder after last segment + if last_segment_end < len(text_segment): + processed_text_chunks.append(text_segment[last_segment_end:]) + + # We just append the processed string as-is + # since we do not want to re-parse it again with Mistune. + formatted_lines.append("".join(processed_text_chunks)) + except Exception as error: + print(f"Warning: Error processing markdown: {error}") + # On error, just output as standard text block + formatted_lines.extend(format_text_block(text_segment, output_format)) + + return formatted_lines + + +def process_messages(message_mapping: Dict[str, Dict], output_format: str) -> List[str]: + """ + Process all messages in conversation order. + + Args: + message_mapping: Dictionary of message data keyed by message ID + output_format: Output format ('html' or 'markdown') + + Returns: + List of formatted lines for complete conversation + + This function: + 1. Builds the message sequence + 2. Processes each message in order + 3. Handles role transitions and tool message blocks + 4. Formats content according to message type and output format + """ + message_sequence = build_message_sequence(message_mapping) + formatted_lines: List[str] = [] + previous_message_role: Optional[str] = None + + for message_id in message_sequence: + mapping_data = message_mapping.get(message_id, {}) + message_data = mapping_data.get("message", mapping_data) + if not message_data: + continue + + message_role = message_data.get("author", {}).get("role", "") + message_content = message_data.get("content", {}) + message_metadata = message_data.get("metadata", {}) + is_user_system_message = message_metadata.get("is_user_system_message") + + # Skip empty messages or messages with no role + if not message_role: + continue + + # Skip empty system messages + if ( + message_role == "system" + and message_content.get("content_type") == "text" + and message_content.get("parts") == [""] + and not is_user_system_message + ): + continue + + message_timestamp = format_timestamp(message_data.get("create_time", 0)) + + # If role changed, close the old block if it was tool => non-tool + if message_role != previous_message_role: + if ( + previous_message_role + and is_tool_message(previous_message_role) + and not is_tool_message(message_role) + ): + formatted_lines.append("\n\n") + + if formatted_lines and formatted_lines[-1].strip(): + formatted_lines.append("\n") + formatted_lines.extend( + generate_heading(message_role, message_timestamp, output_format) + ) + + # For system/user messages with context, use the context data as content + user_context_message_data = message_metadata.get("user_context_message_data", {}) + if ( + message_role in ("system", "user") + and is_user_system_message + and user_context_message_data + ): + about_user = user_context_message_data.get("about_user_message", "") + about_model = user_context_message_data.get("about_model_message", "") + message_role = "system" + + message_content = { + "content_type": "text", + "parts": [ + f"### About User:\n{about_user}\n\n### About Assistant:\n{about_model}" + ] + } + formatted_lines.extend( + generate_heading(message_role, message_timestamp, output_format) + ) + + # Handle message content based on type + if is_tool_message(message_role): + # tool messages remain open until we switch roles + formatted_lines.extend( + process_tool_content(message_content, message_timestamp) + ) + else: + formatted_lines.extend( + handle_regular_message(message_content, output_format) + ) + + previous_message_role = message_role + + # Close final tool message if needed + if previous_message_role and is_tool_message(previous_message_role): + formatted_lines.append("\n\n") + + return formatted_lines + + +def extract_one_file(input_path: str, out_dir: Optional[str], out_fmt: str) -> None: + """ + Extract a single JSON file into a Markdown or HTML file. + + Args: + input_path: Path to the JSON file + out_dir: Optional output directory + out_fmt: Output format ('markdown' or 'html') + + This function: + 1. Loads the JSON file + 2. Extracts metadata (title, timestamps) + 3. Generates unique output filename + 4. Processes all messages + 5. Writes formatted output file + """ + data = load_json_file(input_path) + if not data: + return + + msg_map = data.get("mapping", {}) + if not msg_map: + return + + title = data.get("title", "Untitled") + ctime = data.get("create_time") + utime = data.get("update_time") + + # Use the same logic for generating the name, but with md/html extension + extension = "html" if out_fmt == "html" else "md" + try: + out_path = generate_unique_filename( + input_path, title, ctime, utime, extension=extension, out_dir=out_dir + ) + except ValueError as e: + print(f"Cannot generate filename for {input_path}: {e}") + return + + print(f"Processing: {input_path}") + print(f"Writing to: {out_path}") + + ctime_str = format_timestamp(ctime) + utime_str = format_timestamp(utime) + + lines: List[str] = [] + + # Header + if out_fmt == "html": + lines.append(HTML_TEMPLATE_HEADER.format( + title=html.escape(title), + start_time=html.escape(ctime_str or "Unknown"), + end_time=html.escape(utime_str or "Unknown"), + )) + else: + lines.append(f"# {title}\nStarting: {ctime_str}\nEnding: {utime_str}\n\n") + + # Body + lines.extend(process_messages(msg_map, out_fmt)) + + # Footer + if out_fmt == "html": + lines.append(HTML_TEMPLATE_FOOTER) + + os.makedirs(os.path.dirname(out_path), exist_ok=True) + try: + with open(out_path, "w", encoding="utf-8") as f: + f.writelines(lines) + except Exception as err: + print(f"Error writing {out_path}: {err}") + + +def process_file_patterns(patterns: List[str], out_dir: Optional[str], out_fmt: str) -> None: + """ + Process multiple file patterns and extract conversations from matching files. + + Args: + patterns: List of file/directory patterns to process + out_dir: Optional output directory for extracted files + out_fmt: Output format ('markdown' or 'html') + + For each pattern: + - If it's a directory, process all .json files in it + - If it's a file pattern, process all matching files + - Skip non-JSON files + """ + all_files: List[str] = [] + for pat in patterns: + if os.path.isdir(pat): + all_files.extend(glob.glob(os.path.join(pat, "*.json"))) + else: + matched = glob.glob(pat) + if not matched: + print(f"No files match: {pat}") + all_files.extend(matched) + + for f in all_files: + if os.path.isfile(f) and f.lower().endswith(".json"): + extract_one_file(f, out_dir, out_fmt) + + +def main(): + """ + Main entry point for the script. + + Parses command line arguments and processes the specified files: + - Takes one or more file patterns as input + - Optional output directory + - Optional output format (markdown or html) + """ + parser = argparse.ArgumentParser(description="Extract conversation logs to Markdown/HTML.") + parser.add_argument("patterns", nargs="+", help="File patterns for JSON input") + parser.add_argument("-o", "--output-dir", help="Output directory") + parser.add_argument("--format", choices=["markdown", "html"], default="markdown", help="Output format") + args = parser.parse_args() + + process_file_patterns(args.patterns, args.output_dir, args.format) + + +if __name__ == "__main__": + main() diff --git a/bin/generate_manifest.sh b/bin/generate_manifest.sh index dc6c4ced..9ca3d449 100755 --- a/bin/generate_manifest.sh +++ b/bin/generate_manifest.sh @@ -1,26 +1,9 @@ #!/usr/bin/env bash +#set -euo pipefail + # Script to generate a manifest file for the venvutil project -# Output file -OUTPUT_FILE="manifest.lst" - -# Check if the script is running in Bash >= 4.0 -if ((BASH_VERSINFO[0] < 4)); then - echo "This script requires Bash version 4.0 or higher." - exit 1 -fi - -# Check which 'stat' command is available and set commands accordingly -if stat --version >/dev/null 2>&1; then - # GNU stat - PERMISSIONS_CMD="stat -c %a" - SIZE_CMD="stat -c %s" -else - # Assume BSD stat (macOS) - PERMISSIONS_CMD="stat -f %A" - SIZE_CMD="stat -f %z" -fi # Function to process entries and generate the manifest process_and_generate_manifest() { @@ -90,8 +73,7 @@ process_and_generate_manifest() { fi # Write entry to manifest with consistent format - echo "$type | $target_location | $source_location | $asset_name | $permissions | | | $size | $checksum" >> "$OUTPUT_FILE" - + echo "$type | $target_location | $source_location | $asset_name | $permissions | | | $size | $checksum" >> "$OUTPUT_FILE" } # Function to get deleted files from git status @@ -110,6 +92,68 @@ get_deleted_files() { git diff --name-status main | grep -e '^D' | sed 's/D.[[:space:]]*//' } + +## Initialization +# Check if the script is running in Bash >= 4.0 +if ((BASH_VERSINFO[0] < 4)); then + echo "This script requires Bash version 4.0 or higher." + exit 1 +fi + +# Determine the real path of the script +[ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" +# Extract script name, directory, and arguments +# MY_NAME appears unused. Verify use (or export if used externally). +# shellcheck disable=SC2034 +MY_NAME=$(basename "${THIS_SCRIPT}") +__SETUP_BIN="$(dirname "${THIS_SCRIPT}")" +__SETUP_BASE=$(dirname "${__SETUP_BIN}") +__SETUP_INCLUDE="${__SETUP_BASE}/bin/shinclude" + +SHINCLUDE="${SHINCLUDE:-""}" +for try in "${SHINCLUDE}" "$(dirname "${THIS_SCRIPT}")/shinclude" "${__SETUP_INCLUDE}" "${HOME}/bin/shinclude"; do + [ -f "${try}/config_lib.sh" ] && { SHINCLUDE="${try}"; break; } +done +[ -z "${SHINCLUDE}" ] && { + cat<<_EOT_ >&2 +ERROR ($MY_NAME): Could not locate \`config_lib.sh\` file. +ERROR ($MY_NAME): Please set install config_lib.sh which came with this repository in one of + the following locations: + - $(dirname "${THIS_SCRIPT}")/shinclude + - $HOME/shinclude + - $HOME/bin/shinclude + or set the environment variable SHINCLUDE to the directory containing config_lib.sh + +_EOT_ + exit 2 # (ENOENT: 2): No such file or directory +} +echo "INFO ($MY_NAME): Using SHINCLUDE directory - ${SHINCLUDE}" >&2 +# shellcheck source=/dev/null +source "${SHINCLUDE}/config_lib.sh" + +pkg_config_vars +# Specify the files and directories to include +load_pkg_config "${__SETUP_BASE}/setup.cf" +# shellcheck disable=SC2206 +include_files=(${include_files[@]:-("README.md" "LICENSE" "setup.sh" "setup.cf" "manifest.lst")}) +# shellcheck disable=SC2206 +include_dirs=(${include_dirs[@]:-("bin" "docs" "conf")}) +# shellcheck disable=SC2206 +exclude_dirs=(".vscode" "tmp" ) +# Output file +OUTPUT_FILE="manifest.lst" + +# Check which 'stat' command is available and set commands accordingly +if stat --version >/dev/null 2>&1; then + # GNU stat + PERMISSIONS_CMD="stat -c %a" + SIZE_CMD="stat -c %s" +else + # Assume BSD stat (macOS) + PERMISSIONS_CMD="stat -f %A" + SIZE_CMD="stat -f %z" +fi + # Start fresh # Add a header to the manifest file echo "# This file uses pipe-separated fields" > "$OUTPUT_FILE" @@ -123,31 +167,12 @@ while read -r deleted_file; do fi done < <(get_deleted_files) -# Specify the files and directories to include -include_files=("README.md" "LICENSE" "setup.sh" "setup.cf" "manifest.lst") -include_dirs=("bin" "docs" "conf") - -# Initialize arrays -hidden_include_dirs=() -visible_include_dirs=() - -# Separate include_dirs into hidden and visible directories -for dir in "${include_dirs[@]}"; do - if [[ "$dir" == .* ]]; then - hidden_include_dirs+=("$dir") - else - visible_include_dirs+=("$dir") - fi -done - # Construct the prune condition prune_conditions=() -if [[ ${#hidden_include_dirs[@]} -gt 0 ]]; then - prune_conditions+=( "-path" "*/.*" ) - prune_conditions+=( "-a" ) - prune_conditions+=( "-not" ) +if [[ ${#exclude_dirs[@]} -gt 0 ]]; then prune_conditions+=( "(" ) - for dir in "${hidden_include_dirs[@]}"; do + prune_conditions+=( "-path" "*/.*" "-o" ) + for dir in "${exclude_dirs[@]}"; do prune_conditions+=( "-path" "./$dir" "-o" "-path" "./$dir/*" "-o" ) done unset 'prune_conditions[-1]' # Remove the last "-o" @@ -173,12 +198,7 @@ for file in "${include_files[@]}"; do done # Add visible directories and their contents to the find arguments -for dir in "${visible_include_dirs[@]}"; do - find_args+=( "-path" "./$dir" "-o" "-path" "./$dir/*" "-o" ) -done - -# Add hidden directories and their contents to the find arguments -for dir in "${hidden_include_dirs[@]}"; do +for dir in "${include_dirs[@]}"; do find_args+=( "-path" "./$dir" "-o" "-path" "./$dir/*" "-o" ) done @@ -189,6 +209,7 @@ find_args+=( ")" "-print" ) # Execute the find command find "${find_args[@]}" | while read -r asset; do + echo "$asset" # Process entries process_and_generate_manifest "$asset" done diff --git a/bin/genmd b/bin/genmd index b4669ddc..7c8b22d2 100755 --- a/bin/genmd +++ b/bin/genmd @@ -133,53 +133,29 @@ # Apache License, Version 2.0 # +# Ensure the script exits on error and treats unset variables as errors +set -euo pipefail + # Get the base directory # Determine the real path of the script -[ -L "${BASH_SOURCE[$((${#BASH_SOURCE[@]} -1))]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[$((${#BASH_SOURCE[@]} -1))]}") || THIS_SCRIPT="${BASH_SOURCE[$((${#BASH_SOURCE[@]} -1))]}" -MY_NAME="$(basename "${THIS_SCRIPT}")" -MY_PATH="$(dirname "${THIS_SCRIPT}")" -#Make sure our associated utilities are in the path -export PATH="${MY_PATH}:${PATH}" +[ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" +# shellcheck disable=SC2034 +MY_NAME=$(basename "${THIS_SCRIPT}") # Check for Bash version 4+ if [ "${BASH_VERSINFO[0]}" -lt 4 ]; then echo "ERROR ($MY_NAME): genmd requires Bash version 4 or higher." >&2 exit 75 # (EPROGMISMATCH: 75): Program version wrong fi -# Ensure the script exits on error and treats unset variables as errors -set -euo pipefail +__VENV_BIN="$(dirname "${THIS_SCRIPT}")" +__VENV_BASE="$(dirname "${__VENV_BIN}")" +__VENV_INCLUDE="${__VENV_BASE}/bin/shinclude" -# Try to locate the SHINCLUDE directory which contains included scripts and functions in a "standard" -# fashion. places we could look are, in this order: -# - $(dirname "${THIS_SCRIPT}") -# - $(dirname "${THIS_SCRIPT}")/shinclude -# - $HOME/shinclude -# - $HOME/bin directory -# - $HOME/bin/shinclude directory -# - from the environment variable `SHINCLUDE` -# -SHINCLUDE="${SHINCLUDE:-""}" -for try in "${SHINCLUDE}" "$(dirname "${THIS_SCRIPT}")/shinclude" "${HOME}/bin/shinclude"; do - [ -f "${try}/errno.sh" ] && { SHINCLUDE="${try}"; break; } -done -[ -z "${SHINCLUDE}" ] && { - cat<<_EOT_ >&2 -ERROR ($MY_NAME): Could not locate \`errno.sh\` directory. -ERROR ($MY_NAME): Please set install errno.sh which came with this repository in one of - the following locations: - - $(dirname "${THIS_SCRIPT}")/shinclude - - $HOME/shinclude - - $HOME/bin/shinclude - or set the environment variable SHINCLUDE to the directory containing errno.sh - -_EOT_ - exit 2 # (ENOENT: 2): No such file or directory -} -echo "INFO ($MY_NAME): Using SHINCLUDE directory - ${SHINCLUDE}" >&2 # shellcheck source=/dev/null -source "${SHINCLUDE}/errno.sh" -MY_NAME="$(basename "${BASH_SOURCE[$((${#BASH_SOURCE[@]} -1))]}" )" +source "${__VENV_INCLUDE}/init_lib.sh" || true +#Make sure our associated utilities are in the path +export PATH="${__VENV_BIN}:${PATH}" # Get the base directory for the script GENMD_BASE="${GENMD_BASE:-${BASE_DIR:-"."}}" log_message "INFO" "Using base directory - $GENMD_BASE" @@ -433,6 +409,48 @@ parse_gitignore() { } +# set_variable - Function to assign values to variables based on their names and type. +# +# Description: +# Assigns a variable with the given name and value. +# +# If the variable is an array, associative, or scalar, it is assigned +# accordingly. Otherwise, it is treated as a scalar. +# +# Parameters: +# $1: The name of the variable to assign. +# $2: The value to assign. +# +set_variable() { + local var_name="$1" + shift + local var_value="$*" + + # Declare and assign separately to avoid masking return values. SC2155 + # shellcheck disable=SC2155 + local var_type=$(var_type "$var_name") + + log_message "DEBUG2" "assign_variable: var_name=$var_name, var_value=\"$var_value\", var_type=$var_type" + + case "$var_type" in + "array") + IFS=' ' readarray -t "$var_name" < <( printf "%s" "$var_value" ) + ;; + "associative") + IFS=' ' mapfile -g -t "$var_name" < <( printf "%s" "$var_value" ) + ;; + "scalar") + declare -g "$var_name"="$var_value" + ;; + "integer") + declare -g "$var_name=$var_value" + ;; + *) + declare -g "$var_name"="$var_value" + ;; + esac +} + # handle_variable - Manage the assignment of variable values based on configuration actions. # # Description: @@ -468,7 +486,7 @@ handle_variable() { # Merge the current value with the value from the config file local -a mergevalue mergevalue=${current_value:-""} - set_variable "$var_name" "${mergevalue[*]} ${var_value}" + set_variable "$var_name" "${mergevalue[*]}" "${var_value}" ;; "set") # Set the variable from the command line or defaults @@ -489,44 +507,6 @@ handle_variable() { esac } -# assign_variable - Function to assign values to variables based on their names and type. -# -# Description: -# Assigns a variable with the given name and value. -# -# If the variable is an array, associative, or scalar, it is assigned -# accordingly. Otherwise, it is treated as a scalar. -# -# Parameters: -# $1: The name of the variable to assign. -# $2: The value to assign. -# -set_variable() { - local var_name="$1" - shift - local value="$*" - - # Declare and assign separately to avoid masking return values. SC2155 - # shellcheck disable=SC2155 - local var_type=$(var_type "$var_name") - - log_message "DEBUG2" "assign_variable: var_name=$var_name, var_value=\"$value\", var_type=$var_type" - - case "$var_type" in - "array") - IFS=' ' readarray -t "$var_name" < <( printf "%s" "$value" ) - ;; - "associative") - IFS=' ' mapfile -g -t "$var_name" < <( printf "%s" "$value" ) - ;; - "scalar") - declare -g "$var_name"="$value" - ;; - *) - declare -g "$var_name"="$value" - ;; - esac -} # load_config - Function to load configuration from a .grc file # diff --git a/bin/numpy-1.26-reinst.sh b/bin/numpy-1.26-reinst.sh deleted file mode 100755 index 1ce74878..00000000 --- a/bin/numpy-1.26-reinst.sh +++ /dev/null @@ -1,3 +0,0 @@ -#!/usr/bin/env bash - -CFLAGS="-I/System/Library/Frameworks/vecLib.framework/Headers -Wl,-framework -Wl,Accelerate -framework Accelerate" pip install numpy==1.26.* --force-reinstall --no-deps --no-cache --no-binary :all: --no-build-isolation --compile -Csetup-args=-Dblas=accelerate -Csetup-args=-Dlapack=accelerate -Csetup-args=-Duse-ilp64=true \ No newline at end of file diff --git a/bin/numpy-comp b/bin/numpy-comp new file mode 120000 index 00000000..8ae5dc86 --- /dev/null +++ b/bin/numpy-comp @@ -0,0 +1 @@ +numpy-comp.sh \ No newline at end of file diff --git a/bin/numpy-comp.sh b/bin/numpy-comp.sh new file mode 100755 index 00000000..313cbd9a --- /dev/null +++ b/bin/numpy-comp.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash + +# Code to install a version of NumPy greater than 1.26.0. This si code which takes version of +# NumPy as an argument and installs that version and validates that the version is greater than 1.26.0. +# If the version is not greater than 1.26.0, the script will exit with a message. If no version is provided, +# The script will default to the highest stable version of 1.26 and supply the use with a message warning +# that version 1.26 will be installed and verify if this is what they want to do before proceeding. + +DEFAULT_VERSION="1.26.*" + +if [ -z "$1" ]; then + echo -e "No version specified. Would you like to install NumPy version ${DEFAULT_VERSION}? [y/N]" + read -r response + if [[ "$response" =~ ^([yY][eE][sS]|[yY])+$ ]]; then + VERSION=$DEFAULT_VERSION + else + echo "Installation cancelled. Please specify a version greater than 1.26.0" + exit 1 + fi +else + VERSION=$1 +fi + +# Handle wildcard versions by using the minimum version number +VERSION_NUM=$(echo "$VERSION" | sed 's/\*//g' | tr -d '.') +MIN_VERSION_NUM=1260 + +# If version contains wildcard, treat it as equal to minimum version +if [[ "$VERSION" == *"*"* ]]; then + VERSION_NUM=$MIN_VERSION_NUM +fi + +if [ "$VERSION_NUM" -lt "$MIN_VERSION_NUM" ]; then + echo "Error: Version must be greater than or equal to 1.26.0" + echo "Specified version: $VERSION" + exit 1 +fi + +echo "Installing NumPy version $VERSION..." +CFLAGS="-I/System/Library/Frameworks/vecLib.framework/Headers -Wl,-framework -Wl,Accelerate -framework Accelerate" pip install numpy=="$VERSION" --force-reinstall --no-deps --no-cache --no-binary :all: --no-build-isolation --compile -Csetup-args=-Dblas=accelerate -Csetup-args=-Dlapack=accelerate -Csetup-args=-Duse-ilp64=true \ No newline at end of file diff --git a/bin/numpy_torture.py b/bin/numpy_torture.py new file mode 100755 index 00000000..dc0ed687 --- /dev/null +++ b/bin/numpy_torture.py @@ -0,0 +1,159 @@ +#!/usr/bin/env python +""" +A NumPy benchmark script for measuring performance and memory usage. + +This script runs a series of common matrix operations using NumPy and measures: +- Execution time for each operation +- Memory usage via psutil +- Supports configurable matrix sizes and iteration counts +- Can output results to file or stdout + +Operations tested: +- Matrix multiplication +- Matrix transposition +- Eigenvalue computation +- Fourier transformation +- Matrix summation + +Usage: + python numpy_torture.py [--size SIZE] [--iterations ITERS] [--output FILE] + +Arguments: + --size SIZE Dimension N for NxN matrices (default: 2000) + --iterations ITERS Number of times to repeat each test (default: 10) + --output FILE Optional file path to write results (default: stdout) + +Output format: + Tab-separated values with columns for package, version, matrix size, + iterations, virtual environment, source, device, test name, runtime (s), + and memory usage (MB). +""" + +import argparse +import time +import os +from typing import Optional + +import numpy as np + +try: + import psutil +except ImportError: + print("psutil is required for memory usage reporting.") + psutil = None + + +def get_process_memory_mb() -> float: + """ + Returns the current resident set size (RSS) of the process in MB. + + Uses psutil to get accurate memory measurements across platforms. + If psutil is not available, returns 0.0. + + Returns: + float: Memory usage in megabytes + """ + if not psutil: + return 0.0 + process = psutil.Process(os.getpid()) + return process.memory_info().rss / (1024 * 1024) + + +def format_output(package: str, version: str, size: int, iterations: int, + venv: str, test_name: str, runtime: float, memory: float, + src: str = "PyPi", device: str = "Default") -> str: + """ + Formats benchmark results into a tab-separated string matching metrics layout. + + Args: + package: Name of the package being tested + version: Version of the package + size: Matrix dimension used in test + iterations: Number of test iterations + venv: Virtual environment name + test_name: Name of the specific test + runtime: Test execution time in seconds + memory: Memory usage in MB + src: Package source (default: "PyPi") + device: Computing device used (default: "Default") + + Returns: + str: Tab-separated string containing all metrics + """ + return (f"{package:<10}\t{version:<10}\t{size:<8}\t{iterations:<8}\t" + f"{venv:<15}\t{src:<8}\t{device:<8}\t{test_name:<30}\t" + f"{runtime:<8.4f}\t{memory:<8.2f}") + + +def main() -> None: + """ + Main function that runs the NumPy benchmark suite. + + Parses command line arguments, sets up the test environment, + runs the benchmark tests, and outputs results. Each test is run + for the specified number of iterations on matrices of the given size. + Results include both execution time and memory usage. + """ + parser = argparse.ArgumentParser(description="NumPy benchmark for matrix ops.") + parser.add_argument("--size", type=int, default=2000, help="Matrix dimension.") + parser.add_argument("--iterations", type=int, default=10, help="Number of iterations per test.") + parser.add_argument("--output", type=str, help="Path to output log file.") + args = parser.parse_args() + + out_file: Optional[object] = None + if args.output: + out_file = open(args.output, "w", encoding="utf-8") + + def log(msg: str) -> None: + print(msg) + if out_file: + out_file.write(f"{msg}\n") + # Print header + header = "Package\tVersion\tSize\tIterations\tVirtual Env\tSrc\tDevice\tTest\tRun Time\tMemory" + log(header) + + venv = os.getenv('CONDA_DEFAULT_ENV', 'unknown') + size = args.size + iterations = args.iterations + + # Create random arrays + a = np.random.rand(size, size) + b = np.random.rand(size, size) + + # Define tests: (test_name, function) + tests = [ + ("Matrix multiplication", lambda: np.dot(a, b)), + ("Matrix transposition", lambda: np.transpose(a)), + ("Eigenvalue computation", lambda: np.linalg.eigvals(a)), + ("Fourier transformation", lambda: np.fft.fft(a)), + ("Matrix Summation", lambda: np.sum(a)), + ] + + for test_name, func in tests: + start_t = time.time() + for _ in range(iterations): + _ = func() # Force computation + elapsed = time.time() - start_t + + # Memory usage after test + mem_mb = get_process_memory_mb() + + # Format and log output + output = format_output( + package="NumPy", + version=np.__version__, + size=size, + iterations=iterations, + venv=venv, + test_name=test_name, + runtime=elapsed, + memory=mem_mb + ) + log(output) + + if out_file: + out_file.close() + + +if __name__ == "__main__": + main() diff --git a/bin/numpybench b/bin/numpybench index 668ef813..4f39834a 100755 --- a/bin/numpybench +++ b/bin/numpybench @@ -1,130 +1,133 @@ -#!/usr/bin/env python3 -""" -NumPy Benchmark Utility - -This script performs a series of matrix operations to benchmark NumPy performance, -particularly useful for testing GPU and Neural Engine acceleration on Apple Silicon. -When NumPy is properly compiled and linked against the Accelerate Framework, -you can expect to see an 8-9x improvement in performance over pre-compiled binaries. - -The benchmark includes: -- Matrix multiplication -- Matrix transposition -- Eigenvalue computation -- Fourier transformation -- Matrix summation - -Each operation is performed on 2500x2500 matrices. -""" +#!/usr/bin/env python +import argparse import os import sys import time -from argparse import ArgumentParser -from datetime import datetime from io import StringIO +from datetime import datetime + import numpy as np -# Initialize argparse -parser = ArgumentParser(description="Run NumPy benchmarks and output results.") -parser.add_argument( - "-o", - "--output", - type=str, - help="Optional file to write detailed results to.", -) -parser.add_argument( - "-s", "--skip-tests", action="store_true", help="Skip time-consuming tests." -) -parser.add_argument( - "-c", - "--count", - type=int, - default=1, - help="Number of iterations for each test (default: 1).", -) - -# Parse the arguments -args = parser.parse_args() - -# Open output file if specified -OUTPUT_FILE = None -if args.output: - OUTPUT_FILE = open(args.output, "w", encoding="utf-8") - - -def print_with_timestamp(message, file=None): - """Print messages with a timestamp. +OUTPUT_FILE = None # Will be assigned if --output is provided + +def timestamped_print(message: str, file=None) -> None: + """ + Print a message with a timestamp. + Optionally writes to a file if provided. Args: - message: The message to print - file: Optional file object to write to + message (str): The message to print + file (file-like object, optional): File object to write output """ - output = f"{datetime.now().strftime('%Y-%m-%d %H:%M:%S')} {message}" - print(output) + out = f"{datetime.now().strftime('%Y-%m-%d %H:%M:%S')} {message}" + print(out) if file: - file.write(output + "\n") - + file.write(out + "\n") -def do_tests(): - """Perform matrix operation benchmarks. - Tests include: - - Matrix multiplication (np.dot) - - Matrix transposition (np.transpose) - - Eigenvalue computation (np.linalg.eigvals) - - Fourier transformation (np.fft.fft) - - Matrix summation (np.sum) +def run_numpy_tests(size: int, iterations: int, file=None) -> None: """ - size = 2500 - a = np.random.rand(size, size) - b = np.random.rand(size, size) + Execute basic NumPy operations multiple times for performance measurements. - # Number of iterations - iterations = args.count + Args: + size (int): Dimension for square matrices + iterations (int): Number of repeats for each test + file (file-like, optional): Where to write log output + """ + timestamped_print(f"Creating random matrices of size {size}x{size}", file=file) + mat_a = np.random.rand(size, size) + mat_b = np.random.rand(size, size) + # Tests to perform tests = [ - ("Matrix multiplication", lambda: np.dot(a, b)), - ("Matrix transposition", lambda: np.transpose(a)), - ("Eigenvalue computation", lambda: np.linalg.eigvals(a)), - ("Fourier transformation", lambda: np.fft.fft(a)), - ("Summation", lambda: np.sum(a)), + ("Matrix multiplication", lambda: np.dot(mat_a, mat_b)), + ("Matrix transposition", lambda: np.transpose(mat_a)), + ("Eigenvalue computation", lambda: np.linalg.eigvals(mat_a)), + ("Fourier transformation", lambda: np.fft.fft(mat_a)), + ("Summation", lambda: np.sum(mat_a)), ] - for name, test_func in tests: - print_with_timestamp(f"BEGIN TEST: {name}", file=OUTPUT_FILE) - start = time.time() + # Run each test 'iterations' times + for test_name, func in tests: + timestamped_print(f"BEGIN TEST: {test_name}", file=file) + start_time = time.time() + for _ in range(iterations): - test_func() - end = time.time() - print_with_timestamp( - f"Time for {name.lower()}: {(end - start):.4f} seconds", file=OUTPUT_FILE + _ = func() # Even if we ignore the result, it forces execution + + elapsed = time.time() - start_time + timestamped_print( + f"Time for {test_name.lower()}: {elapsed:.4f} seconds", + file=file ) - print_with_timestamp("END TEST / BEGIN NEXT TEST", file=OUTPUT_FILE) + timestamped_print("END TEST / BEGIN NEXT TEST", file=file) -def main(): - """Run the NumPy benchmark suite.""" - print_with_timestamp( - f"Producing information for VENV ----> {os.getenv('CONDA_DEFAULT_ENV')}", - file=OUTPUT_FILE, - ) +def show_numpy_config(file=None) -> None: + """ + Capture and print NumPy configuration info with timestamps. - # Capture np.show_config() output and print it line by line with timestamps + Args: + file (file-like, optional): Where to write output + """ old_stdout = sys.stdout - new_stdout = StringIO() - sys.stdout = new_stdout + capture_stdout = StringIO() + sys.stdout = capture_stdout + np.show_config() + sys.stdout = old_stdout + lines = capture_stdout.getvalue().strip().split("\n") + for line in lines: + timestamped_print(line, file=file) + + +def main() -> None: + """ + Main entry point to run the NumPy torture tests. + Allows specifying matrix size, iteration count, + skip-tests option, and output file. + """ + parser = argparse.ArgumentParser(description="NumPy torture test.") + parser.add_argument( + "--size", + type=int, + default=2500, + help="Matrix dimension (square); default=2500" + ) + parser.add_argument( + "--count", + type=int, + default=1, + help="Number of iterations per test; default=1" + ) + parser.add_argument( + "--skip-tests", + action="store_true", + help="Skip the performance checks entirely" + ) + parser.add_argument( + "--output", + type=str, + help="Path to output log file" + ) + + args = parser.parse_args() - for line in new_stdout.getvalue().split("\n"): - print_with_timestamp(line, file=OUTPUT_FILE) + global OUTPUT_FILE + if args.output: + OUTPUT_FILE = open(args.output, "w", encoding="utf-8") + + # Show environment info + venv = os.getenv("CONDA_DEFAULT_ENV", "Unknown environment") + timestamped_print(f"Using environment: {venv}", file=OUTPUT_FILE) + timestamped_print("NumPy configuration:", file=OUTPUT_FILE) + show_numpy_config(file=OUTPUT_FILE) if args.skip_tests: - print_with_timestamp( - "############### SKIPPING PERFORMANCE CHECKS", file=OUTPUT_FILE - ) + timestamped_print("SKIPPING PERFORMANCE CHECKS", file=OUTPUT_FILE) else: - do_tests() + run_numpy_tests(size=args.size, iterations=args.count, file=OUTPUT_FILE) if OUTPUT_FILE: OUTPUT_FILE.close() @@ -132,3 +135,4 @@ def main(): if __name__ == "__main__": main() + diff --git a/bin/purgevenv b/bin/purgevenv index 85d9880b..9a96f3c3 100755 --- a/bin/purgevenv +++ b/bin/purgevenv @@ -11,8 +11,8 @@ set_args() { } set_args -[ -f ${MY_INCLUDE}/init_env.sh ] && . ${MY_INCLUDE}/init_env.sh \ - || ( echo "${MY_NAME}: Could not find init_env.sh in INCLUDE_DIR: ${MY_INCLUDE}"; exit 1 ) +[ -f ${MY_INCLUDE}/venvutil_lib.sh ] && . ${MY_INCLUDE}/venvutil_lib.sh \ + || ( echo "${MY_NAME}: Could not find venvutil_lib.sh in INCLUDE_DIR: ${MY_INCLUDE}"; exit 1 ) set_args # Delete the environments. diff --git a/bin/rename-chat b/bin/rename-chat new file mode 120000 index 00000000..b8275d80 --- /dev/null +++ b/bin/rename-chat @@ -0,0 +1 @@ +rename-chat.py \ No newline at end of file diff --git a/bin/rename-chat.py b/bin/rename-chat.py new file mode 100755 index 00000000..e88fdb50 --- /dev/null +++ b/bin/rename-chat.py @@ -0,0 +1,207 @@ +#!/usr/bin/env python + +import os +import re +import json +from datetime import datetime +from typing import Dict, Optional, List +import argparse +import glob + +# Date/time formats +FILENAME_DATE_FORMAT = "%Y-%m-%d-%H%M%S" +DATETIME_FORMAT = "%Y-%m-%d %H:%M:%S" + + +def fix_timestamp(ts: Optional[float]) -> Optional[float]: + """ + Fix overly large timestamps by moving the decimal so + they're in the ~10-digit Unix timestamp range. + Returns None if input is None. + """ + if ts is None: + return None + # Convert to string to count digits before decimal + ts_str = str(float(ts)) + whole_digits = ts_str.split('.', maxsplit=1)[0] + # If more than 10 digits before decimal, adjust + if len(whole_digits) > 10: + power = len(whole_digits) - 10 + return float(ts) / (10**power) + return float(ts) + + +def format_timestamp(ts: Optional[float]) -> str: + """ + Format a numeric timestamp into YYYY-MM-DD HH:MM:SS, after fixing if necessary. + Returns an empty string if None or invalid. + """ + ts_fixed = fix_timestamp(ts) + if ts_fixed is None: + return "" + dt = datetime.fromtimestamp(ts_fixed) + return dt.strftime(DATETIME_FORMAT) + + +def parse_datetime_string(dt_str: str) -> datetime: + """ + Parse 'YYYY-MM-DD HH:MM:SS' into a datetime object. + On error, return datetime.now() as a fallback. + """ + try: + return datetime.strptime(dt_str, DATETIME_FORMAT) + except ValueError: + return datetime.now() + + +def sanitize_title(title: str) -> str: + """ + Replace problematic characters with underscores, remove extra underscores, + strip leading/trailing punctuation, and ensure non-empty. + """ + s = re.sub(r"[^\w\-.]", "_", title) + s = re.sub(r"_+", "_", s).strip("._-") + return s or "untitled" + + +def generate_unique_filename( + input_file: str, + title: str, + create_time: Optional[float], + update_time: Optional[float], + extension: str = "json", + out_dir: Optional[str] = None +) -> str: + """ + Generate a unique filename of form: + YYYY-MM-DD-HHMMSS_YYYY-MM-DD-HHMMSS-TITLE.ext + + - Fix timestamps if necessary. + - Use create_time if update_time is missing. + - Check for collisions, append -NN if needed (max 99 collisions). + + Raises ValueError if we fail after 99 collisions. + """ + + # Fix both timestamps + ctime_fixed = fix_timestamp(create_time) + utime_fixed = fix_timestamp(update_time) + + if ctime_fixed is None: + raise ValueError("Invalid or missing create_time") + + # If update_time is missing, use create_time + if utime_fixed is None: + utime_fixed = ctime_fixed + + # Convert each to string + ctime_str = datetime.fromtimestamp(ctime_fixed).strftime(FILENAME_DATE_FORMAT) + utime_str = datetime.fromtimestamp(utime_fixed).strftime(FILENAME_DATE_FORMAT) + cleaned_title = sanitize_title(title) + + dir_path = out_dir if out_dir else os.path.dirname(input_file) + os.makedirs(dir_path, exist_ok=True) + + base_name = f"{ctime_str}_{utime_str}-{cleaned_title}" + filename = os.path.join(dir_path, f"{base_name}.{extension}") + + counter = 0 + while os.path.exists(filename): + counter += 1 + if counter > 99: + raise ValueError(f"Too many duplicates for {filename}") + filename = os.path.join(dir_path, f"{base_name}-{counter:02d}.{extension}") + return filename + + +def load_json_file(file_path: str) -> Optional[Dict]: + """ + Load JSON data from a file, returning a dict or None if invalid. + """ + try: + with open(file_path, "r", encoding="utf-8") as f: + data = json.load(f) + if isinstance(data, dict): + return data + print(f"Error: {file_path} did not contain a JSON object.") + except Exception as e: + print(f"Error reading JSON from {file_path}: {e}") + return None + + +def confirm_and_rename(old_path: str, new_path: str, auto_yes: bool) -> None: + print(f"Would rename:\n {old_path}\n-> {new_path}") + if auto_yes or input("Proceed with rename? (y/n): ").lower().strip() == "y": + try: + os.rename(old_path, new_path) + print(f"Renamed to: {new_path}") + except OSError as e: + print(f"Error renaming {old_path}: {e}") + else: + print("Rename cancelled") + + +def rename_one_json(file_path: str, auto_yes: bool) -> None: + data = load_json_file(file_path) + if not data: + return + + title = data.get("title", "").strip() + ctime = data.get("create_time") + utime = data.get("update_time") + + if not title: + print(f"Skipping {file_path}: Missing or empty title.") + return + if not ctime: + print(f"Skipping {file_path}: Missing create_time.") + return + + try: + new_name = generate_unique_filename(file_path, title, ctime, utime, extension="json") + confirm_and_rename(file_path, new_name, auto_yes) + except ValueError as e: + # Possibly creation time is invalid, fallback to file stat times + print(f"Error generating name for {file_path}: {e}") + try: + st = os.stat(file_path) + ctime = st.st_ctime + utime = st.st_mtime + print("Using file system timestamps as fallback.") + new_name = generate_unique_filename(file_path, title, ctime, utime, extension="json") + confirm_and_rename(file_path, new_name, auto_yes) + except (OSError, ValueError) as e2: + print(f"Failed fallback for {file_path}: {e2}") + + +def process_files(patterns: List[str], auto_yes: bool) -> None: + files_to_rename = [] + for pattern in patterns: + if os.path.isdir(pattern): + files_to_rename.extend(glob.glob(os.path.join(pattern, "*.json"))) + else: + matched = glob.glob(pattern) + if not matched: + print(f"No files match: {pattern}") + files_to_rename.extend(matched) + + if not files_to_rename: + print("No files found to process.") + return + + print(f"Found {len(files_to_rename)} files.") + for path in files_to_rename: + if os.path.isfile(path) and path.lower().endswith(".json"): + print(f"\nProcessing: {path}") + rename_one_json(path, auto_yes) + + +def main(): + parser = argparse.ArgumentParser(description="Rename JSON files based on title and timestamps.") + parser.add_argument("patterns", nargs="+", help="File patterns (wildcards, directories, etc.)") + parser.add_argument("-y", "--yes", action="store_true", help="Auto-confirm all renames.") + args = parser.parse_args() + process_files(args.patterns, args.yes) + +if __name__ == "__main__": + main() diff --git a/bin/runbench b/bin/runbench index 79d65662..c1bf1bef 100755 --- a/bin/runbench +++ b/bin/runbench @@ -9,8 +9,8 @@ MY_ARGS=$* MY_INCLUDE=$( ( [ -d "${MY_BIN}/shinclude" ] && echo "${MY_BIN}/shinclude" ) || echo "${HOME}/shinclude" ) # Source in some useful shortcut functions for VENV management. -[ -f ${MY_INCLUDE}/init_env.sh ] && . ${MY_INCLUDE}/init_env.sh \ - || ( echo "${MY_NAME}: Could not find init_env.sh in INCLUDEDIR: ${MY_INCLUDE}"; exit 1 ) +[ -f ${MY_INCLUDE}/venvutil_lib.sh ] && . ${MY_INCLUDE}/venvutil_lib.sh \ + || ( echo "${MY_NAME}: Could not find venvutil_lib.sh in INCLUDEDIR: ${MY_INCLUDE}"; exit 1 ) export PATH=${BENCH_BASE}/bin:${PATH} diff --git a/bin/shinclude/config_lib.sh b/bin/shinclude/config_lib.sh new file mode 100755 index 00000000..83f027f7 --- /dev/null +++ b/bin/shinclude/config_lib.sh @@ -0,0 +1,378 @@ +#!/usr/bin/env bash +# # Script: config_lib.sh +# `config_lib.sh` - Configuration Management and Variable Handling +# +# ## Description +# - **Purpose**: +# - Manages configuration file loading +# - Handles variable initialization +# - Controls configuration precedence +# - Provides configuration validation +# +# ## Configuration handling +# - Dependency resolution +# +# ## Configuration handling +# - Modeled to use pkg-config like files. +# - Performs variable expansion on assigned values including arrays. +# - Variables can be set using the following syntax: +# - prefix=$HOME/local/venvutil +# - exec_prefix=${prefix} +# - libdir=${exec_prefix}/lib +# - includedir=${prefix}/include +# - bindir=${exec_prefix}/bin +# - datadir=${prefix}/share +# - sysconfdir=${prefix}/etc +# +# ## Variable Handling +# - using an associative array with the variable names as keys and an action to use the values. +# - Associative array contains (key) VARNAME: (value) Action pairs. +# - Actions are the following: +# - "set" - set the variable to the value +# - "merge" - merge the value into the variable +# - "config" - from the config settings +# - "discard" - discard and use the default or standard config value. +# +# ## Usage +# - Source this script in your Bash scripts to utilize its functions. +# ```bash +# source_lib config_lib +# ``` +# ## Input Parameters +# - None. +# ## Output +# - Sets variables from the setup.cf file for package installation. +# ## Exceptions +# - Returns specific error codes if the setup.cf file is not found or invalid. +# ## Initialization +# - Ensures the script is sourced only once and initializes necessary variables. + +## Initialization +[ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" +if ! declare -p __VENV_SOURCED >/dev/null 2>&1; then declare -g -A __VENV_SOURCED; fi +if [[ "${__VENV_SOURCED[${THIS_SCRIPT}]:-}" == 1 ]]; then + # echo "************************* SKIPPED SKIPPED SKIPPED SKIPPED ************************* -----> $(basename "${THIS_SCRIPT}")" >&2 + return +fi +__VENV_SOURCED["${THIS_SCRIPT}"]=1 +# echo "************************* READING READING READING READING ************************* -----> $(basename "${THIS_SCRIPT}")" >&2 + +# shellcheck disable=SC2034 +MY_NAME=$(basename "${THIS_SCRIPT}") +__VENV_BIN=$(dirname "$(dirname "${THIS_SCRIPT}")") +__VENV_BASE=$(dirname "${__VENV_BIN}") +__VENV_INCLUDE="${__VENV_BASE}/bin/shinclude" + +# Get the init_lib.sh script +# shellcheck source=/dev/null +source "${__VENV_INCLUDE}/init_lib.sh" + +# Get the errno_lib.sh script +source_lib errno_lib + +# Get the string_lib.sh script +# source_lib string_lib + +# Get the type_lib.sh script +source_lib type_lib + +# Declare global variables +# The values are the values by key in the .cf file regardless fo the operator used to set them. +# shellcheck disable=SC2034 +declare -g -A pkg_config_values=() +# The set variables are the variables which have set values using the `=` operator +# shellcheck disable=SC2034 +declare -g -a pkg_config_set_vars=() +# The desc variables are the variables which have desc values using the `:` operator +# shellcheck disable=SC2034 +declare -g -a pkg_config_desc_vars=() + +# Add internal functions to the __VENV_INTERNAL_FUNCTIONS array. +if ! declare -p __VENV_INTERNAL_FUNCTIONS >/dev/null 2>&1; then declare -ga __VENV_INTERNAL_FUNCTIONS; fi +# Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a. +# shellcheck disable=SC2206 +__VENV_INTERNAL_FUNCTIONS=( + ${__VENV_INTERNAL_FUNCTIONS[@]} + "expand_variables" + "load_pkg_config" +) + +# # Function: `pkg_config_vars` +# `pkg_config_vars` - Sets up the actions for the variables in a package config file. +# +# ## Description +# - **Purpose**: +# - Sets up the actions for the variables in a package config file. +# - **Usage**: +# - `pkg_config_vars` +# - **Input Parameters**: +# - None. +# - **Output**: +# - None. +# - **Exceptions**: +# - None. +pkg_config_vars() { + # shellcheck disable=SC2034 + declare -g -A var_actions=( + ["Cflags"]="set" + ["Conflicts"]="merge" + ["Contribute"]="merge" + ["Description"]="set" + ["Libs"]="set" + ["License"]="merge" + ["Name"]="set" + ["Repository"]="set" + ["Requires"]="merge" + ["Support"]="merge" + ["Version"]="set" + ["bindir"]="set" + ["datadir"]="set" + ["exec_prefix"]="set" + ["include_dirs"]="set" + ["include_files"]="set" + ["includedir"]="set" + ["libdir"]="set" + ["prefix"]="set" + ["sysconfdir"]="set" + ) +} + +# # Function: expand_variable +# `expand_variable` - Expands variables in a given string. +# ## Description +# - **Purpose**: +# - This function takes a string and expands variables within it. +# - **Usage**: +# - `expand_variable "string_with_variables"` +# - **Input Parameters**: +# - `string_with_variables`: The string containing variables to be expanded. +# - **Output**: +# - The expanded string with variables replaced by their values. +# +expand_variable() { + local input="$1" + + # Sanitize the input by escaping special characters if necessary + # For example, you might want to escape quotes or backslashes + sanitized_input=$(echo "$input" | sed 's/[&;|\n\\>=]/\\&/g') + + # Use eval to expand variables safely, handling both ${var} and $var notation + eval echo "$sanitized_input" +} + +# # Function: load_pkg_config +# `load_pkg_config` - Loads package configuration from setup.cf file. +# ## Description +# - **Purpose**: +# - This function reads the setup.cf file and sets variables for package installation. +# - **Usage**: +# - `load_pkg_config` +# - **Input Parameters**: +# - None. +# - **Output**: +# - Sets variables from the setup.cf file for package installation. +# - **Exceptions**: +# - Returns specific error codes if the setup.cf file is not found or invalid. +# - **Examples** setup.cf file: +# ``` +# # Package Configuration File for venvutil +# ## Define variables +# prefix=$HOME/local/venvutil +# exec_prefix=${prefix} +# libdir=${exec_prefix}/lib +# includedir=${prefix}/include +# bindir=${exec_prefix}/bin +# datadir=${prefix}/share +# sysconfdir=${prefix}/etc +# include_dirs=("bin" "docs" "conf") +# include_files=("README.md" "LICENSE" "setup.sh" "setup.cf" "manifest.lst") +# +# # Package metadata +# Name: venvutil +# Description: Virtual Environment Utilities +# Version: 0.4.0 +# Repository: https://github.com/unixwzrd/venvutil +# License: Apache License, Version 2.0 +# Support: https://github.com/unixwzrd/venvutil/issues +# Contribute: https://patreon.com/unixwzrd +# Contribute: https://www.ko-fi.com/unixwzrd +# +# # Dependencies (if any) +# Requires: python >= 3.10 +# Requires: bash >= 4.0 +# Requires: Conda >= 22.11 +# Requires: macOS +# Requires: Linux +# Conflicts: +# +# # Compiler and linker flags (if applicable) +# # Cflags: -I${includedir} +# # Libs: -L${libdir} -lvenvutil +# ``` +load_pkg_config() { + local config_file="$1" + if [[ ! -f "$config_file" ]]; then + errno_exit ENOENT "No such config file: $config_file" + fi + + local key value expanded line + + while IFS= read -r line || [[ -n "$line" ]]; do + # Trim whitespace from both ends + line="$(sed 's/^[[:space:]]*//; s/[[:space:]]*$//' <<< "$line")" + + # Skip empty lines and comments + [[ -z "$line" || "$line" =~ ^# ]] && continue + + # Case 1: Array assignment using `=` and `()` + # Example: include_files=("README.md" "LICENSE" "setup.sh") + if [[ "$line" =~ ^([A-Za-z_][A-Za-z0-9_]*)=\((.*)\)$ ]]; then + key="${BASH_REMATCH[1]}" + value="${BASH_REMATCH[2]}" + pkg_config_values["$key"]="$value" + pkg_config_set_vars+=("$key") + declare -g -a "$key" + local -a tmp_array=() + for item in ${value}; do + if ! expanded="$(expand_variable "$item")"; then + log_message "ERROR" "Invalid quoted scalar assignment: $line" + fi + tmp_array+=($expanded) + done + # Pass the entire array as a reference + handle_variable "${key}" tmp_array + continue + fi + + # Case 2: Normal scalar assignment using lvalue=rvalue + # The rvalue may or may not have "" however any $VAR, ${VAR} will be expanded. + # If the rvalue is enclosed in '' anything within the '' will be taken as literal. + # Example: prefix=$HOME/local/venvutil + if [[ "$line" =~ ^([A-Za-z_][A-Za-z0-9_]*)=(.*)$ ]]; then + key="${BASH_REMATCH[1]}" + value="${BASH_REMATCH[2]}" + # shellcheck disable=SC2034 + pkg_config_values["$key"]="$value" + pkg_config_set_vars+=("$key") + if ! expanded="$(expand_variable "$value")"; then + log_message "ERROR" "Invalid scalar assignment: $line" + continue + fi + declare -g "${key}" + handle_variable "$key" "expanded" + continue + fi + + # Case 3: Key-value pair with colon + # Example: Description: Virtual Environment Utilities + # Example: Requires: python >= 3.10 + # Example: Conflicts: + if [[ "$line" =~ ^([A-Za-z_][A-Za-z0-9_]*):([[:space:]]*([^[:space:]].*[^[:space:]])?)?$ ]]; then + key="${BASH_REMATCH[1]}" + value="${BASH_REMATCH[3]:-}" # Use empty string if no value captured + log_message "DEBUG" "Found key-value pair: $key : $value" + if ! var_type ${key} > /dev/null 2>&1 ; then + declare -g -a "${key}" + fi + if [ -n "$value" ]; then + if ! expanded="$(expand_variable "$value")"; then + log_message "ERROR" "Invalid quoted assignment: $line" + continue + fi + handle_variable "$key" expanded + else + # Handle empty value case + handle_variable "$key" ' ' + fi + continue + fi + + log_message "WARNING" "Line doesn't match known pattern: '$line'" + done < "$config_file" + return 0 +} + +# # Function: write_config +# `write_config` - Write variables in valid Bash syntax (scalar vs array). +# ## Description +# - **Purpose**: +# - Writes variables in valid Bash syntax (scalar vs array). +# - **Usage**: +# - `write_config "/path/to/output.conf" config_variables[@]` +# - **Input Parameters**: +# - `config_file`: The path to the output file. +# - `config_variables`: An array of variable names to write. +# - **Output**: +# - Writes the variables to the output file in valid Bash syntax. +write_config() { + local config_file="$1" + shift + local -a vars_to_write=("${!1}") + + : > "$config_file" # Overwrite file + log_message "INFO" "Writing config to $config_file" + + for var_name in "${vars_to_write[@]}"; do + # Skip empty var_name + [[ -z "${var_name}" ]] && continue + + local var_type + var_type="$(var_type "$var_name")" + + case "$var_type" in + "array") + # Build array literal: var_name=("elem1" "elem2") + local -a arr_copy=("${!var_name}") + local array_literal='(' + for elem in "${arr_copy[@]}"; do + local escaped="${elem//\"/\\\"}" + array_literal+="\"$escaped\" " + done + array_literal="${array_literal%% }" # remove trailing space + array_literal+=')' + echo "$var_name=$array_literal" >> "$config_file" + ;; + "associative") + # Harder to re-serialize. Implement if needed. + echo "# $var_name is associative. Implement your own logic." >> "$config_file" + ;; + "scalar"|"unknown") + local val="${!var_name}" + local escaped="${val//\"/\\\"}" + echo "$var_name=\"$escaped\"" >> "$config_file" + ;; + esac + done + log_message "INFO" "Done writing config to $config_file" +} + +# # Function: parse_manifest_metadata +# `parse_manifest_metadata` - Parses manifest metadata. +# ## Description +# - **Purpose**: +# - This function reads the manifest file and sets variables for package installation. +# - **Usage**: +# - `parse_manifest_metadata `# - **Input Parameters**: +# - `manifest_file`: The path to the manifest file. +# - **Output**: +# - Sets variables from the manifest file for package installation. +parse_manifest_metadata() { + local manifest_file="$1" + if [ ! -f "$manifest_file" ]; then + echo "ERROR: Manifest file $manifest_file not found." >&2 + exit 2 + fi + echo "Parsing manifest metadata..." >&2 + while IFS='| ' read -r line || [[ -n "$line" ]]; do + [[ "$line" =~ ^#.*$ ]] && continue # Skip comments + [[ -z "$line" ]] && break # Stop at first blank line (end of metadata) + if [[ "$line" =~ ^[A-Za-z_]+=.*$ ]]; then + eval "$line" + fi + done < "$manifest_file" + return 0 +} + + +__rc__=0 +return ${__rc__} diff --git a/bin/shinclude/errno.sh b/bin/shinclude/errno_lib.sh similarity index 58% rename from bin/shinclude/errno.sh rename to bin/shinclude/errno_lib.sh index 55e366e1..85fa8477 100755 --- a/bin/shinclude/errno.sh +++ b/bin/shinclude/errno_lib.sh @@ -1,26 +1,122 @@ #!/usr/bin/env bash - -# # Script: errno.sh -# `errno.sh` - Provides POSIX errno codes and utilities for Bash scripts +# # Script: errno_lib.sh +# `errno_lib.sh` - Provides POSIX errno codes and utilities for Bash scripts # ## Description # - **Purpose**: # - Offers functions to retrieve and manage POSIX error codes within Bash scripts. -# - **Usage**: +# ## Usage # - Source this script in your Bash scripts to utilize error code utilities. # ```bash -# source /path/to/errno.sh +# source /path/to/errno_lib.sh # ``` -# - **Input Parameters**: +# ## Input Parameters # - None. -# - **Output**: +# ## Output # - Functions that output error codes and messages. -# - **Exceptions**: +# ## Exceptions # - Returns specific error codes if system `errno.h` is not found or invalid errno codes are provided. -# - **Initialization**: +# ## Initialization # - Ensures the script is sourced only once and initializes necessary variables. # # ## Dependencies -# - `util_funcs.sh` (for utility functions like `to_upper`) +# - `string_lib.sh` (for utility functions like `to_upper`) + +## Initialization +[ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" +if ! declare -p __VENV_SOURCED >/dev/null 2>&1; then declare -g -A __VENV_SOURCED; fi +if [[ "${__VENV_SOURCED[${THIS_SCRIPT}]:-}" == 1 ]]; then + # echo "************************* SKIPPED SKIPPED SKIPPED SKIPPED ************************* -----> $(basename "${THIS_SCRIPT}")" >&2 + return +fi +__VENV_SOURCED["${THIS_SCRIPT}"]=1 + +# shellcheck disable=SC2034 +MY_NAME=$(basename "${THIS_SCRIPT}") +__VENV_BIN=$(dirname "$(dirname "${THIS_SCRIPT}")") +__VENV_BASE=$(dirname "${__VENV_BIN}") +__VENV_INCLUDE="${__VENV_BASE}/bin/shinclude" + +# Get the init_lib.sh script +# shellcheck source=/dev/null +source "${__VENV_INCLUDE}/init_lib.sh" + +# Get the string_lib.sh script +source_lib string_lib + +# Get the helpsys_lib.sh script +# source_lib helpsys_lib + +# Add internal functions to the __VENV_INTERNAL_FUNCTIONS array. +if ! declare -p __VENV_INTERNAL_FUNCTIONS >/dev/null 2>&1; then declare -ga __VENV_INTERNAL_FUNCTIONS; fi +# Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a. +# shellcheck disable=SC2206 +__VENV_INTERNAL_FUNCTIONS=( + ${__VENV_INTERNAL_FUNCTIONS[@]} + "errval" +) + +# Set default debug level, if not already set. +if ! declare -p debug_level >/dev/null 2>&1; then declare -g debug_level=30; fi +# Return code +if ! declare -p __rc__ >/dev/null 2>&1; then declare -g __rc__=0; fi + + +# # Function set_debug +# `set_debug` - Sets the debug level for the script. +# +# ## Description +# - **Purpose**: +# - Sets the debug level for the script. +# - Debug level may be given as a value between 0 and 99 +# - Debug level may be given as a text log level of the highest level to display: +# DEBUG, INFO, WARNING, ERROR, CRITICAL, or SILENT +# - **Usage**: +# - `set_debug ` +# - **Input Parameters**: +# - `debug_level`: The debug level to set. +# - **Output**: +# - Sets the debug level for the script. +# - **Exceptions**: +# - Warns if invalid level provided and keeps previous level +set_debug() { + __rc__=0 + local OPTIND=1 + while getopts "h" opt; do + case $opt in + h) vhelp "${FUNCNAME[0]}"; __rc__=0; return 0 ;; + \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; __rc__=22; return ${__rc__} ;; + esac + done + shift $((OPTIND - 1)) + + local new_level="$1" + local current_level + + current_level="$debug_level" + + #Need to set this to 30 is it's not already set so warning messages do not get suppressed + debug_level=20 + + # If numeric, validate range + if [[ "$new_level" =~ ^[0-9]+$ ]]; then + if ! [ "$new_level" -ge 0 ] && [ "$new_level" -le 99 ]; then + log_message "WARNING" "Invalid debug level '$new_level'. Must be between 0-99. Keeping current level ($debug_level)." + __rc__=22 + fi + else + new_level=$(errval "$new_level") + fi + + if [ "$new_level" -eq "$current_level" ]; then + log_message "INFO" "Debug level unchanged from current level $current_level" + fi + + # Restore previous debug level + debug_level="$new_level" + + return ${__rc__} +} + # # Function: errno # `errno` - Provides POSIX errno codes and values for use in scripts or lookup of error codes on the command line. @@ -57,7 +153,7 @@ errno() { done shift $((OPTIND - 1)) - if [ "$1" -eq 0 ]; then + if [ "$1" == "0" ]; then echo "No error" __rc__=0 return ${__rc__} @@ -80,17 +176,24 @@ errno() { if [[ "$errno_code" =~ ^[0-9]+$ ]]; then # shellcheck disable=SC1087 - line=$(grep -wE "#define [A-Z0-9_]+[[:space:]]+$errno_code[[:space:]]" "$errno_file") + line=$(grep -wE "#define [A-Z0-9_]+[[:space:]]+$errno_code[[:space:]]" "$errno_file" | head -n 1) errno_code=$(echo "$line" | awk '{print $2}') else # Not using braces for that, it's a regular expression here. # Use braces when expanding arrays, e.g. ${array[idx]} (or ${var}[.. to quiet). # shellcheck disable=SC1087 - line=$(grep -wE "#define $errno_code[ \t]*" "$errno_file") + line=$(grep -wE "#define $errno_code[ \t]*" "$errno_file" | head -n 1) fi errno_num=$(echo "$line" | awk '{print $3}') - errno_text=$(echo "$line" | sed -e 's/#define[ \t]*[A-Z0-9_]*[ \t]*[0-9]*[ \t]*\/\* \(.*\) \*\//\1/') + errno_text=$(echo "$line" | sed -e 's/#define[ \t]*[A-Z0-9_]*[ \t]*[A-Z0-9_]*[ \t]*\/\* \(.*\) \*\//\1/') + + # If errno_num is not numeric, it's likely a macro reference + if [[ "$errno_num" =~ ^[a-zA-Z]+$ ]]; then + # Call errno with the macro name to get numeric value + errno "$errno_num" >/dev/null 2>&1 + errno_num=$? + fi if [ -z "$errno_num" ]; then echo "Error: Invalid errno code $errno_code" >&2 @@ -154,8 +257,15 @@ errfind() { echo "$lines" | while read -r line; do errno_code=$(echo "$line" | awk '{print $2}') errno_num=$(echo "$line" | awk '{print $3}') - errno_text=$(echo "$line" | sed -e 's/#define[ \t]*[A-Z0-9_]*[ \t]*[0-9]*[ \t]*\/\* \(.*\) \*\//\1/') - + errno_text=$(echo "$line" | sed -e 's/#define[ \t]*[A-Z0-9_]*[ \t]*[A-Z0-9_]*[ \t]*\/\* \(.*\) \*\//\1/') + + # If errno_num is not numeric, it's likely a macro reference + if [[ "$errno_num" =~ ^[a-zA-Z]+$ ]]; then + # Call errno with the macro name to get numeric value + errno "$errno_num" >/dev/null 2>&1 + errno_num=$? + fi + echo "($errno_code: $errno_num): $errno_text" done @@ -164,70 +274,73 @@ errfind() { } # # Function: errno_warn -# `errno_warn` - Prints a warning using the errno function to STDERR and returns the error number. +# `errno_warn` - Prints a warning message to STDERR and returns the error number. # ## Description # - **Purpose**: -# - Prints a warning message to STDERR using the `errno` function and sets the return code. It -# will report the error without exiting the script. You may use the POSIX error code or the -# error number. +# - Prints a warning message to STDERR using the provided error code and optional message. +# - Sets the return code but does not exit the script. +# - Accepts either POSIX error codes (e.g. EACCES) or error numbers (e.g. 13). # - **Usage**: -# - `errno_warn ` +# - `errno_warn [message]` # - **Example**: -# - `errno_warn EACCES` -# - `errno_warn 13` +# - `errno_warn EACCES "Failed to access file"` +# - `errno_warn 13 "Permission denied"` # - **Input Parameters**: -# - `errno_code`: The errno code to generate a warning for. +# - `errno_code`: The error code to use (POSIX name or number) +# - `message`: (Optional) Additional message to include in the warning # - **Output**: -# - Outputs a warning message to STDERR with caller info including: -# - Function name that called errno_warn -# - Line number where errno_warn was called -# - Source file where errno_warn was called -# - Function name that called the function that called errno_warn -# - Line number where that function was called -# - Source file containing that function call -# - **Exceptions**: -# - Returns the error number associated with the provided errno code. +# - Prints warning messages to STDERR including: +# - Optional custom message if provided +# - Call stack trace with function name, line number and file +# - Error message corresponding to the error code +# - **Return Value**: +# - Returns the numeric error code # errno_warn() { __rc__=$1 - echo "WARNING: $(errno "${__rc__}")" >&2 - echo "WARNING: FUNCTION: ${FUNCNAME[0]} LINE: ${BASH_LINENO[0]} FILE: ${BASH_SOURCE[0]}" >&2 + shift + local message="$*" + error_text=$(errno "${__rc__}"); __rc__=$? + [ -n "${message}" ] && log_message "WARNING" "${message}" + log_message "WARNING" "FUNCTION: ${FUNCNAME[1]} LINE: ${BASH_LINENO[1]} FILE: ${BASH_SOURCE[-1]}" + log_message "WARNING" "${error_text}" return "${__rc__}" } # # Function: errno_exit -# `errno_exit` - Prints an error to STDERR using the errno function and exits with the error number. +# `errno_exit` - Prints an error message to STDERR and exits with the error number. # ## Description # - **Purpose**: -# - Prints an error message to STDERR using the `errno` function and exits the script with the corresponding error number. -# You may use the POSIX error code or the error number. +# - Prints an error message to STDERR using the provided error code and optional message. +# - Exits the script with the corresponding error number. +# - Accepts either POSIX error codes (e.g. EACCES) or error numbers (e.g. 13). # - **Usage**: -# - `errno_exit ` +# - `errno_exit [message]` # - **Example**: -# - `errno_exit EACCES` -# - `errno_exit 13` +# - `errno_exit EACCES "Failed to access file"` +# - `errno_exit 13 "Permission denied"` # - **Input Parameters**: -# - `errno_code`: The errno code to generate an error for. +# - `errno_code`: The error code to use (POSIX name or number) +# - `message`: (Optional) Additional message to include in the error # - **Output**: -# - Outputs an error message to STDERR with caller info including: -# - Function name that called errno_exit -# - Line number where errno_exit was called -# - Source file where errno_exit was called -# - Function name that called the function that called errno_exit -# - Line number where that function was called -# - Source file containing that function call -# - **Exceptions**: -# - Exits the script with the provided error number. +# - Prints error messages to STDERR including: +# - Optional custom message if provided +# - Call stack trace with function name, line number and file +# - Error message corresponding to the error code +# - **Exit Status**: +# - Exits with the numeric error code # errno_exit() { __rc__=$1 - echo "ERROR: $(errno "${__rc__}")" >&2 - echo "ERROR: FUNCTION: ${FUNCNAME[0]} LINE: ${BASH_LINENO[0]} FILE: ${BASH_SOURCE[0]}" >&2 + shift + local message="$*" + error_text=$(errno "${__rc__}"); __rc__=$? + [ -n "${message}" ] && log_message "ERROR" "${message}" + log_message "ERROR" "FUNCTION: ${FUNCNAME[1]} LINE: ${BASH_LINENO[1]} FILE: ${BASH_SOURCE[-1]}" + log_message "ERROR" "${error_text}" exit "${__rc__}" } - - # # Function: errval # `errval` - Returns the numeric value associated with a log level. # @@ -271,10 +384,9 @@ errval() { ["CRITICAL"]=50 ["SILENT"]=99 ) - echo "${message_class[$1]}" + echo "${message_class[$1]:-debug_level}" } - ## Function: log_message # `log_message` - Prints a message to STDERR based on the provided log level. # ## Description @@ -309,6 +421,8 @@ log_message() { done shift $((OPTIND - 1)) + local script_name + script_name=$(basename "${BASH_SOURCE[-1]}") local message_level="$1"; shift local message_class message_class=$(errval "$message_level") @@ -316,56 +430,33 @@ log_message() { # Check if the provided message level exists in the message_class array if [[ -z "${message_class+_}" ]]; then - echo "($MY_NAME) WARNING: Unknown log level '$message_level'. Message: $message_out" >&2 + echo "($script_name) WARNING: Unknown log level '$message_level'. Message: $message_out" >&2 errno_exit 9 fi # echo " LOG_MESSAGE CALLED: ${message_level} ${debug_level} ($MY_NAME): ${message_out}" >&2 # Compare the current debug_level with the message's severity level if [ "$debug_level" -le "${message_class}" ]; then - echo "$MY_NAME ${message_level}($debug_level): ${message_out}" >&2 + echo "$script_name ${message_level}($debug_level): ${message_out}" >&2 fi } -## Initialization -__VENV_SOURCED_LIST=${__VENV_SOURCED_LIST:-""} -# Determine the real path of the script -[ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" -# Remove quotes from right-hand side of =~ to match as a regex rather than literally. shellcheck SC2076 -# Don't source this script if it's already been sourced. The SC message is intentional the list is treated like -# string to search for the string in the list/array. -# shellcheck disable=SC2076 -[[ "${__VENV_SOURCED_LIST}" =~ "${THIS_SCRIPT}" ]] && return || __VENV_SOURCED_LIST="${__VENV_SOURCED_LIST} ${THIS_SCRIPT}" -echo "Sourcing: ${THIS_SCRIPT}" - -# Extract script name, directory, and arguments -# MY_NAME appears unused. Verify use (or export if used externally). -# shellcheck disable=SC2034 -MY_NAME=$(basename "${THIS_SCRIPT}") -__VENV_BIN=$(dirname "$(dirname "${THIS_SCRIPT}")") -__VENV_BASE=$(dirname "${__VENV_BIN}") -__VENV_ARGS=$* -__VENV_INCLUDE="${__VENV_BASE}/bin/shinclude" - -# Add internal functions to the __VENV_INTERNAL_FUNCTIONS array. -# Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a. -# shellcheck disable=SC2206 -__VENV_INTERNAL_FUNCTIONS=( - ${__VENV_INTERNAL_FUNCTIONS[@]} - "errval" -) - -# Set default debug level, if not already set. -debug_level=${debug_level:-30} - -# Ensure util_funcs.sh is sourced for utility functions -if declare -f "source_util_script" >/dev/null 2>&1; then - source_util_script "util_funcs" - log_message "INFO" "Sourced util_funcs.sh" -else - # shellcheck source=/dev/null - source "${__VENV_INCLUDE}/util_funcs.sh" - log_message "INFO" "Sourced ${__VENV_INCLUDE}/util_funcs.sh" -fi +# # Function _deprecated +# `_deprecated` - Prints a deprecation warning message to STDERR. +# +# ## Description +# - **Purpose**: +# - Prints a deprecation warning message to STDERR. +# - **Usage**: +# - `_deprecated ` +# - **Input Parameters**: +# - `message`: The message to print. +# - **Output**: +# - Prints a deprecation warning message to STDERR. +# +_deprecated() { + log_message "WARN" "Function ${FUNCNAME[0]} is deprecated. ${*}" +} __rc__=0 +return ${__rc__} \ No newline at end of file diff --git a/bin/shinclude/help_sys.sh b/bin/shinclude/helpsys_lib.sh similarity index 94% rename from bin/shinclude/help_sys.sh rename to bin/shinclude/helpsys_lib.sh index 24ec86a2..e5d76e5e 100755 --- a/bin/shinclude/help_sys.sh +++ b/bin/shinclude/helpsys_lib.sh @@ -1,47 +1,70 @@ #!/usr/bin/env bash - -# # Script: help_sys.sh -# `help_sys.sh` - Help System Functions for Bash Scripts +# # Script: helpsys_lib.sh +# `helpsys_lib.sh` - Help System Functions for Bash Scripts # ## Description # - **Purpose**: # - Provides a dynamic help system for all sourced bash scripts. # - It can list available functions, provide detailed information about each function, and list sourced scripts. -# - **Usage**: +# ## Usage # - Source this script in other bash scripts to enable the dynamic help system. # ```bash -# source help_sys.sh +# source helpsys_lib.sh # ``` -# - **Input Parameters**: +# ## Input Parameters # - None. All input is handled by the individual functions. -# - **Output**: +# ## Output # - Enables a help system that can be accessed by calling `help` in the terminal. # - Supports generating Markdown documentation. -# - **Exceptions**: +# ## Exceptions # - Some functions may return specific error codes or print error messages to STDERR. # - Refer to individual function documentation for details. -# - **Environment**: +# ## Environment # - **MD_PROCESSOR**: Set to the markdown processor of your choice. If `glow` is in your path, it will use that. -# Determine the real path of the script +## Initialization [ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" -# Don't source this script if it's already been sourced. -# shellcheck disable=SC2076 -[[ "${__VENV_SOURCED_LIST}" =~ "${THIS_SCRIPT}" ]] && return || __VENV_SOURCED_LIST="${__VENV_SOURCED_LIST} ${THIS_SCRIPT}" -echo "Sourcing: ${THIS_SCRIPT}" - -# Extract script name, directory, and arguments -declare -g __VENV_BIN -declare -g __VENV_BASE -declare -g __VENV_ARGS=$* +if ! declare -p __VENV_SOURCED >/dev/null 2>&1; then declare -g -A __VENV_SOURCED; fi +if [[ "${__VENV_SOURCED[${THIS_SCRIPT}]:-}" == 1 ]]; then + # echo "************************* SKIPPED SKIPPED SKIPPED SKIPPED ************************* -----> $(basename "${THIS_SCRIPT}")" >&2 + return +fi +__VENV_SOURCED["${THIS_SCRIPT}"]=1 + +# shellcheck disable=SC2034 +MY_NAME=$(basename "${THIS_SCRIPT}") __VENV_BIN=$(dirname "$(dirname "${THIS_SCRIPT}")") __VENV_BASE=$(dirname "${__VENV_BIN}") +__VENV_INCLUDE="${__VENV_BASE}/bin/shinclude" + +# Dependencies +# Get the init_lib.sh script +# shellcheck source=/dev/null +# source "${__VENV_INCLUDE}/init_lib.sh" +# Get the errno_lib.sh script +# source_lib errno_lib +# Get the util_lib.sh script +# source_lib util_lib + +# Initialize associative arrays to store function names and their corresponding documentation +declare -g -A __VENV_FUNCTIONS +declare -g -A __VENV_SCRIPTS + +# Initialize arrays to store sorted names for functions and scripts +declare -g -a sorted_function_names +declare -g -a sorted_script_names + +# Initialize variables to store the length of the longest script and function names +declare -g longest_script_name=0 +declare -g longest_function_name=0 # Help System Initialization # Use an environment variable for markdown processor, defaulting to 'glow' declare -g MD_PROCESSOR=${MD_PROCESSOR:-"glow"} -# Define an array of internal functions to exclude from help and documentation +# Add internal functions to the __VENV_INTERNAL_FUNCTIONS array. +if ! declare -p __VENV_INTERNAL_FUNCTIONS >/dev/null 2>&1; then declare -ga __VENV_INTERNAL_FUNCTIONS; fi +# Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a # shellcheck disable=SC2206 declare -g -a __VENV_INTERNAL_FUNCTIONS=( ${__VENV_INTERNAL_FUNCTIONS[@]} @@ -72,18 +95,6 @@ declare -g -a __VENV_INTERNAL_FUNCTIONS=( "write_table_footer" ) -# Initialize associative arrays to store function names and their corresponding documentation -declare -g -A __VENV_FUNCTIONS -declare -g -A __VENV_SCRIPTS - -# Initialize arrays to store sorted names for functions and scripts -declare -g -a sorted_function_names -declare -g -a sorted_script_names - -# Initialize variables to store the length of the longest script and function names -declare -g longest_script_name=0 -declare -g longest_function_name=0 - # # Function: process_scripts # `process_scripts` - Process scripts in a given directory. # ## Description @@ -156,7 +167,7 @@ process_scripts() { # - **Scope**: # - Global. Modifies the global array `__VENV_FUNCTIONS`. # - **Input Parameters**: -# - None. Internally iterates over the scripts listed in the `__VENV_SOURCED_LIST` array. +# - None. # - **Output**: # - Populates `__VENV_FUNCTIONS` with function names and their corresponding documentation. # - Sorts `__VENV_FUNCTIONS` based on function names. @@ -557,18 +568,18 @@ write_page_footer() { local file_path="$1" local date_mark - date_mark=$(date "+Generated: %Y %m %d at %H:%M:%S") + date_mark=$(date "+Generated on: %Y-%m-%d at %H:%M:%S") { echo "---" echo "" echo "Website: [unixwzrd.ai](https://unixwzrd.ai)" echo "Github Repo: [venvutil](https://github.com/unixwzrd/venvutil)" - echo "Copyright (c) 2024 Michael Sullivan" + echo "Copyright (c) 2025 Michael Sullivan" echo "Apache License, Version 2.0" echo "" echo "Generated Markdown Documentation" - echo "Generated on: ${date_mark}" + echo "${date_mark}" # Add other footer content here } >> "${file_path}" } @@ -597,7 +608,7 @@ generate_markdown() { local conf_file="conf/help_sys.conf" local shdoc_base shdoc_base="$(docs_base_path)" - [ -d "${shdoc_base}" ] || mkdir -p "${shdoc_base}" + mkdir -p "${shdoc_base}" local in_progress_timestamp="${shdoc_base}/.in-progress" local completed_timestamp="${shdoc_base}/AUTO_GENERATED_DO_NOT_MODIFY_OR_PLACE_FILES_HERE" @@ -615,8 +626,7 @@ generate_markdown() { local scripts_docs_dir="${shdoc_base}/${script_dir}" local script_docs_path="${scripts_docs_dir}/scripts" local function_docs_path="${scripts_docs_dir}/functions" - [ -d "${function_docs_path}" ] || mkdir -p "${function_docs_path}" - [ -d "${script_docs_path}" ] || mkdir -p "${script_docs_path}" + mkdir -p "${function_docs_path}" "${script_docs_path}" local script_files readarray -t script_files < <(file "${script_dir}"/* | grep "shell script" | cut -d":" -f1) @@ -654,8 +664,8 @@ generate_markdown() { # In the script, skip blank lines, lines with only whitespace, or # lines beginning with '#' optionally followed by whitespace - if [[ "${script_entry}" == true && ("${line}" =~ ^[[:space:]]*$ || \ - "${line}" =~ ^#[[:space:]]*$) ]]; then + if [[ "${script_entry}" == true && \ + ( "${line}" =~ ^[[:space:]]*$ || "${line}" =~ ^#[[:space:]]*$ ) ]]; then continue fi @@ -746,6 +756,8 @@ generate_markdown() { # Now find and delete old markdown files find "${shdoc_base}" -type f -name '*.md' ! -newer "${completed_timestamp}" -exec rm {} \; printf "\n" + # disable this check because it is going to return anyway. + # shellcheck disable=SC2164 cd "${current_dir}" > /dev/null 2>&1 } @@ -941,14 +953,10 @@ vhelp() { # Use the markdown processor if available, otherwise default to 'cat' command -v "${MD_PROCESSOR}" > /dev/null 2>&1 && md_command="${MD_PROCESSOR}" || md_command="cat" - # Check if the subcommand is a known script name (without the .sh extension) - # shellcheck disable=SC2068 - for script in ${__VENV_SOURCED_LIST[@]}; do - if [[ "${script##*/}" == "${subcommand}" ]]; then - is_script=1 - break - fi - done + # Check if the subcommand is a known script name + if [[ -n "${__VENV_SCRIPTS["${subcommand}"]:-""}" ]]; then + is_script=1 + fi case "${subcommand}" in "generate_markdown") @@ -974,3 +982,6 @@ vhelp() { ;; esac } + +__rc__=0 +return ${__rc__} \ No newline at end of file diff --git a/bin/shinclude/init_env.sh b/bin/shinclude/init_env.sh deleted file mode 100755 index 3259bf4c..00000000 --- a/bin/shinclude/init_env.sh +++ /dev/null @@ -1,103 +0,0 @@ -#!/usr/bin/env bash - -# # Script: init_env.sh -# `init_env.sh` - Initialize Environment and Source Utility Scripts -# ## Description -# - **Purpose**: -# - Initializes the environment for bash scripting, particularly in the context of managing virtual environments. It sets up the necessary environment and sources utility scripts required for the proper functioning of other scripts in the system. It is responsible for orchestrating the environment setup in the correct order and can also be used to source additional environment or setup scripts as required, such as `.env.local` files. -# - **Usage**: -# - Source this script in other bash scripts to import the necessary environment and utility -# functions. It also contains a function that can be called to perform environment setup tasks -# in user scripts. To use it, include the following line in your bash scripts: -# ```bash -# source /path/to/init_env.sh -# ``` -# - **Input Parameters**: -# - None. The script operates without requiring any input parameters. -# - **Output**: -# - Sets up the environment, sources utility scripts, and prepares the system for managing virtual environments. -# - **Exceptions**: -# - Exits with code 1 if it fails to find any of the required scripts or if any part of the -# initialization process fails. -# -# ## Dependencies -# - Utility scripts located in `__VENV_INCLUDE`: -# - `util_funcs.sh` -# - `help_sys.sh` -# - `errno.sh` -# - `venv_funcs.sh` -# - `wrapper_funcs.sh` -# - Conda environment - - -# # Function: source_util_script -# `source_util_script` - Sources a utility script from the specified directory. -# ## Description -# - **Purpose**: -# - Sources a utility script from the specified directory. It's a helper function used within the -# `init_env.sh` script to modularity load additional scripts as needed. -# - **Usage**: -# - `source_util_script "script_name"` -# - **Input Parameters**: -# - `script_name`: The name of the script to source (without the `.sh` extension). -# - **Output**: -# - Sources the specified script if found. Otherwise, outputs an error message and returns with -# an exit code of 1. -# - **Exceptions**: -# - Returns with exit code 1 if the specified script is not found in the directory `__VENV_INCLUDE`. -# -source_util_script() { - local script_name="$1" - local script_path="${__VENV_INCLUDE}/${script_name}.sh" - if [ -f "$script_path" ]; then - # shellcheck source=/dev/null - . "$script_path" - else - echo "${__VENV_INCLUDE}: Error sourcing script ${script_name}.sh in INCLUDE_DIR: ${__VENV_INCLUDE}" >&2 - return 1 - fi -} - -## Initialization -# Determine the real path of the script -[ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" -# Don't source this script if it's already been sourced. -# shellcheck disable=SC2076 -[[ "${__VENV_SOURCED_LIST}" =~ "${THIS_SCRIPT}" ]] && return || __VENV_SOURCED_LIST="${__VENV_SOURCED_LIST} ${THIS_SCRIPT}" -echo "Sourcing: ${THIS_SCRIPT}" - -# Extract script name, directory, and arguments -__VENV_BIN=$(dirname "$(dirname "${THIS_SCRIPT}")") -__VENV_BASE=$(dirname "${__VENV_BIN}") -__VENV_ARGS=$* -__VENV_INCLUDE="${__VENV_BASE}/bin/shinclude" - -# shellcheck disable=SC2206 -__VENV_INTERNAL_FUNCTIONS=( - ${__VENV_INTERNAL_FUNCTIONS[@]} - "source_util_script" -) - -# Initialize Conda environment -# shellcheck disable=SC2016 -__conda_setup="$('${HOME}/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)" -if [ $? -eq 0 ]; then - eval "$__conda_setup" -else - if [ -f "${HOME}/miniconda3/etc/profile.d/conda.sh" ]; then - # shellcheck source=/dev/null - . "${HOME}/miniconda3/etc/profile.d/conda.sh" - else - export PATH="${HOME}/miniconda3/bin:$PATH" - fi -fi -unset __conda_setup - -# Source utility functions -source_util_script "util_funcs" -source_util_script "errno" -source_util_script "help_sys" -source_util_script "wrapper_funcs" -source_util_script "venv_funcs" - -__rc__=0 diff --git a/bin/shinclude/init_lib.sh b/bin/shinclude/init_lib.sh new file mode 100755 index 00000000..389cc194 --- /dev/null +++ b/bin/shinclude/init_lib.sh @@ -0,0 +1,169 @@ +#!/usr/bin/env bash +# # Script: init_lib.sh +# `init_lib.sh` - Library Initialization and Environment Setup +# +# ## Description +# - **Purpose**: +# - Initializes the environment for bash scripting, particularly in the context of managing virtual environments. It sets up the necessary environment and sources utility scripts required for the proper functioning of other scripts in the system. It is responsible for orchestrating the environment setup in the correct order and can also be used to source additional environment or setup scripts as required, such as `.env.local` files. +# ## Usage +# - Source this script in other bash scripts to import the necessary environment and utility +# functions. It also contains a function that can be called to perform environment setup tasks +# in user scripts. To use it, include the following line in your bash scripts: +# ```bash +# source /path/to/init_lib.sh +# ``` +# ## Input Parameters +# - None. The script operates without requiring any input parameters. +# ## Output +# - Sets up the environment, sources utility scripts, and prepares the system for managing virtual environments. +# ## Exceptions +# - Exits with code 1 if it fails to find any of the required scripts or if any part of the +# initialization process fails. +# +# ## Dependencies +# - Utility scripts located in `__VENV_INCLUDE`: +# - `util_lib.sh` +# - `helpsys_lib.sh` +# - `errno_lib.sh` +# - `venv_lib.sh` +# - `wrapper_lib.sh` +# - Conda environment + +## Initialization +[[ -L "${BASH_SOURCE[0]}" ]] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" +# Guard against re-sourcing init_lib.sh +if ! declare -p __VENV_SOURCED >/dev/null 2>&1; then declare -g -A __VENV_SOURCED; fi +if [[ "${__VENV_SOURCED[${THIS_SCRIPT}]:-}" == 1 ]]; then + # echo "************************* SKIPPED SKIPPED SKIPPED SKIPPED ************************* -----> $(basename "${THIS_SCRIPT}")" >&2 + return +fi +# Mark this script as sourced +__VENV_SOURCED[${THIS_SCRIPT}]=1 +# echo "************************* SOURCED SOURCED SOURCED SOURCED ************************* -----> $(basename "${THIS_SCRIPT}")" >&2 + +# Extract script name, directory, and arguments +declare -g __VENV_BIN +declare -g __VENV_BASE +declare -g __VENV_ARGS=$* +MY_NAME=$(basename "${THIS_SCRIPT}") +__VENV_BIN=$(dirname "$(dirname "${THIS_SCRIPT}")") +__VENV_BASE=$(dirname "${__VENV_BIN}") +__VENV_INCLUDE="${__VENV_BASE}/bin/shinclude" + +# Add internal functions to the __VENV_INTERNAL_FUNCTIONS array. +if ! declare -p __VENV_INTERNAL_FUNCTIONS >/dev/null 2>&1; then declare -ga __VENV_INTERNAL_FUNCTIONS; fi +# Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a. +# shellcheck disable=SC2206 +__VENV_INTERNAL_FUNCTIONS=( + ${__VENV_INTERNAL_FUNCTIONS[@]} +) + +# Stack value +if ! declare -p __sv__ >/dev/null 2>&1; then declare -g __sv__=0; fi +# Return code +if ! declare -p __rc__ >/dev/null 2>&1; then declare -g __rc__=0; fi + + +# TODO Figure out why this function does not work in the other lib scripts. +# # Function: _source_check +# `_source_check` - Guard against re-sourcing the same script +# ## Description +# - **Purpose**: +# - Prevents re-sourcing the same script multiple times. +# - **Usage**: +# - `_source_check "${BASH_SOURCE[0]}"` +# - **Input Parameters**: +# - `file_to_source`: The path to the script to source. +# - **Output**: +# - Returns 1 if the script has already been sourced, otherwise returns 0. +# - **Exceptions**: +# - None. +# +check_lib() { + local filename="$1" + [ -L "${filename}" ] && filename=$(readlink -f "${filename}") + if [[ "${__VENV_SOURCED[${filename}]:-0}" == 1 ]]; then + return 0 + fi + __VENV_SOURCED[${filename}]=1 + return 1 +} + +_source_check() { + local sh_lib_name="$1" + + # Resolve symlinks + if [[ -L "${sh_lib_name}" ]]; then + sh_lib_name=$(readlink -f "${sh_lib_name}") + fi + if [[ "${__VENV_SOURCED[$sh_lib_name]:-0}" == 1 ]]; then + return 1 + fi + __VENV_SOURCED[$sh_lib_name]=1 + return 0 +} + +# # Function: source_lib +# `source_lib` - Sources a utility script from the specified directory. +# ## Description +# - **Purpose**: +# - Sources a utility script from the specified directory. It's a helper function used within the +# `init_env.sh` script to modularity load additional scripts as needed. +# - Try to locate the SH_LIB directory which contains included scripts and functions in a +# "standard" fashion. places we could look are, in this order: +# - $(dirname "${THIS_SCRIPT}") +# - $(dirname "${THIS_SCRIPT}")/shinclude +# - $HOME/shinclude +# - $HOME/bin directory +# - $HOME/bin/shinclude directory +# - from the environment variable `SH_LIB` +# - **Usage**: +# - `source_lib "script_name"` +# - **Input Parameters**: +# - `script_name`: The name of the script to source (without the `.sh` extension). +# - **Output**: +# - Sources the specified script if found. Otherwise, outputs an error message and returns with +# an exit code of 1. +# - **Exceptions**: +# - Returns with exit code 1 if the specified script is not found in the directory `__VENV_INCLUDE`. +# +source_lib() { + local sh_lib_name="$1" + # echo "************************* SOURCE_LIB SOURCE_LIB SOURCE_LIB SOURCE_LIB ************************* source_lib -----> ${1}" >&2 + # set | grep -E '^__VENV_SOURCED' >&2 + # set | grep -E '^BASH_SOURCE' >&2 + # Doesn't matter if this gets called with a .sh extension, we'll strip it off and put it back. + sh_lib_name="$(basename -s .sh "$sh_lib_name").sh" + local dirname + [ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" + MY_NAME="$(basename "${THIS_SCRIPT}")" + SH_LIB="${SH_LIB:-""}" + + for dirname in "${SH_LIB}" "$(dirname "${THIS_SCRIPT}")" "${THIS_SCRIPT}/shinclude" "${HOME}/shinclude" "${HOME}/bin/shinclude"; do + [ -f "${dirname}/${sh_lib_name}" ] && { SH_LIB="${dirname}"; break; } + done + + if [[ -z "${SH_LIB:-}" ]]; then + cat<<_EOT_ >&2 +ERROR ($MY_NAME): Could not locate \`${sh_lib_name}\` file. +ERROR ($MY_NAME): Please set install ${sh_lib_name} which came with this repository in one of + the following locations: + - $(dirname "${THIS_SCRIPT}")/shinclude + - $HOME/shinclude + - $HOME/bin/shinclude + or set the environment variable SH_LIB to the directory containing ${sh_lib_name} + +_EOT_ + errno_exit ENOENT # (ENOENT: 2): No such file or directory + fi + + # shellcheck source=/dev/null + source "${SH_LIB}/${sh_lib_name}" + return +} + +# Get the venvutil_lib.sh script +# source_lib venvutil_lib + +__rc__=0 +return ${__rc__} diff --git a/bin/shinclude/string_lib.sh b/bin/shinclude/string_lib.sh new file mode 100755 index 00000000..58298d05 --- /dev/null +++ b/bin/shinclude/string_lib.sh @@ -0,0 +1,299 @@ +#!/usr/bin/env bash +# # Script: string_lib.sh +# `string_lib.sh` - Consolidated string sanitization & variable expansion library +# +# ## Description +# - **Purpose**: +# - Provides functions for: +# - Sanitizing strings with a custom "allowed character set" +# - Escaping specific characters with a backslash +# - Expanding shell variables in a string (handling $VAR or ${VAR}) +# ## Usage +# - Source this script in your Bash scripts to utilize its functions. +# ```bash +# source_lib string_lib +# ``` +# ## Input Parameters +# - None +# ## Output +# - None +# ## Exceptions +# - None + +## Initialization +[ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" +if ! declare -p __VENV_SOURCED >/dev/null 2>&1; then declare -g -A __VENV_SOURCED; fi +if [[ "${__VENV_SOURCED[${THIS_SCRIPT}]:-}" == 1 ]]; then + # echo "************************* SKIPPED SKIPPED SKIPPED SKIPPED ************************* -----> $(basename "${THIS_SCRIPT}")" >&2 + return +fi +__VENV_SOURCED["${THIS_SCRIPT}"]=1 + +# shellcheck disable=SC2034 +MY_NAME=$(basename "${THIS_SCRIPT}") +__VENV_BIN=$(dirname "$(dirname "${THIS_SCRIPT}")") +__VENV_BASE=$(dirname "${__VENV_BIN}") +__VENV_INCLUDE="${__VENV_BASE}/bin/shinclude" + +# Get the init_lib.sh script +# shellcheck source=/dev/null +# ource "${__VENV_INCLUDE}/init_lib.sh" + +# Get the type_lib.sh script +# source_lib type_lib + +# Add internal functions to the __VENV_INTERNAL_FUNCTIONS array. +if ! declare -p __VENV_INTERNAL_FUNCTIONS >/dev/null 2>&1; then declare -ga __VENV_INTERNAL_FUNCTIONS; fi +# Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a +# shellcheck disable=SC2206 +__VENV_INTERNAL_FUNCTIONS=( + ${__VENV_INTERNAL_FUNCTIONS[@]} + "colortext" +) + +# # Function: to_upper +# `to_upper` - Convert a String to Uppercase +# +# ## Description +# - **Purpose**: +# - Converts a string to uppercase. +# - **Usage**: +# - `upper_str=$(to_upper "hello")` +# - **Input Parameters**: +# - `str`: The string to convert. +# - **Output**: +# - The uppercase version of the input string. +# - **Exceptions**: +# - None. +# +to_upper() { + local str="$1" + echo "${str^^}" +} + +# # Function: strip_space +# `strip_space` - Remove Leading and Trailing Whitespaces +# +# ## Description +# - **Purpose**: +# - Removes leading and trailing spaces from the input string. +# - **Usage**: +# - `strip_space " string "` +# - **Input Parameters**: +# - `str`: The string from which to remove leading and trailing whitespaces. +# - **Output**: +# - A new string with whitespaces removed from both ends. +# - **Exceptions**: +# - None. +# +strip_space() { + local argstring="$*" + # Remove leading and trailing whitespaces + echo "$argstring" | sed 's/^ *//;s/ *$//' +} + +# # Function: sanitize +# `sanitize` - Removes every character not in the specified allowed set. +# +# ## Description +# - **Purpose**: +# - Removes every character not in the specified allowed set. +# - **Usage**: +# - `sanitized_value=$(sanitize "Hello? *" 'a-zA-Z0-9._*\- ')` +# - **Input Parameters**: +# - `dirty_string`: The string to sanitize. +# - `allowed_chars`: The characters to keep in the string. +# - **Output**: +# - The sanitized string. +# - **Examples**: +# - `sanitize "Hello? *" 'a-zA-Z0-9._*\- '` +# - `sanitize "Hello? *" 'a-zA-Z0-9._*\- '` +# - **Exceptions**: +# - None. +# +sanitize() { + local dirty_string="$1" + local allowed_chars="$2" + local clean_string + + # Use sed or parameter expansion: + # If you have special bracket-chars in allowed_chars, you may need to escape them before sed. + clean_string="$(echo "$dirty_string" | sed "s/[^${allowed_chars}]//g")" + + log_message "DEBUG" "Dirty string: '$dirty_string'" + log_message "DEBUG" "Clean string: '$clean_string'" + + echo "$clean_string" +} + +# # Function: stringclean +# `stringclean` - Sanitize a String by Removing Non-Alphanumeric Characters +# +# ## Description +# - **Purpose**: +# - Sanitizes a string by removing all characters except alphabets and numbers. +# - **Usage**: +# - `cleaned_str=$(stringclean "Hello, World!")` +# - **Input Parameters**: +# - `str`: The string to sanitize. +# - **Output**: +# - The sanitized string containing only alphanumeric characters. +# - **Exceptions**: +# - None. +# +stringclean() { + local str="$1" + _deprecated "Use: \`sanitize \"${str}\" \"[^a-zA-Z0-9]\"\` instead." + sanitize "${str}" "[^a-zA-Z0-9]" +} + +# # Function: escape_string +# `escape_string` - Escape Special Characters in a String +# +# ## Description +# - **Purpose**: +# - Escapes special characters in a string to make it safe for shell commands. +# - **Usage**: +# - `escaped_str=$(escape_string "Hello & goodbye!" '&"<>;')` +# - **Input Parameters**: +# - `dirty_string`: The string to escape. +# - `chars_to_escape`: The characters to escape. +# - **Output**: +# - The escaped string. +# - **Exceptions**: +# - None. +# +escape_string() { + local dirty_string="$1" + local chars_to_escape="$2" + + # Escape sed special chars in the bracket expression + local safe_pattern + safe_pattern="$(echo "$chars_to_escape" \ + | sed 's/\\/\\\\/g; s/\-/\\-/g; s/\]/\\]/g; s/\^/\\^/g')" + + local escaped + escaped="$(echo "$dirty_string" | sed "s/\\([$safe_pattern]\\)/\\\\\\1/g")" + echo "$escaped" +} + +# # Function: colortext +# `colortext` - Prints a message to STDERR with ANSI color codes. +# +# ## Description +# - **Purpose**: +# - Prints a message to STDERR with ANSI color codes. +# - **Usage**: +# - `colortext ` +# - **Options**: +# - `-h` Show this help message +# - **Input Parameters**: +# - `text_color`: The text color to use. +# - `background_color`: The background color to use. +# - `style_code`: The style code to use. +# - `message`: The message to print. +# - **Output**: +# - Prints a message to STDERR with ANSI color codes. +# +colortext() { + # TODO - NOT WORKING + local OPTIND=1 + # Parse options + while getopts "h" opt; do + case $opt in + h) vhelp "${FUNCNAME[0]}"; return 0 ;; + \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; return 1 ;; + esac + done + shift $((OPTIND - 1)) + + # ANSI COLOR CODES + declare -A FOREGROUND_COLORS=( + ["BLACK"]="\033[0;30m" + ["RED"]="\033[0;31m" + ["GREEN"]="\033[0;32m" + ["YELLOW"]="\033[0;33m" + ["BLUE"]="\033[0;34m" + ["MAGENTA"]="\033[0;35m" + ["CYAN"]="\033[0;36m" + ["WHITE"]="\033[0;37m" + ["BRIGHT_BLACK"]="\033[0;90m" + ["BRIGHT_RED"]="\033[0;91m" + ["BRIGHT_GREEN"]="\033[0;92m" + ["BRIGHT_YELLOW"]="\033[0;93m" + ["BRIGHT_BLUE"]="\033[0;94m" + ["BRIGHT_MAGENTA"]="\033[0;95m" + ["BRIGHT_CYAN"]="\033[0;96m" + ["BRIGHT_WHITE"]="\033[0;97m" + ) + + declare -A BACKGROUND_COLORS=( + ["BLACK"]="\033[0;40m" + ["RED"]="\033[0;41m" + ["GREEN"]="\033[0;42m" + ["YELLOW"]="\033[0;43m" + ["BLUE"]="\033[0;44m" + ["MAGENTA"]="\033[0;45m" + ["CYAN"]="\033[0;46m" + ["WHITE"]="\033[0;47m" + ["BRIGHT_BLACK"]="\033[0;100m" + ["BRIGHT_RED"]="\033[0;101m" + ["BRIGHT_GREEN"]="\033[0;102m" + ["BRIGHT_YELLOW"]="\033[0;103m" + ["BRIGHT_BLUE"]="\033[0;104m" + ["BRIGHT_MAGENTA"]="\033[0;105m" + ["BRIGHT_CYAN"]="\033[0;106m" + ["BRIGHT_WHITE"]="\033[0;107m" + ) + + declare -A STYLE_CODES=( + ["BOLD"]="\033[1m" + ["DIM"]="\033[2m" + ["ITALIC"]="\033[3m" + ["UNDERLINE"]="\033[4m" + ["BLINK"]="\033[5m" + ["REVERSE"]="\033[7m" + ["HIDDEN"]="\033[8m" + ["STRIKE"]="\033[9m" + ["DOUBLE_UNDERLINE"]="\033[21m" + ["OVERLINE"]="\033[53m" + ["RESET"]="\033[0m" + ["CLEAR"]="\033[2J" + ) + + # ANSI COLOR END CODE + declare -g ANSI_END="\033[0m" + + local text_color="" + local background_color="" + local style_code="" + local message="" + + # Check if first arg is a valid text color + if [[ -n "$1" && "${FOREGROUND_COLORS[$1]}" ]]; then + text_color="$1" + shift + fi + + # Check if next arg is a valid background color + if [[ -n "$1" && "${BACKGROUND_COLORS[$1]}" ]]; then + background_color="$1" + shift + fi + + # Check if next arg is a valid style code + if [[ -n "$1" && "${STYLE_CODES[$1]}" ]]; then + style_code="$1" + shift + fi + + # Remaining args form the message + message="$*" + + message="${ANSI_END}${text_color}${background_color}${style_code}${message}${ANSI_END}" + echo -e "${message}" +} + + +__rc__=0 +return ${__rc__} diff --git a/bin/shinclude/type_lib.sh b/bin/shinclude/type_lib.sh new file mode 100755 index 00000000..02ab5a44 --- /dev/null +++ b/bin/shinclude/type_lib.sh @@ -0,0 +1,251 @@ +#!/usr/bin/env bash +# # Script: type_lib.sh +# `type_lib.sh` - Support functions for variable type handling. +# ## Description +# - **Purpose**: +# - Offers functions to handle variable types. +# ## Usage +# - Source this script in your Bash scripts to utilize its functions. +# ```bash +# source /path/to/type_lib.sh +# ``` +# ## Input Parameters +# - None. +# ## Output +# - Sets variables from the setup.cf file for package installation. +# ## Exceptions +# - Returns specific error codes if the setup.cf file is not found or invalid. +# ## Initialization +# - Ensures the script is sourced only once and initializes necessary variables. + +## Initialization +[ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" +if ! declare -p __VENV_SOURCED >/dev/null 2>&1; then declare -g -A __VENV_SOURCED; fi +if [[ "${__VENV_SOURCED[${THIS_SCRIPT}]:-}" == 1 ]]; then + # echo "************************* SKIPPED SKIPPED SKIPPED SKIPPED ************************* -----> $(basename "${THIS_SCRIPT}")" >&2 + return +fi +__VENV_SOURCED["${THIS_SCRIPT}"]=1 +# echo "************************* READING READING READING READING ************************* -----> $(basename "${THIS_SCRIPT}")" >&2 + +# shellcheck disable=SC2034 +MY_NAME=$(basename "${THIS_SCRIPT}") +__VENV_BIN=$(dirname "$(dirname "${THIS_SCRIPT}")") +__VENV_BASE=$(dirname "${__VENV_BIN}") +__VENV_INCLUDE="${__VENV_BASE}/bin/shinclude" + +# Get the init_lib.sh script +# shellcheck source=/dev/null +# source "${__VENV_INCLUDE}/init_lib.sh" + +# Get the errno_lib.sh script +# source_lib errno_lib + +# Add internal functions to the __VENV_INTERNAL_FUNCTIONS array. +if ! declare -p __VENV_INTERNAL_FUNCTIONS >/dev/null 2>&1; then declare -ga __VENV_INTERNAL_FUNCTIONS; fi +# shellcheck disable=SC2206 +__VENV_INTERNAL_FUNCTIONS=( + ${__VENV_INTERNAL_FUNCTIONS[@]} +) + +# # Function: var_type +# `var_type` - Get the Type of a Variable +# +# ## Description +# - **Purpose**: +# - Retrieves the type of a variable. +# - **Usage**: +# - `var_type [-h] "var_name"` +# - **Options**: +# - `-h` Show this help message +# - **Examples**: +# - `var_type "my_variable"` +# - `var_type=$(var_type "my_variable") +# - **Input Parameters**: +# - `var_name`: The name of the variable. +# - **Output**: +# - The type of the variable as a string. Can be one of `array`, `associative`, `scalar`, or `unknown`. +# - **Exceptions**: +# - None. +var_type() { + local OPTIND=1 + # Parse options + while getopts "h" opt; do + case $opt in + h) vhelp "${FUNCNAME[0]}"; return 0 ;; + \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; return 1 ;; + esac + done + shift $((OPTIND - 1)) + + local var_name="$1" + local var_type + var_type=$(declare -p "$var_name" 2>/dev/null | cut -d ' ' -f 2) + + __rc__=0 + case "$var_type" in + -a) + echo "array" + ;; + -A) + echo "associative" + ;; + -i) + echo "integer" + ;; + --) + echo "scalar" + ;; + -ar) + echo "read only array" + __rc__=1 + ;; + -Ar) + echo "read only associative" + __rc__=1 + ;; + -ir) + echo "read only integer" + __rc__=1 + ;; + -r) + echo "read only scalar" + __rc__=1 + ;; + *) + echo "unknown" + __rc__=93 + ;; + esac + + return ${__rc__} +} + +# # Function: set_variable +# `set_variable` - Assign a value to a named variable, respecting existing type or inferring new +# +# ## Description +# - **Purpose**: +# - Assigns a value to a named variable, respecting its existing type or inferring a new type. +# - Handles arrays, associative arrays, scalars, and integers. +# - **Usage**: +# - `set_variable "var_name" "value_ref1"` +# - `set_variable "var_name" "value_ref1" "value_ref2"` +# - **Input Parameters**: +# - `var_name`: The name of the variable to set. +# - `value_ref1`: Name reference to first value/array to assign. +# - `value_ref2`: (Optional) Name reference to second value/array to merge with first. +# - **Output**: +# - None. Sets the target variable with the provided value(s). +# - **Exceptions**: +# - None. Will create new variable if it doesn't exist. +set_variable() { + local var_name=$1 + local -n var_ref1=$2 + local var_ref2="" + if [ -n "${3:-}" ]; then + local -n var_ref2=$3 + fi + + # log_message "DEBUG2" "assign_variable: var_name=$var_name, var_value=\"$var_value\", var_type=$var_type" + + local var_type + var_type=$(var_type "$var_name") + + case "$var_type" in + "array") + declare -g -a "${var_name}" + local new_array=() + new_array=( "${var_ref1[@]}" "${var_ref2[@]}" ) + readarray -t "${var_name}" < <(for element in "${new_array[@]}"; do echo "$element"; done) + ;; + "associative") + declare -g -A "${var_name}" + # shellcheck disable=SC2178 + declare -n new_array="${var_name}" + for key in "${!var_ref1[@]}"; do + new_array[$key]="${var_ref1[$key]}" + done + if [ -n "$var_ref2" ]; then + for key in "${!var_ref2[@]}"; do + if [ -n "${new_array[$key]}" ]; then + new_array[$key]="${new_array[$key]}, ${var_ref2[$key]}" + else + new_array[$key]="${var_ref2[$key]}" + fi + done + fi + ;; + "scalar") + declare -g "${var_name}"="${var_ref1}" + ;; + "integer") + declare -g "${var_name}"="${var_ref1}" + ;; + *) + declare -g "${var_name}"="${var_ref1:-}" + ;; + esac +} + +# # Function: handle_variable +# `handle_variable` - Manage variable assignments based on predefined actions +# +# ## Description +# - **Purpose**: +# - Controls how variables are assigned based on a predefined action table. +# - Supports merging, setting, preserving config values, or discarding changes. +# - **Usage**: +# - `handle_variable "var_name" "value_ref"` +# - **Input Parameters**: +# - `var_name`: The name of the variable to handle. +# - `value_ref`: Name reference to the value to potentially assign. +# - **Output**: +# - None. Modifies the target variable according to its action rule. +# - **Exceptions**: +# - Exits with code 2 if the action for the variable is unknown. +# - **Required Global Variables**: +# - `var_actions`: Associative array mapping variable names to actions: +# - "merge": Combines existing value with new value +# - "set": Uses the new value +# - "config": Preserves the config file value +# - "discard": Keeps the original value +handle_variable() { + local var_name=$1 + [[ -z "$2" || "$2" =~ ^[[:space:]]*$ ]] && + { errno_warn EINVAL "handle_variable: var_name=$var_name, value_ref='$2'"; return $?; } + + # shellcheck disable=SC2034 + local -n var_name_val=$2 + # shellcheck disable=SC2034 + local -n current_value=${var_name} + + # TODO Handle different action tables. + + # shellcheck disable=SC2154 + case "${var_actions["$var_name"]}" in + "merge") + # Merge the current value with the value from the config file + set_variable "$var_name" "current_value" "var_name_val" + ;; + "set") + # Set the variable from the command line or defaults + set_variable "$var_name" "var_name_val" + ;; + "config") + # Set the variable from the config file setting + set_variable "$var_name" "current_value" + ;; + "discard") + # Preserve the original value + set_variable "$var_name" "current_value" + ;; + *) + log_message "ERROR" "Unknown config_variable_action for variable \"$var_name\": ${var_actions["$var_name"]}" + exit 2 # (ENOENT: 2): No such file or directory + ;; + esac +} + +__rc__=0 +return ${__rc__} diff --git a/bin/shinclude/util_funcs.sh b/bin/shinclude/util_lib.sh similarity index 73% rename from bin/shinclude/util_funcs.sh rename to bin/shinclude/util_lib.sh index a0aa6b6c..d2046cf7 100755 --- a/bin/shinclude/util_funcs.sh +++ b/bin/shinclude/util_lib.sh @@ -1,49 +1,53 @@ #!/usr/bin/env bash - -# # Script: util_funcs.sh -# `util_funcs.sh` - Utility Functions for Bash Scripts +# # Script: util_lib.sh +# `util_lib.sh` - Utility Functions and Cross-Platform Support +# # ## Description # - **Purpose**: -# - Provides a collection of utility functions to assist in various common tasks within Bash scripting. These functions offer streamlined solutions for string manipulation, number padding, and stack operations, enhancing the efficiency and readability of Bash scripts that incorporate them. -# - **Usage**: +# - Provides a collection of utility functions to assist in various common tasks within Bash scripting. These functions offer streamlined solutions for number padding, stack operations, and process tree display, enhancing the efficiency and readability of Bash scripts that incorporate them. +# ## Usage # - Source this script within other Bash scripts to make the utility functions available for use: # ```bash -# source /path/to/util_funcs.sh +# source /path/to/util_lib.sh # ``` -# - **Input Parameters**: +# ## Input Parameters # - None. This script is intended to be sourced by other scripts and does not require input parameters. -# - **Output**: +# ## Output # - Provides utility functions that can be called from other Bash scripts. -# - **Exceptions**: +# ## Exceptions # - Some functions within the script may return specific error codes depending on their internal logic. Refer to the individual function documentation for detailed exception handling. # # ## Dependencies # - None explicitly stated. The script is designed to be self-contained, relying only on standard Bash features. - - -# Utility functions - -# # Function: strip_space -# `strip_space` - Remove Leading and Trailing Whitespaces -# -# ## Description -# - **Purpose**: -# - Removes leading and trailing spaces from the input string. -# - **Usage**: -# - `strip_space " string "` -# - **Input Parameters**: -# - `str`: The string from which to remove leading and trailing whitespaces. -# - **Output**: -# - A new string with whitespaces removed from both ends. -# - **Exceptions**: -# - None. -# -strip_space() { - local argstring="$*" - # Remove leading and trailing whitespaces - echo "$argstring" | sed 's/^ *//;s/ *$//' -} +## Initialization +[ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" +if ! declare -p __VENV_SOURCED >/dev/null 2>&1; then declare -g -A __VENV_SOURCED; fi +if [[ "${__VENV_SOURCED[${THIS_SCRIPT}]:-}" == 1 ]]; then + # echo "************************* SKIPPED SKIPPED SKIPPED SKIPPED ************************* -----> $(basename "${THIS_SCRIPT}")" >&2 + return +fi +__VENV_SOURCED["${THIS_SCRIPT}"]=1 + +# shellcheck disable=SC2034 +MY_NAME=$(basename "${THIS_SCRIPT}") +__VENV_BIN=$(dirname "$(dirname "${THIS_SCRIPT}")") +__VENV_BASE=$(dirname "${__VENV_BIN}") +__VENV_INCLUDE="${__VENV_BASE}/bin/shinclude" + +# Get the init_lib.sh script +# shellcheck source=/dev/null +# source "${__VENV_INCLUDE}/init_lib.sh" + +# get the string_lib.sh script +# source_lib string_lib + +# Add internal functions to the __VENV_INTERNAL_FUNCTIONS array. +if ! declare -p __VENV_INTERNAL_FUNCTIONS >/dev/null 2>&1; then declare -ga __VENV_INTERNAL_FUNCTIONS; fi +# shellcheck disable=SC2206 +__VENV_INTERNAL_FUNCTIONS=( + ${__VENV_INTERNAL_FUNCTIONS[@]} +) # # Function: zero_pad # `zero_pad` - Pad a Single-Digit Number with a Leading Zero @@ -325,46 +329,6 @@ stack_op() { esac } -# # Function: stringclean -# `stringclean` - Sanitize a String by Removing Non-Alphanumeric Characters -# -# ## Description -# - **Purpose**: -# - Sanitizes a string by removing all characters except alphabets and numbers. -# - **Usage**: -# - `cleaned_str=$(stringclean "Hello, World!")` -# - **Input Parameters**: -# - `str`: The string to sanitize. -# - **Output**: -# - The sanitized string containing only alphanumeric characters. -# - **Exceptions**: -# - None. -# -stringclean() { - local str="$1" - echo "${str//[^a-zA-Z0-9]/}" -} - -# # Function: to_upper -# `to_upper` - Convert a String to Uppercase -# -# ## Description -# - **Purpose**: -# - Converts a string to uppercase. -# - **Usage**: -# - `upper_str=$(to_upper "hello")` -# - **Input Parameters**: -# - `str`: The string to convert. -# - **Output**: -# - The uppercase version of the input string. -# - **Exceptions**: -# - None. -# -to_upper() { - local str="$1" - echo "${str^^}" -} - # # Function: ptree # `ptree` - Display the Process Tree for a Given PID # @@ -388,95 +352,37 @@ ptree() { # Parse options while getopts "h" opt; do case $opt in - h) vhelp "${FUNCNAME[0]}"; return 0 ;; - \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; return 1 ;; + h) + vhelp "${FUNCNAME[0]}" + return 0 + ;; + \?) + echo "Invalid option: -$OPTARG" >&2 + vhelp "${FUNCNAME[0]}" + return 1 + ;; esac done shift $((OPTIND - 1)) local pid="$1" local indent="${2:-" "}" - - # Get terminal width - local term_width - term_width=$(tput cols) - # Calculate effective width for command output - local effective_width=$((term_width - ${#indent} - 14)) + # Get terminal width and calculate effective width for the command output + local term_width effective_width + term_width=$(tput cols) + effective_width=$(( term_width - ${#indent} - 14 )) # Display the current process with indentation and truncate command based on effective width ps -o pid,ppid,command -p "$pid" | awk -v indent="$indent" -v width="$effective_width" 'NR>1 {printf "%s%s %s %s\n", indent, $1, $2, substr($0, index($0,$3), width)}' - # Get child processes + # Get child processes using ps and awk (more reliable than pgrep -P in some environments) local children - children=$(pgrep -P "$pid") - - # Recurse for each child process + children=$(ps -eo pid,ppid | awk -v ppid="$pid" '$2==ppid {print $1}') for child in $children; do ptree "$child" " $indent" done } -# # Function: var_type -# `var_type` - Get the Type of a Variable -# -# ## Description -# - **Purpose**: -# - Retrieves the type of a variable. -# - **Usage**: -# - `var_type [-h] "var_name"` -# - **Options**: -# - `-h` Show this help message -# - **Examples**: -# - `var_type "my_variable"` -# - `var_type=$(var_type "my_variable") -# - **Input Parameters**: -# - `var_name`: The name of the variable. -# - **Output**: -# - The type of the variable as a string. Can be one of `array`, `associative`, `scalar`, or `unknown`. -# - **Exceptions**: -# - None. -var_type() { - local OPTIND=1 - # Parse options - while getopts "h" opt; do - case $opt in - h) vhelp "${FUNCNAME[0]}"; return 0 ;; - \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; return 1 ;; - esac - done - shift $((OPTIND - 1)) - - local var_name="$1" - local var_type - var_type=$(declare -p "$var_name" 2>/dev/null | cut -d ' ' -f 2) - - case "$var_type" in - -a) - echo "array" - ;; - -A) - echo "associative" - ;; - --) - echo "scalar" - ;; - *) - echo "unknown" - ;; - esac -} - - -# Capture the fully qualified path of the sourced script -[ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" -# Don't source this script if it's already been sourced. -# The RHS has to be in "" to match the array. -# shellcheck disable=SC2076 -[[ "${__VENV_SOURCED_LIST}" =~ "${THIS_SCRIPT}" ]] && return || __VENV_SOURCED_LIST="${__VENV_SOURCED_LIST} ${THIS_SCRIPT}" -echo "Sourcing: ${THIS_SCRIPT}" - -# shellcheck disable=SC2206 -__VENV_INTERNAL_FUNCTIONS=( - ${__VENV_INTERNAL_FUNCTIONS[@]} -) \ No newline at end of file +__rc__=0 +return ${__rc__} diff --git a/bin/shinclude/venv_funcs.sh b/bin/shinclude/venv_lib.sh similarity index 69% rename from bin/shinclude/venv_funcs.sh rename to bin/shinclude/venv_lib.sh index 1e75f336..10510245 100755 --- a/bin/shinclude/venv_funcs.sh +++ b/bin/shinclude/venv_lib.sh @@ -1,13 +1,12 @@ #!/usr/bin/env bash - -# # Script: venv_funcs.sh -# venv_funcs.sh - Virtual Environment Management Functions for Bash Scripts -# ## Description -# - **Purpose**: -# - This script provides a collection of functions to manage conda virtual environments. -# - Functions include creating, deleting, switching, and cloning environments, among others. +# # Script: venv_lib.sh +# `venv_lib.sh` - Virtual Environment Management Functions for Bash Scripts +# ## Description +# - **Purpose**: +# - This script provides a collection of functions to manage conda virtual environments. +# - Functions include creating, deleting, switching, and cloning environments, among others. # -# ## Usage +# ## Usage # - Source this script in other bash scripts to import the virtual environment management functions. # - For example, in another script: `source venv_funcs.sh`. # @@ -41,29 +40,72 @@ # - `lastenv()`: Retrieve the Last Environment with a Given Prefix. # - `benv()`: Create a New Base Virtual Environment. # - `nenv()`: Create a New Virtual Environment in a Series. +# - `vren()`: Rename a Virtual Environment # -# - **Usage Example**: +# ## Usage Example # ```shellscript -# source venv_funcs.sh +# source venv_lib.sh # benv myenv # cact myenv # ``` # -# - **Dependencies**: +# ## Dependencies # - This script depends on the `conda` command-line tool for managing virtual environments. # - The `util_funcs.sh` script is also required and should be located in the same directory as this script. # -# - **Notes**: +# ## Notes # - This script assumes that the `conda` command is available in the system's PATH. # - It is recommended to source this script in other scripts rather than executing it directly. # - Make sure to set the appropriate permissions on this script to allow execution. # -# - **Author**: [Your Name] -# - **Last Modified**: [Date] -# -# - **Version**: [Version Number] +# ## Author +# - Michael Sullivan +# - https://unixwzrd.ai/ +# - https://github.com/unixwzrd +# - https://github.com/unixwzrd/venvutil # +## Initialization +[ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" +if ! declare -p __VENV_SOURCED >/dev/null 2>&1; then declare -g __VENV_SOURCED; fi +if [[ "${__VENV_SOURCED[${THIS_SCRIPT}]:-}" == 1 ]]; then + # echo "************************* SKIPPED SKIPPED SKIPPED SKIPPED ************************* -----> $(basename "${THIS_SCRIPT}")" >&2 + return +fi +__VENV_SOURCED["${THIS_SCRIPT}"]=1 + +# shellcheck disable=SC2034 +MY_NAME=$(basename "${THIS_SCRIPT}") +__VENV_BIN=$(dirname "$(dirname "${THIS_SCRIPT}")") +__VENV_BASE=$(dirname "${__VENV_BIN}") +__VENV_INCLUDE="${__VENV_BASE}/bin/shinclude" + +# Get the init_lib.sh script +# shellcheck source=/dev/null +source "${__VENV_INCLUDE}/init_lib.sh" + +# Get the helpsys_lib.sh script +# source_lib helpsys_lib + +# Get the util_lib.sh script +# source_lib util_lib + +# Get the wrapper_lib.sh script +# source_lib wrapper_lib + +# Add internal functions to the __VENV_INTERNAL_FUNCTIONS array. +if ! declare -p __VENV_INTERNAL_FUNCTIONS >/dev/null 2>&1; then declare -ga __VENV_INTERNAL_FUNCTIONS; fi +# shellcheck disable=SC2206 +__VENV_INTERNAL_FUNCTIONS=( + ${__VENV_INTERNAL_FUNCTIONS[@]} + "push_venv" + "pop_venv" + "__set_venv_vars" +) + +# Initialize the stack +declare -ga __VENV_STACK=() + # # Function: push_venv # `push_venv` - Specialized push the default VENV onto the stack. @@ -102,7 +144,8 @@ push_venv() { pop_venv() { local stack_value pop_stack __VENV_STACK - stack_value=${__sv__} + # shellcheck disable=SC2154 + echo "${__sv__}" return "${__rc__}" } @@ -242,6 +285,7 @@ vdsc() { # - `cact [-h] [env_name]` # - **Options**: # - `-h` Show this help message +# - `-x` Enable debug mode # - **Input Parameters**: # - `env_name` (string) - The name of the environment to activate. # - **Output**: @@ -252,9 +296,10 @@ vdsc() { cact() { local OPTIND=1 # Parse options - while getopts "h" opt; do + while getopts "hx" opt; do case $opt in h) vhelp "${FUNCNAME[0]}"; return 0 ;; + x) set -x; local set_here="y" ;; \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; return 1 ;; esac done @@ -275,7 +320,7 @@ cact() { fi # Pop from stack if top of stack matches the new environment - if [[ "${__VENV_STACK[${#__VENV_STACK[@]}]}" == "$new_env" ]]; then + if [[ "${__VENV_STACK[${#__VENV_STACK[@]}]:-}" == "$new_env" ]]; then pop_venv fi @@ -290,6 +335,9 @@ cact() { # Activate new environment echo "Activating new environment: ${__VENV_NAME}..." conda activate "${__VENV_NAME}" || { echo "Error: Failed to activate new environment." 1>&2; return 1; } + if [ "${set_here:-}" == 'y' ]; then + set +x + fi } # # Function: dact @@ -302,6 +350,7 @@ cact() { # - `dact [-h]` # - **Options**: # - `-h` Show this help message +# - `-x` Enable debug mode # - **Input Parameters**: # - None # - **Output**: @@ -312,9 +361,10 @@ cact() { dact() { local OPTIND=1 # Parse options - while getopts "h" opt; do + while getopts "hx" opt; do case $opt in h) vhelp "${FUNCNAME[0]}"; return 0 ;; + x) set -x; local set_here="y" ;; \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; return 1 ;; esac done @@ -343,6 +393,9 @@ dact() { pop_venv # shellcheck disable=SC2034 stack_value="${__sv__}" + if [ "${set_here:-}" == 'y' ]; then + set +x + fi return "${__rc__}" } @@ -356,6 +409,7 @@ dact() { # - `pact [-h]` # - **Options**: # - `-h` Show this help message +# - `-x` Enable debug mode # - **Input Parameters**: # - None # - **Output**: @@ -367,9 +421,10 @@ pact() { local OPTIND=1 # Parse options - while getopts "h" opt; do + while getopts "hx" opt; do case $opt in h) vhelp "${FUNCNAME[0]}"; return 0 ;; + x) set -x; local set_here="y" ;; \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; return 1 ;; esac done @@ -385,6 +440,9 @@ pact() { else echo "No previous environment to switch to." fi + if [ "${set_here:-}" == 'y' ]; then + set +x + fi } # # Function: lenv @@ -403,6 +461,7 @@ pact() { # - `-t` Sort by last update time # - `-r` Reverse the sort order # - `-h` Show this help message +# - `-x` Enable debug mode # - **Output**: # - A list of all existing conda virtual environments with their last modification date. # - The active environment is marked with an asterisk. @@ -419,16 +478,18 @@ lenv() { local sort_by_time=false local sort_opts="" local time_opts=" " - local sort_key="2" + local sort_key="3" + local date_spacing="11" local OPTIND=1 # Parse options - while getopts "ltrh" opt; do + while getopts "ltrhx" opt; do case $opt in t) sort_by_time=true ;; r) sort_opts="-r" ;; - l) time_opts="."; sort_key="3";; + l) time_opts="."; sort_key="3"; date_spacing="20";; h) vhelp "${FUNCNAME[0]}"; return 0 ;; + x) set -x; local set_here="y" ;; \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; return 1 ;; esac done @@ -436,48 +497,77 @@ lenv() { # Get environment info excluding comments local envs_info - envs_info=$(conda info --envs | grep -E -v '^#') - - # Find max environment name length - local max_name_len=0 - while IFS= read -r line; do - local name_len - name_len=$(echo "$line" | awk '{print $1}' | sed 's/\*//' | wc -c) - ((name_len > max_name_len)) && max_name_len=$name_len - done <<<"$envs_info" - max_name_len=$((max_name_len + 1)) + envs_info=$(conda info --envs | grep -E -v '^#' ) - # Create formatted output + # Create a temporary file for sorting local temp_file temp_file=$(mktemp) + # Collect max field lengths for consistent formatting + local max_name_len=11 + local max_python_len=0 + while IFS= read -r line; do - local env_name env_path creation_date marker - env_name=$(echo "$line" | awk '{print $1}' | sed 's/\*//') - env_path=$(echo "$line" | awk '{print $NF}' ) - # Get creation date or N/A if path doesn't exist - creation_date="N/A" - [ -d "$env_path" ] && creation_date=$(stat -c "%y" "$env_path" | cut -d"${time_opts}" -f1) - # Remove $HOME from the path - env_path=${env_path/$HOME/\~} - - # Add active marker if environment is current + local env_name env_path modification_date python_version marker + + # Extract environment name and path + env_name=$(echo "$line" | awk '{print $1}') + [ -z "$env_name" ] && continue + + env_path=$(echo "$line" | awk '{print $NF}') # Last column + + # Get modification date + modification_date="N/A" + [ -d "$env_path" ] && modification_date=$(stat -c "%y" "$env_path" | cut -d"${time_opts}" -f1) + + # Get Python version if available + python_version="Unknown" + if [ -d "$env_path" ]; then + python_version=$("$env_path"/bin/python --version 2>&1) + python_version=${python_version#* } + python_version=${python_version%% *} + fi + + # Track max lengths for formatting + (( ${#env_name} > max_name_len )) && max_name_len=${#env_name} + (( ${#python_version} > max_python_len )) && max_python_len=${#python_version} + + # Replace $HOME at the beginning with `~` + env_path=${env_path/#$HOME/\~} + + # Check if active environment marker=" " [[ "$line" == *\** ]] && marker="*" - printf "%s %-${max_name_len}s %s %s\n" \ - "$creation_date" "$env_name" "$marker" "$env_path" >> "$temp_file" + # Store formatted data in temp file for sorting + echo "$modification_date|$python_version|$env_name|$marker|$env_path" >> "$temp_file" + done <<< "$envs_info" - done <<<"$envs_info" + # Adjust column spacing + max_name_len=$(( max_name_len + 2 )) + max_python_len=$(( max_python_len + 1 )) - # Display sorted or unsorted output - if $sort_by_time; then - sort -n $sort_opts "$temp_file" - else - sort -k${sort_key} $sort_opts "$temp_file" - fi + # Print header + printf "%-${date_spacing}s %-${max_python_len}s %-${max_name_len}s %s %s\n" \ + "Date" "Python" "Environment" " " "Path" + + # Sort and format output after sorting + { + if $sort_by_time; then + sort -t '|' -n $sort_opts "$temp_file" + else + sort -t '|' -k ${sort_key} $sort_opts "$temp_file" + fi + } | while IFS='|' read -r mod_date py_ver env_name mark env_path; do + printf "%-${date_spacing}s %-${max_python_len}s %-${max_name_len}s %s %s\n" \ + "$mod_date" "$py_ver" "$env_name" "$mark" "$env_path" + done rm "$temp_file" + + if [ "${set_here:-}" == 'y' ]; then + set +x + fi } # # Function: lastenv @@ -508,17 +598,20 @@ lastenv() { # ## Description # - **Purpose**: # - Creates a new base conda virtual environment and activates it. +# - If no packages specified, creates environment with latest Python and pip. # - **Usage**: -# - `benv [-h] ENV_NAME [EXTRA_OPTIONS]` +# - `benv [-h] ENV_NAME [PACKAGES...] [OPTIONS...] # - **Options**: # - `-h` Show this help message +# - `-x` Enable debug mode # - **Input Parameters**: # - `ENV_NAME` (string) - The name of the new environment to create. -# - `EXTRA_OPTIONS` (string, optional) - Additional options to pass to `conda create`. +# - `PACKAGES` (string, optional) - Packages to install. Defaults to latest Python and pip. +# - `OPTIONS` (string, optional) - Additional options to pass to `conda create`. # - ** Examples**: -# - `benv pa1` -# - `benv pa1 -c conda-forge` -# - `benv pa1 python=3.11` +# - `benv pa1` - Creates env with latest Python and pip +# - `benv pa1 python=3.11 numpy pandas` - Creates env with specific packages +# - `benv pa1 -c conda-forge python=3.11` - Uses conda-forge channel # - **Output**: # - Creates and activates the new environment. # - **Exceptions**: @@ -527,19 +620,33 @@ lastenv() { benv() { local OPTIND=1 # Parse options - while getopts "h" opt; do + while getopts "hx" opt; do case $opt in h) vhelp "${FUNCNAME[0]}"; return 0 ;; + x) set -x; local set_here="y" ;; \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; return 1 ;; esac done shift $((OPTIND - 1)) + # Check for environment name + if [ -z "$1" ]; then + echo "Error: Environment name is required" >&2 + vhelp "${FUNCNAME[0]}" + return 1 + fi + local env_name="$1"; shift - local extra_options="$*" + local packages="$*" + + # If no packages specified, use latest Python and pip + if [ -z "$packages" ]; then + echo "Warning: No packages specified. Creating environment with latest Python." >&2 + packages="python" + fi - echo "Creating base virtual environment ${env_name} ${extra_options}" - conda create -n "${env_name}" "${extra_options}" -y || { + echo "Creating base virtual environment ${env_name} with packages: ${packages}" + conda create -n "${env_name}" ${packages} -y || { echo "Error: Failed to create environment ${env_name}" >&2 __rc__=1 return "${__rc__}" @@ -547,6 +654,9 @@ benv() { echo "Base environment created - activating ${env_name}" cact "${env_name}" + if [ "${set_here:-}" == 'y' ]; then + set +x + fi } # # Function: nenv @@ -559,6 +669,7 @@ benv() { # - `nenv [-h] PREFIX [EXTRA_OPTIONS]` # - **Options**: # - `-h` Show this help message +# - `-x` Enable debug mode # - **Input Parameters**: # - `PREFIX` (string) - The prefix to identify the series of environments. # - `EXTRA_OPTIONS` (string, optional) - Additional options to pass to the environment creation. @@ -570,9 +681,10 @@ benv() { nenv() { local OPTIND=1 # Parse options - while getopts "h" opt; do + while getopts "hx" opt; do case $opt in h) vhelp "${FUNCNAME[0]}"; return 0 ;; + x) set -x; local set_here="y" ;; \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; return 1 ;; esac done @@ -593,6 +705,9 @@ nenv() { __VENV_PREFIX="${prefix}" # Create a clone of the base environment ccln "base" + if [ "${set_here:-}" == 'y' ]; then + set +x + fi } # # Function: denv @@ -605,6 +720,7 @@ nenv() { # - `denv [-h] [env_name]` # - **Options**: # - `-h` Show this help message +# - `-x` Enable debug mode # - **Input Parameters**: # - `env_name` (string) - The name of the environment to delete. # - **Output**: @@ -615,9 +731,10 @@ nenv() { denv() { local OPTIND=1 # Parse options - while getopts "h" opt; do + while getopts "hx" opt; do case $opt in h) vhelp "${FUNCNAME[0]}"; return 0 ;; + x) set -x; local set_here="y" ;; \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; return 1 ;; esac done @@ -633,6 +750,9 @@ denv() { echo "Removing environment -> ${env_to_delete}" conda remove --all -n "${env_to_delete}" -y + if [ "${set_here:-}" == 'y' ]; then + set +x + fi } # # Function: renv @@ -645,6 +765,7 @@ denv() { # - `renv [-h]` # - **Options**: # - `-h` Show this help message +# - `-x` Enable debug mode # - **Input Parameters**: # - None # - **Output**: @@ -655,9 +776,10 @@ denv() { renv() { local OPTIND=1 # Parse options - while getopts "h" opt; do + while getopts "hx" opt; do case $opt in h) vhelp "${FUNCNAME[0]}"; return 0 ;; + x) set -x; local set_here="y" ;; \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; return 1 ;; esac done @@ -680,9 +802,12 @@ renv() { dact # Deactivate the current environment denv "${env_to_delete}" # Delete the environment cact "${previous_env}" # Reactivate the previous environment + if [ "${set_here:-}" == 'y' ]; then + set +x + fi } -# # Function: clan +# # Function: ccln # `ccln` - Clone current Virtual Environment # # ## Description @@ -695,6 +820,7 @@ renv() { # - `ccln [-h] [new_env_name]` # - **Options**: # - `-h` Show this help message +# - `-x` Enable debug mode # - **Input Parameters**: # - `new_env_name` (string) - The name of the new cloned environment. # - **Output**: @@ -705,9 +831,10 @@ renv() { ccln() { local OPTIND=1 # Parse options - while getopts "h" opt; do + while getopts "hx" opt; do case $opt in h) vhelp "${FUNCNAME[0]}"; return 0 ;; + x) set -x; local set_here="y" ;; \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; return 1 ;; esac done @@ -746,6 +873,104 @@ ccln() { # Switch to the newly created VENV cact "${new_name}" + + if [ "${set_here:-}" == 'y' ]; then + set +x + fi +} + +# # Function: vren +# `vren` - Rename a Virtual Environment. +# +# ## Description +# - **Purpose**: +# - Renames a virtual environment and makes that environment current. +# - **Usage**: +# - `vren [-h] [new_name]` +# - **Options**: +# - `-h` Show this help message +# - `-x` Enable debug mode +# - **Input Parameters**: +# - `old_name` (optional string) - The current name of the environment or +# the current environment if not specified. +# - `new_name` (string) - The new name of the environment. +# - **Output**: +# - Renames the specified environment. +# - **Exceptions**: +# - Errors if the environment does not exist. +# +vren() { + local OPTIND=1 + # Parse options + while getopts "hx" opt; do + case $opt in + h) vhelp "${FUNCNAME[0]}"; return 0 ;; + x) set -x; local set_here="y" ;; + \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; return 1 ;; + esac + done + shift $((OPTIND - 1)) + + local old_name new_name + + # Handle argument cases + if [ $# -eq 0 ]; then + echo "Error: A new name is required to rename the environment." >&2 + return 22 # EINVAL + elif [ $# -eq 1 ]; then + old_name="${CONDA_DEFAULT_ENV}" + new_name="$1" + else + old_name="$1" + new_name="$2" + fi + + # Check if old environment exists + if ! conda env list | grep -q "^${old_name} "; then + echo "Error: Environment '${old_name}' does not exist." >&2 + __rc__=2 # ENOENT + return ${__rc__} + fi + + # Check if trying to rename to same name + if [ "${new_name}" = "${old_name}" ]; then + echo "Error: Cannot rename environment to itself." >&2 + __rc__=1 # EPERM + return ${__rc__} + fi + + # Check if new name already exists + if conda env list | grep -q "^${new_name} "; then + echo "Error: Environment '${new_name}' already exists." >&2 + __rc__=17 # EEXIST + return ${__rc__} + fi + + # If renaming current environment, deactivate it first + if [ "${old_name}" = "${CONDA_DEFAULT_ENV}" ]; then + if ! dact; then + __rc__=$? + return ${__rc__} + fi + fi + + # Perform the rename + echo "Renaming environment '${old_name}' to '${new_name}'..." + conda rename -n "${old_name}" "${new_name}" || { + __rc__=$? + echo "Error: Failed to rename environment." >&2 + return ${__rc__} + } + + # Activate the renamed environment + cact "${new_name}" + + __rc__=$? + + if [ "${set_here:-}" == 'y' ]; then + set +x + fi + return ${__rc__} } # # Function: venvdiff @@ -758,6 +983,7 @@ ccln() { # - `venvdiff [-h] [env1] [env2]` # - **Options**: # - `-h` Show this help message +# - `-x` Enable debug mode # - **Input Parameters**: # - `env1` (string) - The first environment to compare. # - `env2` (string) - The second environment to compare. @@ -766,12 +992,13 @@ ccln() { # - **Exceptions**: # - Errors if either environment does not exist. # -venvdiff() { +vdiff() { local OPTIND=1 # Parse options - while getopts "h" opt; do + while getopts "hx" opt; do case $opt in h) vhelp "${FUNCNAME[0]}"; return 0 ;; + x) set -x; local set_here="y" ;; \?) echo "Invalid option: -$OPTARG" >&2; vhelp "${FUNCNAME[0]}"; return 1 ;; esac done @@ -800,52 +1027,12 @@ venvdiff() { echo "Comparing packages in $env1 and $env2:" diff -y <(echo "$env1_packages") <(echo "$env2_packages") -} - -## Initialization -__VENV_SOURCED_LIST=${__VENV_SOURCED_LIST:-""} -# Capture the fully qualified path of the sourced script -[ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" -# Don't source this script if it's already been sourced. -# The RHS has to be in "" to match the array. -# shellcheck disable=SC2076 -[[ "${__VENV_SOURCED_LIST}" =~ "${THIS_SCRIPT}" ]] && return || __VENV_SOURCED_LIST="${__VENV_SOURCED_LIST} ${THIS_SCRIPT}" + if [ "${set_here:-}" == 'y' ]; then + set +x + fi +} -# Extract script name, directory, and arguments -__VENV_BIN=$(dirname "$(dirname "${THIS_SCRIPT}")") -__VENV_BASE=$(dirname "${__VENV_BIN}") -__VENV_ARGS=$* -__VENV_INCLUDE="${__VENV_BASE}/bin/shinclude" - -# Ensure util_funcs.sh is sourced for utility functions -if declare -f "errno" >/dev/null 2>&1; then - source_util_script "errno" - log_message "INFO" "Sourced errno.sh" -else - # shellcheck source=/dev/null - source "${__VENV_INCLUDE}/errno.sh" - log_message "INFO" "Sourced ${__VENV_INCLUDE}/errno.sh" -fi - -# shellcheck disable=SC2206 -__VENV_INTERNAL_FUNCTIONS=( - ${__VENV_INTERNAL_FUNCTIONS[@]} - "push_venv" - "pop_venv" - "__set_venv_vars" -) - -# Initialize the stack -__VENV_STACK=() -# Stack value -declare -g __sv__ - -# This is so we can pass a return code up through sub-shells since set values are lost in sub-shells. -# This to set the return code and exit the function passing the to return or exit. echo would be -# the last command in the function and we would get the return code of the function. -# -# __rc__ is internal and is in a our function shell includes. -# It would be nice to come up with a fairly "automatic" way to do this. __rc__=0 +return ${__rc__} \ No newline at end of file diff --git a/bin/shinclude/venvutil_lib.sh b/bin/shinclude/venvutil_lib.sh new file mode 100755 index 00000000..71cda07e --- /dev/null +++ b/bin/shinclude/venvutil_lib.sh @@ -0,0 +1,95 @@ +#!/usr/bin/env bash +# # Script: venvutil_lib.sh +# `venvutil_lib.sh` - Primary Entry Point for VenvUtil Library System +# +# ## Description +# - **Purpose**: +# - Primary entry point and initialization for the VenvUtil library system +# - Manages library loading sequence and dependencies +# - Provides core environment management functionality +# +# ## Usage +# - Direct source: `source /path/to/venvutil_lib.sh` +# - Via helper: `source_lib venvutil_lib` +# +# ## Library Loading Sequence +# 1. config_lib - Configuration management +# 2. errno_lib - Error handling and codes +# 3. helpsys_lib - Help system functionality +# 4. string_lib - String manipulation and display +# 5. type_lib - Type checking and validation +# 6. util_lib - Utility functions +# 7. venv_lib - Virtual environment management +# 8. wrapper_lib - Command wrapping and logging +# +# ## Dependencies +# - Bash 4.0 or higher +# - Core library files in same directory +# - Python package managers (pip/conda) for some functionality +# +# ## Environment Variables +# - `__VENV_SOURCED` - Tracks loaded libraries +# - `__VENV_BASE` - Base directory for VenvUtil +# - `__VENV_BIN` - Binary directory location +# - `__VENV_INCLUDE` - Library include directory +# +# ## Debug Support +# - Set `DEBUG_VENVUTIL=ON` for debug output +# - Individual functions support -x flag for debug mode +# +# ## Return Codes +# - 0: Success +# - Non-zero: Various error conditions (see errno_lib.sh) +# +# ## Examples +# ```bash +# # Direct usage +# source /path/to/venvutil_lib.sh +# +# # Enable debug mode +# DEBUG_VENVUTIL=ON source /path/to/venvutil_lib.sh +# ``` +# +# ## Notes +# - This is the primary entry point for the VenvUtil system +# - All other libraries should be loaded through this file +# - Direct sourcing of other libraries is discouraged + +## Initialization +[ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" +if ! declare -p __VENV_SOURCED >/dev/null 2>&1; then declare -g -A __VENV_SOURCED; fi +if [[ "${__VENV_SOURCED[${THIS_SCRIPT}]:-}" == 1 ]]; then + # echo "************************* SKIPPED SKIPPED SKIPPED SKIPPED ************************* -----> $(basename "${THIS_SCRIPT}")" >&2 + return +fi +__VENV_SOURCED["${THIS_SCRIPT}"]=1 + +# shellcheck disable=SC2034 +MY_NAME=$(basename "${THIS_SCRIPT}") +__VENV_BIN=$(dirname "$(dirname "${THIS_SCRIPT}")") +__VENV_BASE=$(dirname "${__VENV_BIN}") +__VENV_INCLUDE="${__VENV_BASE}/bin/shinclude" + +# Get the init_lib.sh script +# shellcheck source=/dev/null +source "${__VENV_INCLUDE}/init_lib.sh" + +# Add internal functions to the __VENV_INTERNAL_FUNCTIONS array. +if ! declare -p __VENV_INTERNAL_FUNCTIONS >/dev/null 2>&1; then declare -ga __VENV_INTERNAL_FUNCTIONS; fi +# Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a. +# shellcheck disable=SC2206 +__VENV_INTERNAL_FUNCTIONS=( + ${__VENV_INTERNAL_FUNCTIONS[@]} +) + +source_lib config_lib +source_lib errno_lib +source_lib helpsys_lib +source_lib string_lib +source_lib type_lib +source_lib util_lib +source_lib venv_lib +source_lib wrapper_lib + +__rc__=0 +return ${__rc__} diff --git a/bin/shinclude/wrapper_funcs.sh b/bin/shinclude/wrapper_lib.sh similarity index 68% rename from bin/shinclude/wrapper_funcs.sh rename to bin/shinclude/wrapper_lib.sh index 8fded99a..36712dca 100755 --- a/bin/shinclude/wrapper_funcs.sh +++ b/bin/shinclude/wrapper_lib.sh @@ -1,32 +1,49 @@ #!/usr/bin/env bash - -# # Script: wrapper_funcs.sh -# `wrapper_funcs.sh` - Python Package Manager Wrapper Functions +# # Script: wrapper_lib.sh +# `wrapper_lib.sh` - Python Package Manager Wrapper Functions # ## Description -# - **Purpose**: +# - **Purpose**: # - Provides enhanced functionality for managing Python package commands by wrapping pip and conda. # - Intercepts and logs changes to virtual environments for rollback, auditing, and future use in venvdiff or vdiff. -# - **Usage**: +# ## Usage # - Source this script in your command line environment to import the wrapper functions. -# - For example, in another script: `source wrapper_funcs.sh`. -# - **Features**: +# - For example, in another script: `source wrapper_lib.sh`. +# ## Features # - Saves a `pip freeze` before any potentially destructive changes to a virtual environment. # - Logs the complete command line to a log file for both conda and pip. # - Persists logs in the `$HOME/.venvutil` directory, even after virtual environments are deleted. -# - **Dependencies**: +# ## Dependencies # - Requires Bash and the Python package managers pip and conda. -# - **Exceptions**: +# ## Exceptions # - Some functions may return specific error codes or print error messages to STDERR. # - Refer to individual function documentation for details. -# Determine the real path of the script -THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") -# Don't source this script if it's already been sourced. -# shellcheck disable=SC2076 -[[ "${__VENV_SOURCED_LIST}" =~ "${THIS_SCRIPT}" ]] && return || __VENV_SOURCED_LIST="${__VENV_SOURCED_LIST} ${THIS_SCRIPT}" -echo "Sourcing: ${THIS_SCRIPT}" +## Initialization +[ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" +if ! declare -p __VENV_SOURCED >/dev/null 2>&1; then declare -g -A __VENV_SOURCED; fi +if [[ "${__VENV_SOURCED[${THIS_SCRIPT}]:-}" == 1 ]]; then + # echo "************************* SKIPPED SKIPPED SKIPPED SKIPPED ************************* -----> $(basename "${THIS_SCRIPT}")" >&2 + return +fi +__VENV_SOURCED["${THIS_SCRIPT}"]=1 +# echo "************************* SOURCED SOURCED SOURCED SOURCED ************************* -----> $(basename "${THIS_SCRIPT}")" >&2 + +# shellcheck disable=SC2034 +MY_NAME=$(basename "${THIS_SCRIPT}") +__VENV_BIN=$(dirname "$(dirname "${THIS_SCRIPT}")") +__VENV_BASE=$(dirname "${__VENV_BIN}") +__VENV_INCLUDE="${__VENV_BASE}/bin/shinclude" + +# Get the init_lib.sh script +# shellcheck source=/dev/null +# source "${__VENV_INCLUDE}/init_lib.sh" -# Define an array of internal functions to exclude from help and documentation +# Get the errno_lib.sh script +# source_lib errno_lib + +# Add internal functions to the __VENV_INTERNAL_FUNCTIONS array. +if ! declare -p __VENV_INTERNAL_FUNCTIONS >/dev/null 2>&1; then declare -ga __VENV_INTERNAL_FUNCTIONS; fi +# Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a. # shellcheck disable=SC2206 __VENV_INTERNAL_FUNCTIONS=( ${__VENV_INTERNAL_FUNCTIONS[@]} @@ -40,44 +57,44 @@ __VENV_INTERNAL_FUNCTIONS=( # `get_function_hash` - Get the hash of a function's definition. # # ## Description -# - **Purpose**: +# - **Purpose**: # - Computes the hash of a function's definition for integrity checks. -# - **Usage**: +# - **Usage**: # - `get_function_hash [function_name]` -# - **Input Parameters**: +# - **Input Parameters**: # - `function_name` (string) - The name of the function to hash. -# - **Output**: +# - **Output**: # - The hash of the function's definition. -# - **Exceptions**: +# - **Exceptions**: # - None # get_function_hash() { - declare -f "$1" | md5 | cut -d' ' -f1 + declare -f "$1" | md5sum | cut -d' ' -f1 } # Define the location of the venvutil config directory export VENVUTIL_CONFIG="${VENVUTIL_CONFIG:-${HOME}/.venvutil}" # Create the directory recursively for the frozen VENV's for recovery. -[[ -d ${VENVUTIL_CONFIG}/freeze ]] || mkdir -p "${VENVUTIL_CONFIG}/freeze" -[[ -d ${VENVUTIL_CONFIG}/log ]] || mkdir -p "${VENVUTIL_CONFIG}/log" +mkdir -p "${VENVUTIL_CONFIG}/freeze" "${VENVUTIL_CONFIG}/log" # # Function: do_wrapper # `do_wrapper` - General wrapper function for logging specific command actions. # # ## Description -# - **Purpose**: +# - **Purpose**: # - Executes a Python package manager command with optional logging based on the specified action. -# - **Usage**: +# - **Usage**: # - `do_wrapper ` -# - **Input Parameters**: +# - **Input Parameters**: # - `cmd` (string) - The command to be executed. # - `additional parameters` (string) - Any additional parameters to be passed to the command. -# - **Output**: +# - **Output**: # - None -# - **Exceptions**: +# - **Exceptions**: # - None # do_wrapper() { + __rc__=0 local cmd="$1"; shift local action="$1" local actions_to_log=("install" "uninstall" "remove" @@ -88,8 +105,12 @@ do_wrapper() { local env_vars env_vars=$( env | sed -E '/^SHELL=/,$d' | sed -E 's/^([A-Za-z_]+)=(.*)$/\1="\2"/' | tr '\n' ' ' ) - # Put the function back to "conda" our PROMPT_COMMAND will put it back after the command - eval "conda() $(declare -f __venv_conda| sed '1d')" + # Put the function back to "conda" we will set it back at the end of this function. + local function_line + function_line=$(declare -f conda | sed '1d') + if [[ -n "${function_line}" ]]; then + eval "conda() ${function_line}" + fi # Make the command be how the user invoked it rather than with the wrappers. local user_cmd @@ -161,6 +182,9 @@ do_wrapper() { __rc__="$?" fi + # In case we are running in a script and not on the command line + __venv_conda_check + return "${__rc__}" } @@ -168,16 +192,16 @@ do_wrapper() { # `pip` - Wrapper function for pip commands. # # ## Description -# - **Purpose**: +# - **Purpose**: # - Wraps pip commands to ensure environment variables are preserved. provides logging # for pip commands and the virtual environment affected -# - **Usage**: +# - **Usage**: # - `pip [arguments]` -# - **Input Parameters**: +# - **Input Parameters**: # - `arguments` (string) - Arguments to pass to pip. -# - **Output**: +# - **Output**: # - Executes the pip command with the provided arguments. -# - **Exceptions**: +# - **Exceptions**: # - None # pip() { @@ -188,40 +212,68 @@ pip() { # `__venv_conda_check` - Ensure conda function is wrapped and check for definition changes. # # ## Description -# - **Purpose**: +# - **Purpose**: # - Checks if the conda function definition has changed and re-hooks if necessary. Replaces # the conda function with a wrapper that logs the command and environment affected. -# - **Usage**: +# - **Usage**: # - `__venv_conda_check` -# - **Input Parameters**: +# - **Input Parameters**: # - None -# - **Output**: +# - **Output**: # - Ensures the conda function is wrapped correctly. -# - **Exceptions**: +# - **Exceptions**: # - None # __venv_conda_check() { + local current_hash current_hash=$(get_function_hash conda) - if [[ "${current_hash}" != "${__venv_conda_hash}" ]]; then + if [[ "${current_hash}" != "${__venv_conda_hash:-}" ]]; then # Capture the current conda function definition and assign it to __venv_conda - eval "__venv_conda() $(declare -f conda | sed '1d')" - + local line + line=$(declare -f conda | sed '1d') + if [[ -n "${line}" ]]; then + eval "__venv_conda() ${line}" 2>/dev/null + fi # Redefine the conda function to include the wrapper conda() { do_wrapper "__venv_conda" "$@" } - # Set the hash to be the new conda function. __venv_conda_hash=$(get_function_hash conda) fi } +if ! declare -p __venv_conda_hash >/dev/null 2>&1; then declare -g __venv_conda_hash; fi # Run through the conda check function to ensure the conda function is wrapped __venv_conda_check # Initial hash of the Conda function. Must always update with new hash after defining. __venv_conda_hash=$(get_function_hash conda) +# >>> conda initialize >>> +# !! Contents within this block are managed by 'conda init' !! +__conda_setup="$('${HOME}/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)" +# shellcheck disable=SC2181 +if [ $? -eq 0 ]; then + eval "$__conda_setup" +else + if [ -f "${HOME}/miniconda3/etc/profile.d/conda.sh" ]; then + . "${HOME}/miniconda3/etc/profile.d/conda.sh" + else + export PATH="${HOME}/miniconda3/bin:$PATH" + fi +fi +unset __conda_setup +# <<< conda initialize <<< + # Modify the PROMPT_COMMAND to continuously check for function `conda` changes -__venv_prompt_command="${PROMPT_COMMAND}" -PROMPT_COMMAND="__venv_conda_check; ${PROMPT_COMMAND}" +__venv_prompt_command="${PROMPT_COMMAND:-}" +PROMPT_COMMAND="__venv_conda_check; ${PROMPT_COMMAND:-}" + +__rc__=0 + +# echo "************************* EXITING EXITING EXITING EXITING ************************* -----> $(basename "${THIS_SCRIPT}")" >&2 + +__venv_conda_check + +return ${__rc__} \ No newline at end of file diff --git a/bin/tokencount.py b/bin/tokencount.py index 3311b3cc..fd7b88a4 100755 --- a/bin/tokencount.py +++ b/bin/tokencount.py @@ -19,7 +19,7 @@ 2. Count tokens from stdin: echo "Some text" | tokencount.py - + 3. Interactive stdin mode: tokencount.py (Type text, press Ctrl+D/Ctrl+Z when done) @@ -34,9 +34,9 @@ Apache License, Version 2.0 """ -import nltk import sys import argparse +import nltk # nltk.download('punkt') # Download the Punkt tokenizer models diff --git a/bin/torch_torture.py b/bin/torch_torture.py new file mode 100755 index 00000000..eec7466a --- /dev/null +++ b/bin/torch_torture.py @@ -0,0 +1,253 @@ +#!/usr/bin/env python +""" +Extended PyTorch benchmark script measuring time and memory usage +for CPU/MPS/CUDA. Includes advanced tests (conv2d, autograd) that +don't have NumPy equivalents. + +This script runs a series of common matrix operations using PyTorch and measures: +- Execution time for each operation +- Memory usage via psutil for CPU and torch.cuda for GPU +- Supports configurable matrix sizes and iteration counts +- Can output results to file or stdout +- Supports multiple compute devices (CPU/CUDA/MPS) + +Operations tested: +- Matrix multiplication +- Matrix transposition +- Matrix summation +- Fourier transformation +- 2D Convolution +- Autograd forward/backward pass + +Usage: + python torch_torture.py [--size SIZE] [--iterations ITERS] [--device DEVICE] [--output FILE] + +Arguments: + --size SIZE Dimension N for NxN matrices (default: 2000) + --iterations ITERS Number of times to repeat each test (default: 10) + --device DEVICE Device to use: cpu | cuda | mps | auto (default: cpu) + --output FILE Optional file path to write results (default: stdout) + +Output format: + Tab-separated values with columns for package, version, matrix size, + iterations, virtual environment, source, device, test name, runtime (s), + and memory usage (MB). +""" + +import argparse +import os +import time +from typing import Optional + +import torch + +try: + import psutil +except ImportError: + print("psutil is required for CPU memory usage reporting.") + psutil = None + + +def get_process_memory_mb() -> float: + """ + Returns the current resident set size (RSS) of the process in MB. + If psutil isn't installed, returns 0.0. + + Uses psutil to get accurate memory measurements across platforms. + If psutil is not available, returns 0.0. + + Returns: + float: Memory usage in megabytes + """ + if not psutil: + return 0.0 + process = psutil.Process(os.getpid()) + return process.memory_info().rss / (1024 * 1024) + + +def get_gpu_memory_mb(device: torch.device) -> float: + """ + Returns allocated GPU memory in MB for 'device' if it's CUDA. + Otherwise returns 0.0 (no MPS memory reporting in PyTorch). + + Args: + device: PyTorch device to check memory usage for + + Returns: + float: GPU memory usage in megabytes, or 0.0 if not CUDA + """ + if device.type == "cuda": + return torch.cuda.memory_allocated(device=device) / (1024 * 1024) + return 0.0 + + +def format_output( + package: str, + version: str, + size: int, + iterations: int, + venv: str, + test_name: str, + runtime: float, + memory: float, + src: str = "PyPi", + device: str = "CPU", +) -> str: + """ + Formats benchmark results into a tab-separated string matching metrics layout. + + Args: + package: Name of the package being tested + version: Version of the package + size: Matrix dimension used in test + iterations: Number of test iterations + venv: Virtual environment name + test_name: Name of the specific test + runtime: Test execution time in seconds + memory: Memory usage in MB + src: Package source (default: "PyPi") + device: Computing device used (default: "CPU") + + Returns: + str: Tab-separated string containing all metrics + """ + return ( + f"{package:<10}\t{version:<10}\t{size:<8}\t{iterations:<8}\t" + f"{venv:<15}\t{src:<8}\t{device:<8}\t{test_name:<30}\t" + f"{runtime:<8.4f}\t{memory:<8.2f}" + ) + + +def main() -> None: + """ + Main function that runs the PyTorch benchmark suite. + + Parses command line arguments, sets up the test environment and device, + runs the benchmark tests, and outputs results. Each test is run + for the specified number of iterations on matrices/tensors of the given size. + Results include both execution time and memory usage (CPU and/or GPU). + + The function handles device selection (CPU/CUDA/MPS), tensor creation, + and defines the test operations to benchmark. It measures both runtime + and memory usage for each test, formatting and logging the results. + """ + parser = argparse.ArgumentParser(description="Extended PyTorch benchmark script.") + parser.add_argument( + "--size", type=int, default=2000, help="Matrix dimension for certain tests." + ) + parser.add_argument( + "--iterations", type=int, default=10, help="Number of iterations per test." + ) + parser.add_argument( + "--device", + type=str, + default="cpu", + help="Device to use: cpu | cuda | mps | auto (auto picks best available)", + ) + parser.add_argument("--output", type=str, help="Path to output log file.") + args = parser.parse_args() + + out_file: Optional[object] = None + if args.output: + out_file = open(args.output, "w", encoding="utf-8") + + def log(msg: str) -> None: + print(msg) + if out_file: + out_file.write(f"{msg}\n") + + # Print header + header = "Package\tVersion\tSize\tIterations\tVirtual Env\tSrc\tDevice\tTest\tRun Time\tMemory" + log(header) + + # Determine device + dev_str = args.device.lower() + if dev_str == "auto": + if torch.cuda.is_available(): + device = torch.device("cuda") + elif torch.backends.mps.is_available(): + device = torch.device("mps") + else: + device = torch.device("cpu") + else: + device = torch.device(dev_str) + + venv = os.getenv("CONDA_DEFAULT_ENV", "unknown") + size = args.size + iterations = args.iterations + + # Create random tensors + a = torch.rand(size, size, device=device) + b = torch.rand(size, size, device=device) + + def op_matmul(): + return torch.mm(a, b) + + def op_transpose(): + return torch.t(a) + + def op_fft(): + return torch.fft.fft(a) + + def op_sum(): + return torch.sum(a) + + def op_convolution(): + conv = torch.nn.Conv2d( + in_channels=3, out_channels=8, kernel_size=3, padding=1 + ).to(device) + inp = torch.randn(16, 3, 64, 64, device=device) + return conv(inp) + + def op_autograd(): + x = torch.randn(size, size, device=device, requires_grad=True) + w = torch.randn(size, size, device=device, requires_grad=True) + y = (x @ w).sum() + y.backward() + x.grad.zero_() + w.grad.zero_() + return y + + tests = [ + ("Matrix multiplication", op_matmul), + ("Matrix transposition", op_transpose), + ("Matrix Summation", op_sum), + ("Fourier transformation", op_fft), + ("Convolution test (Conv2D)", op_convolution), + ("Autograd test (forward/back)", op_autograd), + ] + + for test_name, func in tests: + if device.type == "cuda": + torch.cuda.reset_peak_memory_stats(device=device) + + start_t = time.time() + for _ in range(iterations): + _ = func() + elapsed = time.time() - start_t + + # Get memory usage + mem_mb = get_process_memory_mb() + if device.type == "cuda": + mem_mb += get_gpu_memory_mb(device) + + # Format and log output + output = format_output( + package="PyTorch", + version=torch.__version__, + size=size, + iterations=iterations, + venv=venv, + test_name=test_name, + runtime=elapsed, + memory=mem_mb, + device=device.type.upper(), + ) + log(output) + + if out_file: + out_file.close() + + +if __name__ == "__main__": + main() diff --git a/docs/Documentation_Generation.md b/docs/Documentation_Generation.md index 058cdbf2..b321bb08 100644 --- a/docs/Documentation_Generation.md +++ b/docs/Documentation_Generation.md @@ -5,7 +5,7 @@ Script documentation begins at the first `#` after the `#!` until the first blan - example ```bash -#1/usr/bin/env bash +#!/usr/bin/env bash # Script: help_sys.sh # `help_sys.sh` - Help System Functions for Bash Scripts # ## Description diff --git a/docs/Metrics_Layout.md b/docs/Metrics_Layout.md new file mode 100644 index 00000000..6fa1320c --- /dev/null +++ b/docs/Metrics_Layout.md @@ -0,0 +1,18 @@ +Package Version Size Iterations Virtual Env Src Device Test Start Time End Time Run Time Memory +NumPy 1.26.4 2000 20 oobadev-macOS PyPi Default Matrix multiplication 2025-01-24 17:46:27 2025-01-24 17:46:31 3.5581 136.62 +NumPy 1.26.4 2000 20 oobadev-macOS PyPi Default Matrix transposition 2025-01-24 17:46:27 2025-01-24 17:46:31 3.5581 136.62 +NumPy 1.26.4 2000 20 oobadev-macOS PyPi Default Eigenvalue computation 2025-01-24 17:46:27 2025-01-24 17:46:31 3.5581 136.62 +NumPy 1.26.4 2000 20 oobadev-macOS PyPi Default Fourier transformation 2025-01-24 17:46:27 2025-01-24 17:46:31 3.5581 136.62 +NumPy 1.26.4 2000 20 oobadev-macOS PyPi Default Matrix Summation 2025-01-24 17:46:27 2025-01-24 17:46:31 3.5581 136.62 +PyTorch 2.5.1 2000 20 oobadev-macOS PyPi CPU Matrix multiplication 2025-01-24 17:52:36 2025-01-24 17:52:36 0.1465 243.53 +PyTorch 2.5.1 2000 20 oobadev-macOS PyPi CPU Matrix transposition 2025-01-24 17:52:36 2025-01-24 17:52:36 0.1465 243.53 +PyTorch 2.5.1 2000 20 oobadev-macOS PyPi CPU Matrix Summation 2025-01-24 17:52:36 2025-01-24 17:52:36 0.1465 243.53 +PyTorch 2.5.1 2000 20 oobadev-macOS PyPi CPU Fourier transformation 2025-01-24 17:52:36 2025-01-24 17:52:36 0.1465 243.53 +PyTorch 2.5.1 2000 20 oobadev-macOS PyPi CPU Convolution test (Conv2D) 2025-01-24 17:52:36 2025-01-24 17:52:36 0.1465 243.53 +PyTorch 2.5.1 2000 20 oobadev-macOS PyPi CPU Autograd test (forward/back) 2025-01-24 17:52:36 2025-01-24 17:52:36 0.1465 243.53 +PyTorch 2.5.1 2000 20 oobadev-macOS PyPi GPU Matrix multiplication 2025-01-24 17:52:36 2025-01-24 17:52:36 0.1465 243.53 +PyTorch 2.5.1 2000 20 oobadev-macOS PyPi GPU Matrix transposition 2025-01-24 17:52:36 2025-01-24 17:52:36 0.1465 243.53 +PyTorch 2.5.1 2000 20 oobadev-macOS PyPi GPU Matrix Summation 2025-01-24 17:52:36 2025-01-24 17:52:36 0.1465 243.53 +PyTorch 2.5.1 2000 20 oobadev-macOS PyPi GPU Fourier transformation 2025-01-24 17:52:36 2025-01-24 17:52:36 0.1465 243.53 +PyTorch 2.5.1 2000 20 oobadev-macOS PyPi GPU Convolution test (Conv2D) 2025-01-24 17:52:36 2025-01-24 17:52:36 0.1465 243.53 +PyTorch 2.5.1 2000 20 oobadev-macOS PyPi GPU Autograd test (forward/back) 2025-01-24 17:52:36 2025-01-24 17:52:36 0.1465 243.53 \ No newline at end of file diff --git a/docs/Standards.md b/docs/Standards.md index 19709477..697d6e1b 100644 --- a/docs/Standards.md +++ b/docs/Standards.md @@ -16,20 +16,23 @@ A template for [Script Documentation Style](/doc/Script_Doc_Templ.md) is found h - **Initialization**: All scripts, even included ones in the project should include the following lines at the beginning for proper operation. ```bash - # Determine the real path of the script + ## Initialization [ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" - # Don't source this script if it's already been sourced. - [[ "${__VENV_SOURCED_LIST}" =~ "${THIS_SCRIPT}" ]] && return || __VENV_SOURCED_LIST="${__VENV_SOURCED_LIST} ${THIS_SCRIPT}" - echo "Sourcing:${THIS_SCRIPT}" - - # Extract script name, directory, and arguments - MY_NAME=$(basename ${THIS_SCRIPT}) - MY_BIN=$(dirname ${THIS_SCRIPT}) - MY_ARGS=$* - MY_INCLUDE="${MY_BIN}/shinclude" - - # Define an array of internal functions to exclude from help and documentation - __VENV_INTERNAL_FUNCTIONS=("init_help_system" "functions to esclude from help documentation" ) + # shellcheck disable=SC2034 + MY_NAME=$(basename "${THIS_SCRIPT}") + __VENV_BIN=$(dirname "$(dirname "${THIS_SCRIPT}")") + __VENV_BASE=$(dirname "${__VENV_BIN}") + __VENV_INCLUDE="${__VENV_BASE}/bin/shinclude" + + # Get the init_lib.sh script + # shellcheck source=/dev/null + source "${__VENV_INCLUDE}/init_lib.sh" + + # Get the errno_lib.sh script + source_lib errno_lib + + __rc__=0 + return ${__rc__} ``` ### Function-Level Documentation diff --git a/docs/chat_tools.md b/docs/chat_tools.md new file mode 100644 index 00000000..134ec02b --- /dev/null +++ b/docs/chat_tools.md @@ -0,0 +1,82 @@ +# ChatGPT Log Management Tools + +## Overview +The chat log management tools work in conjunction with the [ChatGPT Chat Log Export Safari Extension](https://github.com/unixwzrd/chatgpt-chatlog-export) to provide a complete solution for exporting, processing, and managing ChatGPT conversations. + +## Tools + +### extract-chat +A Python script for converting ChatGPT JSON chat logs to readable Markdown or HTML format. + +#### Features +- Supports both Markdown and HTML output formats +- Preserves conversation structure and formatting +- Handles system messages, user messages, and tool outputs +- Maintains timestamps and metadata +- Supports batch processing of multiple files + +#### Usage +```bash +extract-chat [-h] [-o OUTPUT_DIR] [--format {markdown,html}] patterns [patterns ...] +``` + +The chat markdown or html files will be placed in the specified directory and use the naming convention START_TIME_END_TIME_Chat_name.[html|md]. + +#### Parameters +- `patterns`: One or more JSON files or directories to process +- `-o, --output-dir`: Output directory for converted files +- `--format`: Output format (markdown or html, default: markdown) + +#### Example Usage +```bash +# Convert a single file +extract-chat conversation.json + +# Convert all JSON files in a directory to HTML +extract-chat ./chats/*.json --format html --output-dir ./converted/ + +# Process specific files +extract-chat chat1.json chat2.json --format markdown +``` + +### rename-chat +Utilities for managing and organizing chat log files. + +#### Features +- Rename chat files based on content +- Extract titles and dates +- Organize files into directories +- Handle duplicate files + +The Python (`rename-chat.py`) may also be invoked using `rename-chat`. + +This will rename the chat logs from the OpenAI naming convention to the START_TIME_END_TIME_Chat_name.json format. In the event of collisions, it will increment a counter and prepend it before the .json extension. + +## Integration with Safari Extension + +1. Install the [ChatGPT Chat Log Export Extension](https://github.com/unixwzrd/chatgpt-chatlog-export) +2. Export conversations from ChatGPT using the extension +3. Process the exported JSON files using these tools + +## Output Formats + +### Markdown +- Clean, readable format +- Compatible with most Markdown viewers +- Preserves code blocks and formatting +- Includes metadata and timestamps + +### HTML +- Rich formatting with syntax highlighting +- Dark mode support +- Collapsible sections for tool outputs +- Responsive design for better readability +- Using a web browser they may be searched and printed to PDF. + +## Future Improvements + +Planned enhancements: +- Conversation analytics and statistics +- Tag-based organization +- Integration with other chat platforms +- Enhanced metadata extraction \ No newline at end of file diff --git a/docs/filetree.md b/docs/filetree.md index b2743fa3..cc4705d5 100644 --- a/docs/filetree.md +++ b/docs/filetree.md @@ -146,7 +146,7 @@ All short options have corresponding long options with double dashes (`--`): │ ├── shinclude/ │ │ ├── errno.sh │ │ ├── help_sys.sh - │ │ ├── init_env.sh + │ │ ├── init_lib.sh │ │ ├── util_funcs.sh │ │ ├── venv_funcs.sh │ │ └── wrapper_funcs.sh diff --git a/docs/installer-manifest.md b/docs/installer-manifest.md index d0a67636..1292fb93 100644 --- a/docs/installer-manifest.md +++ b/docs/installer-manifest.md @@ -15,26 +15,26 @@ This described the manifest file for the installer. The types are - Data Structure | field number | name | required/optional | default | description | - | :----------: | :-----------: | :---------------: | :--------------: | :-------------------------------------------------------------------------------: | - | 1 | `type` | required | no default | c, d, f, h, l, - Cancel, directory, file, hard link, or symbolic link | - | 2 | `destination` | required | no default | Destination location of the object | - | 3 | `source` | required | no default | Source location of the object for a link this is the file or directory to link to | - | 4 | `name` | required | no default | Name of the object | - | 5 | `permissions` | required | no default | Permissions of the object | - | 6 | `owner` | optional | `/usr/bin/id -u` | Owner of the object | - | 7 | `group` | optional | `/usr/bin/id -g` | Group of the object | - | 8 | `size` | optional | no default | Size of the object | - | 9 | `checksum` | optional | no default | Checksum of the object | + |------------|:------------|:------------------:|----------------|-------------------------------------------------------------------------------| + | 1 | `type` | required | no default | c, d, f, h, l, - Cancel, directory, file, hard link, or symbolic link | + | 2 | `destination` | required | no default | Destination location of the object | + | 3 | `source` | required | no default | Source location of the object for a link this is the file or directory to link to | + | 4 | `name` | required | no default | Name of the object | + | 5 | `permissions` | required | no default | Permissions of the object | + | 6 | `owner` | optional | `/usr/bin/id -u` | Owner of the object | + | 7 | `group` | optional | `/usr/bin/id -g` | Group of the object | + | 8 | `size` | optional | no default | Size of the object | + | 9 | `checksum` | optional | no default | Checksum of the object | - Layout - | type | destination | source | name | permissions | owner | group | size | checksum | - | :---: | :---------: | :--------: | :--------: | :---------: | :---: | :---: | :---: | :----------: | - | c | bin | asset_name | link_name | 755 | user | staff | 30 | 3a2c0c5d7b5e | - | d | bin | bin | lib_dir | 2755 | user | staff | 30 | | - | f | | | README.md | 644 | user | staff | 30 | 3a2c0c5d7b5e | - | h | bin | asset_name | link_name | 755 | user | staff | 30 | 3a2c0c5d7b5e | - | l | bin | | asset_name | | | | | | + | type | destination | source | name | permissions | owner | group | size | checksum | + |:----:|-------------|-----------|-----------|-------------|--------|-------|------|--------------| + | c | bin | asset_name | link_name | 755 | user | staff | 30 | 3a2c0c5d7b5e | + | d | bin | bin | lib_dir | 2755 | user | staff | 30 | | + | f | | | README.md | 644 | user | staff | 30 | 3a2c0c5d7b5e | + | h | bin | asset_name | link_name | 755 | user | staff | 30 | 3a2c0c5d7b5e | + | l | bin | | asset_name | | | | | | - Example @@ -90,21 +90,41 @@ To add a library to the set of packages pkg-config knows about, simply install a - Example pkg-config file ```conf -# This is a comment -prefix=/home/hp/unst # this defines a variable -exec_prefix=${prefix} # defining another variable in terms of the first -libdir=${exec_prefix}/lib -includedir=${prefix}/include +# Package Configuration File for venvutil -Name: GObject # human-readable name -Description: Object/type system for GLib # human-readable description -Version: 1.3.1 -URL: http://www.gtk.org -Requires: glib-2.0 = 1.3.1 -Conflicts: foobar <= 4.5 -Libs: -L${libdir} -lgobject-1.3 -Libs.private: -lm -Cflags: -I${includedir}/glib-2.0 -I${libdir}/glib/include +# Define variables +prefix=$HOME/local/venvutil +exec_prefix=${prefix} +libdir="${exec_prefix}/lib" +includedir=${prefix}/include +bindir=${exec_prefix}/bin +datadir=${prefix}/share +sysconfdir=${prefix}/etc +include_files=("README.md" "LICENSE" "setup.sh" "setup.cf" "manifest.lst") +include_dirs=("bin" "docs" "conf" "modules" "${TMPDIR}") + +# Package metadata +Name: venvutil +Description: Virtual Environment Utilities +Description: "${TERM} Virtual Environment Utilities" +Version: 0.4.0 +Repository: https://github.com/unixwzrd/venvutil +License: Apache License, Version 2.0 +Support: https://github.com/unixwzrd/venvutil/issues +Contribute: https://patreon.com/unixwzrd +Contribute: https://www.ko-fi.com/unixwzrd + +# Dependencies (if any) +Requires: python >= 3.10 +Requires: bash >= 4.0 +Requires: Conda >= 22.11 +Requires: macOS +Requires: Linux +Conflicts: + +# Compiler and linker flags (if applicable) +# Cflags: -I${includedir} +# Libs: -L${libdir} -lvenvutil ``` You would normally generate the file using configure, so that the prefix, etc. are set to the proper values. The GNU Autoconf manual recommends generating files like .pc files at build time rather than configure time, so when you build the .pc file is a matter of taste and preference. @@ -112,3 +132,9 @@ You would normally generate the file using configure, so that the prefix, etc. a Files have two kinds of line: keyword lines start with a keyword plus a colon, and variable definitions start with an alphanumeric string plus an equals sign. Keywords are defined in advance and have special meaning to pkg-config; variables do not, you can have any variables that you wish (however, users may expect to retrieve the usual directory name variables). - **Note** that variable references are written `"${foo}"`; you can escape literal `"${"` as `"$${"`. +- **Note** Array variables may be written as `include_dir=("bin" "docs" "conf" "modules")` or `include_dir=(${include_dir[@]})` +- **Note** Keywords which are repeated with different values will be merged into a single array using the keyword as the variable name. +- **Note** Key Value consist of the Keyword followed by a `:` and then the value, which may contain any characters. +- **Note** If there is a space between the `:` and the value, it is ignored. +- **Note** The value may or may not have "quotes" around it. +- **Note** There may be variables in any location in the values. diff --git a/docs/migration_guide.md b/docs/migration_guide.md new file mode 100644 index 00000000..efccba67 --- /dev/null +++ b/docs/migration_guide.md @@ -0,0 +1,204 @@ +# Migration Guide - Version 20250206-00_R1 + +This guide helps you migrate your scripts and configurations to the new library structure introduced in version 20250206-00_R1 + +## Major Changes + +### Library Reorganization +All shell library files have been renamed from `.sh` to `_lib.sh` for better organization and clarity: + +| Old Name | New Name | +|----------|----------| +| errno.sh | errno_lib.sh | +| help_sys.sh | helpsys_lib.sh | +| init_env.sh | init_lib.sh | +| util_funcs.sh | util_lib.sh | +| venv_funcs.sh | venv_lib.sh | +| wrapper_funcs.sh | wrapper_lib.sh | + +New libraries added: +- `config_lib.sh`: Configuration management +- `type_lib.sh`: Type checking and validation +- `string_lib.sh`: String manipulation + +### Updating Your Scripts + +1. Update source statements: + ```bash + # Old way + source "${INCLUDE_DIR}/errno.sh" + + # New way + source "${INCLUDE_DIR}/errno_lib.sh" + ``` + +2. Use the new source_lib function: + ```bash + # Recommended way + source_lib errno_lib + ``` + +### Function Changes + +#### Environment Management +- `lenv` now supports headers and Python version display +- `ccln` supports sequence-less cloning +- New `renv` function for environment renaming + +Example usage: +```bash +# List all environments with Python versions +lenv + +# Clone without sequence number +ccln myenv newenv + +# Rename environment +renv oldname newname +``` + +### Configuration Changes + +1. Update your configuration files: + ```bash + # Old format + VENV_INCLUDE="${VENV_BASE}/bin/shinclude" + + # New format + VENV_LIB="${VENV_BASE}/bin/shinclude" + ``` + +## New Features + +### Type Checking +```bash +# Use new type checking functions +var_type "my_var" +``` + +### String Manipulation +```bash +# New string utilities +clean_string "input string" +format_path "/path/to/file" +sanitize_input "user input" +``` + +## Breaking Changes + +1. Direct `.sh` extension references will fail +2. Old environment variable names are deprecated +3. Some function signatures have changed + +### Deprecated Features +- Direct shell script sourcing +- Old environment variable names +- Legacy configuration formats + +## Troubleshooting + +### Common Issues + +1. Script sourcing fails: + ```bash + # Error + source: errno.sh: No such file or directory + + # Fix + source_lib errno_lib + ``` + +2. Function not found: + ```bash + # Error + function_name: command not found + + # Fix + source_lib util_lib # Ensure correct library is sourced + ``` + +3. Configuration errors: + ```bash + # Error + VENV_INCLUDE: variable not found + + # Fix + Update to VENV_LIB in configuration + ``` + +## Best Practices + +1. Use `source_lib` instead of direct sourcing +2. Update all script headers +3. Use new type checking functions +4. Implement error handling +5. Update configuration files + +## Testing Your Migration + +1. Create a test environment: + ```bash + benv test-migration + ``` + +2. Test your scripts: + ```bash + # Run with debug mode + DEBUG=1 your_script.sh + ``` + +3. Verify functionality: + ```bash + # Check environment management + lenv + ccln test-env + vren old-name new-name + ``` + +## Getting Help + +- Use `vhelp` for function documentation +- Check error messages with `errfind` +- Review logs in `~/.venvutil/logs` +- Submit issues on GitHub + +## Future Compatibility + +- Keep scripts updated with `source_lib` +- Use new function signatures +- Follow type checking guidelines +- Maintain configuration standards + +### Library Dependencies and Initialization + +The new library structure introduces two main initialization libraries: + +- `init_lib.sh`: Core initialization and environment setup +- `venvutil_lib.sh`: Main virtual environment utilities that sources all required dependencies + +Using these libraries simplifies dependency management: + +```bash +# Old way - manually sourcing each dependency +source "${INCLUDE_DIR}/init_env.sh" + +# New way - automatic dependency management +source_lib init_lib # Core initialization + + # or this both will have the same effect. + +source_lib venvutil_lib # Sources all venv-related dependencies +``` + +Dependencies are now handled automatically: +- `init_lib.sh` provides core initialization +- `venvutil_lib.sh` sources: + - Type checking functions + - Error handling + - String manipulation + - Configuration management + - Virtual environment functions + - Wrapper utilities + +This ensures consistent initialization and reduces dependency-related errors. +``` \ No newline at end of file diff --git a/docs/performance_testing.md b/docs/performance_testing.md new file mode 100644 index 00000000..34d1374a --- /dev/null +++ b/docs/performance_testing.md @@ -0,0 +1,170 @@ +# Performance Testing Guide + +## Overview +This guide covers the performance testing tools included in the venvutil package. These tools are specifically designed to help you verify and optimize Python package performance on Apple Silicon, particularly focusing on GPU and Neural Engine utilization through the Accelerate framework. + +## Tools + +### torch_torture.py +A PyTorch stress testing tool for evaluating GPU and Neural Engine performance on Apple Silicon. +Helps verify proper MPS (Metal Performance Shaders) backend utilization. + +#### Usage +```bash +torch_torture.py [-h] [-s SIZE] [-i ITERATIONS] [-d DEVICE] [-o OUTPUT] +``` + +#### Parameters +- `-s, --size`: Matrix size for operations (default: 1000) +- `-i, --iterations`: Number of test iterations (default: 10) +- `-d, --device`: Device to use ('cuda', 'mps', or 'cpu') +- `-o, --output`: Output file for results + +#### Tests Performed +- Matrix multiplication +- Convolution operations +- Tensor operations +- Memory transfer speeds +- Training simulation + +### numpy_torture.py +A NumPy stress testing tool for evaluating CPU and Accelerate framework optimizations. +Verifies proper utilization of Apple's Accelerate framework and BLAS optimizations. + +#### Usage +```bash +numpy_torture.py [-h] [-s SIZE] [-i ITERATIONS] [-o OUTPUT] +``` + +#### Parameters +- `-s, --size`: Matrix size for operations (default: 1000) +- `-i, --iterations`: Number of test iterations (default: 10) +- `-o, --output`: Output file for results + +#### Tests Performed +- Matrix multiplication +- Matrix transposition +- Eigenvalue computation +- Fourier transformation +- Matrix summation + +### numpy-comp +A utility for recompiling NumPy with Apple Silicon optimizations. + +#### Usage +```bash +numpy-comp [-h] [-v VERSION] +``` + +This tool ensures NumPy is built with: +- Apple Accelerate framework integration +- BLAS/LAPACK optimizations +- ILP64 support for large arrays +- Platform-specific compiler optimizations + +Example command used internally: +```bash +CFLAGS="-I/System/Library/Frameworks/vecLib.framework/Headers -Wl,-framework -Wl,Accelerate -framework Accelerate" pip install numpy==1.26.* --force-reinstall --no-deps --no-cache --no-binary :all: --no-build-isolation --compile -Csetup-args=-Dblas=accelerate -Csetup-args=-Dlapack=accelerate -Csetup-args=-Duse-ilp64=true +``` + +### compare_test +An experimental framework for performance comparisons. This tool is provided as a starting point for custom testing but is not officially supported. + +#### Usage +```bash +compare_test [-h] [-c CONFIG] [-t TESTS] [-o OUTPUT] +``` + +#### Parameters +- `-c, --config`: Test configuration file +- `-t, --tests`: Specific tests to run +- `-o, --output`: Output file for results + +## Metrics Layout + +The performance metrics are reported in a standardized format with tab separated values as shown here: + +``` +Package Version Size Iterations Virtual Env Src Device Test Start Time End Time Run Time Memory +``` + +### Fields Description +- `Package`: Testing package (NumPy/PyTorch) +- `Version`: Package version +- `Size`: Test matrix size +- `Iterations`: Number of test iterations +- `Virtual Env`: Environment name +- `Src`: Package source +- `Device`: Computing device +- `Test`: Test operation name +- `Start/End Time`: Test timing +- `Run Time`: Operation duration +- `Memory`: Peak memory usage + +## Best Practices + +### Environment Setup +1. Create a clean virtual environment for testing +2. Install only required packages +3. Document all environment variables +4. Use consistent Python versions + +### Test Execution +1. Run tests multiple times +2. Vary matrix sizes +3. Test different devices +4. Compare with baseline results + +### Results Analysis +1. Look for performance regressions +2. Compare across different configurations +3. Document anomalies +4. Track trends over time + +## Troubleshooting + +### Common Issues +1. Memory errors + - Reduce matrix size + - Check available system memory + - Monitor swap usage + +2. Device errors + - Verify device availability + - Check driver versions + - Monitor temperature + +3. Performance inconsistencies + - Check system load + - Monitor thermal throttling + - Verify no background processes + +## Example Workflow + +1. Setup test environment: + ```bash + benv perftest + pip install -r requirements.txt + ``` + +2. Run baseline tests: + ```bash + numpy_torture.py -s 1000 -i 20 -o baseline.csv + ``` + +3. Run comparison tests: + ```bash + compare_test -c test_config.yml -o results.csv + ``` + +4. Analyze results: + ```bash + python analyze_results.py baseline.csv results.csv + ``` + +## Future Improvements + +- Add more test operations +- Support distributed testing +- Enhance reporting formats +- Add visualization tools \ No newline at end of file diff --git a/docs/shdoc/README.md b/docs/shdoc/README.md index bc38d6b1..10338596 100644 --- a/docs/shdoc/README.md +++ b/docs/shdoc/README.md @@ -6,12 +6,16 @@ | Script | Description | |:--|:--| -| [errno.sh](bin/shinclude/errno_sh.md) | Provides POSIX errno codes and utilities for Bash scripts | -| [help_sys.sh](bin/shinclude/help_sys_sh.md) | Help System Functions for Bash Scripts | -| [init_env.sh](bin/shinclude/init_env_sh.md) | Initialize Environment and Source Utility Scripts | -| [util_funcs.sh](bin/shinclude/util_funcs_sh.md) | Utility Functions for Bash Scripts | -| [venv_funcs.sh](bin/shinclude/venv_funcs_sh.md) | Virtual Environment Management Functions for Bash Scripts | -| [wrapper_funcs.sh](bin/shinclude/wrapper_funcs_sh.md) | Python Package Manager Wrapper Functions | +| [config_lib.sh](bin/shinclude/config_lib_sh.md) | Configuration Management and Variable Handling | +| [errno_lib.sh](bin/shinclude/errno_lib_sh.md) | Provides POSIX errno codes and utilities for Bash scripts | +| [helpsys_lib.sh](bin/shinclude/helpsys_lib_sh.md) | Help System Functions for Bash Scripts | +| [init_lib.sh](bin/shinclude/init_lib_sh.md) | Library Initialization and Environment Setup | +| [string_lib.sh](bin/shinclude/string_lib_sh.md) | Consolidated string sanitization & variable expansion library | +| [type_lib.sh](bin/shinclude/type_lib_sh.md) | Support functions for variable type handling. | +| [util_lib.sh](bin/shinclude/util_lib_sh.md) | Utility Functions and Cross-Platform Support | +| [venv_lib.sh](bin/shinclude/venv_lib_sh.md) | Virtual Environment Management Functions for Bash Scripts | +| [venvutil_lib.sh](bin/shinclude/venvutil_lib_sh.md) | Primary Entry Point for VenvUtil Library System | +| [wrapper_lib.sh](bin/shinclude/wrapper_lib_sh.md) | Python Package Manager Wrapper Functions | --- @@ -19,8 +23,8 @@ Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:56 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/config_lib_sh.md b/docs/shdoc/bin/shinclude/config_lib_sh.md new file mode 100644 index 00000000..5c120366 --- /dev/null +++ b/docs/shdoc/bin/shinclude/config_lib_sh.md @@ -0,0 +1,25 @@ +# Functions Defined in Script: config_lib.sh + +### [config_lib.sh](/docs/shdoc/bin/shinclude/scripts/config_lib.sh.md) - Configuration Management and Variable Handling + +## List of Functions Defined + +| Function | Description | +|:--|:--| +| [pkg_config_vars](functions/pkg_config_vars.md) | Sets up the actions for the variables in a package config file. | +| [expand_variable](functions/expand_variable.md) | Expands variables in a given string. | +| [load_pkg_config](functions/load_pkg_config.md) | Loads package configuration from setup.cf file. | +| [write_config](functions/write_config.md) | Write variables in valid Bash syntax (scalar vs array). | +| [parse_manifest_metadata](functions/parse_manifest_metadata.md) | Parses manifest metadata. | + +--- + +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:19 diff --git a/docs/shdoc/bin/shinclude/errno_sh.md b/docs/shdoc/bin/shinclude/errno_lib_sh.md similarity index 50% rename from docs/shdoc/bin/shinclude/errno_sh.md rename to docs/shdoc/bin/shinclude/errno_lib_sh.md index 4b56c7df..f2de2f59 100644 --- a/docs/shdoc/bin/shinclude/errno_sh.md +++ b/docs/shdoc/bin/shinclude/errno_lib_sh.md @@ -1,17 +1,19 @@ -# Functions Defined in Script: errno.sh +# Functions Defined in Script: errno_lib.sh -### [errno.sh](/docs/shdoc/bin/shinclude/scripts/errno.sh.md) - Provides POSIX errno codes and utilities for Bash scripts +### [errno_lib.sh](/docs/shdoc/bin/shinclude/scripts/errno_lib.sh.md) - Provides POSIX errno codes and utilities for Bash scripts ## List of Functions Defined | Function | Description | |:--|:--| +| [set_debug](functions/set_debug.md) | Sets the debug level for the script. | | [errno](functions/errno.md) | Provides POSIX errno codes and values for use in scripts or lookup of error codes on the command line. | | [errfind](functions/errfind.md) | Find the error code for a given string. | -| [errno_warn](functions/errno_warn.md) | Prints a warning using the errno function to STDERR and returns the error number. | -| [errno_exit](functions/errno_exit.md) | Prints an error to STDERR using the errno function and exits with the error number. | +| [errno_warn](functions/errno_warn.md) | Prints a warning message to STDERR and returns the error number. | +| [errno_exit](functions/errno_exit.md) | Prints an error message to STDERR and exits with the error number. | | [errval](functions/errval.md) | Returns the numeric value associated with a log level. | | [log_message](functions/log_message.md) | Prints a message to STDERR based on the provided log level. | +| [_deprecated](functions/_deprecated.md) | Prints a deprecation warning message to STDERR. | --- @@ -19,8 +21,8 @@ Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/__set_venv_vars.md b/docs/shdoc/bin/shinclude/functions/__set_venv_vars.md index 33929653..7ba3c1b7 100644 --- a/docs/shdoc/bin/shinclude/functions/__set_venv_vars.md +++ b/docs/shdoc/bin/shinclude/functions/__set_venv_vars.md @@ -15,13 +15,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/__venv_conda_check.md b/docs/shdoc/bin/shinclude/functions/__venv_conda_check.md index 8052f907..47a1d955 100644 --- a/docs/shdoc/bin/shinclude/functions/__venv_conda_check.md +++ b/docs/shdoc/bin/shinclude/functions/__venv_conda_check.md @@ -2,27 +2,27 @@ # Function: __venv_conda_check `__venv_conda_check` - Ensure conda function is wrapped and check for definition changes. ## Description -- **Purpose**: +- **Purpose**: - Checks if the conda function definition has changed and re-hooks if necessary. Replaces the conda function with a wrapper that logs the command and environment affected. -- **Usage**: +- **Usage**: - `__venv_conda_check` -- **Input Parameters**: +- **Input Parameters**: - None -- **Output**: +- **Output**: - Ensures the conda function is wrapped correctly. -- **Exceptions**: +- **Exceptions**: - None ## Definition -* [wrapper_funcs.sh](../wrapper_funcs_sh.md) +* [wrapper_lib.sh](../wrapper_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:56 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/_deprecated.md b/docs/shdoc/bin/shinclude/functions/_deprecated.md new file mode 100644 index 00000000..85d6dee3 --- /dev/null +++ b/docs/shdoc/bin/shinclude/functions/_deprecated.md @@ -0,0 +1,25 @@ +## _deprecated +# Function _deprecated +`_deprecated` - Prints a deprecation warning message to STDERR. +## Description +- **Purpose**: + - Prints a deprecation warning message to STDERR. +- **Usage**: + - `_deprecated ` +- **Input Parameters**: + - `message`: The message to print. +- **Output**: + - Prints a deprecation warning message to STDERR. + +## Definition + +* [errno_lib.sh](../errno_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/_source_check.md b/docs/shdoc/bin/shinclude/functions/_source_check.md new file mode 100644 index 00000000..89ea2d7e --- /dev/null +++ b/docs/shdoc/bin/shinclude/functions/_source_check.md @@ -0,0 +1,14 @@ +## _source_check + +## Definition + +* [init_lib.sh](../init_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/benv.md b/docs/shdoc/bin/shinclude/functions/benv.md index 8bca15f3..d15361e8 100644 --- a/docs/shdoc/bin/shinclude/functions/benv.md +++ b/docs/shdoc/bin/shinclude/functions/benv.md @@ -4,17 +4,20 @@ ## Description - **Purpose**: - Creates a new base conda virtual environment and activates it. + - If no packages specified, creates environment with latest Python and pip. - **Usage**: - - `benv [-h] ENV_NAME [EXTRA_OPTIONS]` + - `benv [-h] ENV_NAME [PACKAGES...] [OPTIONS...] - **Options**: - `-h` Show this help message + - `-x` Enable debug mode - **Input Parameters**: - `ENV_NAME` (string) - The name of the new environment to create. - - `EXTRA_OPTIONS` (string, optional) - Additional options to pass to `conda create`. + - `PACKAGES` (string, optional) - Packages to install. Defaults to latest Python and pip. + - `OPTIONS` (string, optional) - Additional options to pass to `conda create`. - ** Examples**: - - `benv pa1` - - `benv pa1 -c conda-forge` - - `benv pa1 python=3.11` + - `benv pa1` - Creates env with latest Python and pip + - `benv pa1 python=3.11 numpy pandas` - Creates env with specific packages + - `benv pa1 -c conda-forge python=3.11` - Uses conda-forge channel - **Output**: - Creates and activates the new environment. - **Exceptions**: @@ -22,13 +25,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/cact.md b/docs/shdoc/bin/shinclude/functions/cact.md index da6cba9e..683a4cae 100644 --- a/docs/shdoc/bin/shinclude/functions/cact.md +++ b/docs/shdoc/bin/shinclude/functions/cact.md @@ -8,6 +8,7 @@ - `cact [-h] [env_name]` - **Options**: - `-h` Show this help message + - `-x` Enable debug mode - **Input Parameters**: - `env_name` (string) - The name of the environment to activate. - **Output**: @@ -17,13 +18,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/ccln.md b/docs/shdoc/bin/shinclude/functions/ccln.md index d36f068c..5e48ac79 100644 --- a/docs/shdoc/bin/shinclude/functions/ccln.md +++ b/docs/shdoc/bin/shinclude/functions/ccln.md @@ -1,5 +1,5 @@ ## ccln -# Function: clan +# Function: ccln `ccln` - Clone current Virtual Environment ## Description - **Purpose**: @@ -11,6 +11,7 @@ - `ccln [-h] [new_env_name]` - **Options**: - `-h` Show this help message + - `-x` Enable debug mode - **Input Parameters**: - `new_env_name` (string) - The name of the new cloned environment. - **Output**: @@ -20,13 +21,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/check_lib.md b/docs/shdoc/bin/shinclude/functions/check_lib.md new file mode 100644 index 00000000..8af165a9 --- /dev/null +++ b/docs/shdoc/bin/shinclude/functions/check_lib.md @@ -0,0 +1,28 @@ +## check_lib +TODO Figure out why this function does not work in the other lib scripts. +# Function: _source_check +`_source_check` - Guard against re-sourcing the same script +## Description +- **Purpose**: + - Prevents re-sourcing the same script multiple times. +- **Usage**: + - `_source_check "${BASH_SOURCE[0]}"` +- **Input Parameters**: + - `file_to_source`: The path to the script to source. +- **Output**: + - Returns 1 if the script has already been sourced, otherwise returns 0. +- **Exceptions**: + - None. + +## Definition + +* [init_lib.sh](../init_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/colortext.md b/docs/shdoc/bin/shinclude/functions/colortext.md new file mode 100644 index 00000000..e3f638cf --- /dev/null +++ b/docs/shdoc/bin/shinclude/functions/colortext.md @@ -0,0 +1,30 @@ +## colortext +# Function: colortext +`colortext` - Prints a message to STDERR with ANSI color codes. +## Description +- **Purpose**: + - Prints a message to STDERR with ANSI color codes. +- **Usage**: + - `colortext ` +- **Options**: + - `-h` Show this help message +- **Input Parameters**: + - `text_color`: The text color to use. + - `background_color`: The background color to use. + - `style_code`: The style code to use. + - `message`: The message to print. +- **Output**: + - Prints a message to STDERR with ANSI color codes. + +## Definition + +* [string_lib.sh](../string_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/dact.md b/docs/shdoc/bin/shinclude/functions/dact.md index adf85e0c..3bec6440 100644 --- a/docs/shdoc/bin/shinclude/functions/dact.md +++ b/docs/shdoc/bin/shinclude/functions/dact.md @@ -8,6 +8,7 @@ - `dact [-h]` - **Options**: - `-h` Show this help message + - `-x` Enable debug mode - **Input Parameters**: - None - **Output**: @@ -17,13 +18,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/denv.md b/docs/shdoc/bin/shinclude/functions/denv.md index 0d3fb8b2..1240e29b 100644 --- a/docs/shdoc/bin/shinclude/functions/denv.md +++ b/docs/shdoc/bin/shinclude/functions/denv.md @@ -8,6 +8,7 @@ - `denv [-h] [env_name]` - **Options**: - `-h` Show this help message + - `-x` Enable debug mode - **Input Parameters**: - `env_name` (string) - The name of the environment to delete. - **Output**: @@ -17,13 +18,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/do_wrapper.md b/docs/shdoc/bin/shinclude/functions/do_wrapper.md index 7985014a..3769411c 100644 --- a/docs/shdoc/bin/shinclude/functions/do_wrapper.md +++ b/docs/shdoc/bin/shinclude/functions/do_wrapper.md @@ -2,27 +2,27 @@ # Function: do_wrapper `do_wrapper` - General wrapper function for logging specific command actions. ## Description -- **Purpose**: +- **Purpose**: - Executes a Python package manager command with optional logging based on the specified action. -- **Usage**: +- **Usage**: - `do_wrapper ` -- **Input Parameters**: +- **Input Parameters**: - `cmd` (string) - The command to be executed. - `additional parameters` (string) - Any additional parameters to be passed to the command. -- **Output**: +- **Output**: - None -- **Exceptions**: +- **Exceptions**: - None ## Definition -* [wrapper_funcs.sh](../wrapper_funcs_sh.md) +* [wrapper_lib.sh](../wrapper_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:56 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/docs_base_path.md b/docs/shdoc/bin/shinclude/functions/docs_base_path.md index 1088b7e9..ac5b48e1 100644 --- a/docs/shdoc/bin/shinclude/functions/docs_base_path.md +++ b/docs/shdoc/bin/shinclude/functions/docs_base_path.md @@ -15,13 +15,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/errfind.md b/docs/shdoc/bin/shinclude/functions/errfind.md index 7849b3a2..97033bea 100644 --- a/docs/shdoc/bin/shinclude/functions/errfind.md +++ b/docs/shdoc/bin/shinclude/functions/errfind.md @@ -17,13 +17,13 @@ ## Definition -* [errno.sh](../errno_sh.md) +* [errno_lib.sh](../errno_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/errno.md b/docs/shdoc/bin/shinclude/functions/errno.md index 5d41277e..69cb064a 100644 --- a/docs/shdoc/bin/shinclude/functions/errno.md +++ b/docs/shdoc/bin/shinclude/functions/errno.md @@ -18,13 +18,13 @@ ## Definition -* [errno.sh](../errno_sh.md) +* [errno_lib.sh](../errno_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/errno_exit.md b/docs/shdoc/bin/shinclude/functions/errno_exit.md index 110e3b9c..03e68748 100644 --- a/docs/shdoc/bin/shinclude/functions/errno_exit.md +++ b/docs/shdoc/bin/shinclude/functions/errno_exit.md @@ -1,37 +1,36 @@ ## errno_exit # Function: errno_exit - `errno_exit` - Prints an error to STDERR using the errno function and exits with the error number. + `errno_exit` - Prints an error message to STDERR and exits with the error number. ## Description - **Purpose**: - - Prints an error message to STDERR using the `errno` function and exits the script with the corresponding error number. - You may use the POSIX error code or the error number. + - Prints an error message to STDERR using the provided error code and optional message. + - Exits the script with the corresponding error number. + - Accepts either POSIX error codes (e.g. EACCES) or error numbers (e.g. 13). - **Usage**: - - `errno_exit ` + - `errno_exit [message]` - **Example**: - - `errno_exit EACCES` - - `errno_exit 13` + - `errno_exit EACCES "Failed to access file"` + - `errno_exit 13 "Permission denied"` - **Input Parameters**: - - `errno_code`: The errno code to generate an error for. + - `errno_code`: The error code to use (POSIX name or number) + - `message`: (Optional) Additional message to include in the error - **Output**: - - Outputs an error message to STDERR with caller info including: - - Function name that called errno_exit - - Line number where errno_exit was called - - Source file where errno_exit was called - - Function name that called the function that called errno_exit - - Line number where that function was called - - Source file containing that function call -- **Exceptions**: - - Exits the script with the provided error number. + - Prints error messages to STDERR including: + - Optional custom message if provided + - Call stack trace with function name, line number and file + - Error message corresponding to the error code +- **Exit Status**: + - Exits with the numeric error code ## Definition -* [errno.sh](../errno_sh.md) +* [errno_lib.sh](../errno_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/errno_warn.md b/docs/shdoc/bin/shinclude/functions/errno_warn.md index 198cf824..b0eaa7fd 100644 --- a/docs/shdoc/bin/shinclude/functions/errno_warn.md +++ b/docs/shdoc/bin/shinclude/functions/errno_warn.md @@ -1,38 +1,36 @@ ## errno_warn # Function: errno_warn - `errno_warn` - Prints a warning using the errno function to STDERR and returns the error number. + `errno_warn` - Prints a warning message to STDERR and returns the error number. ## Description - **Purpose**: - - Prints a warning message to STDERR using the `errno` function and sets the return code. It - will report the error without exiting the script. You may use the POSIX error code or the - error number. + - Prints a warning message to STDERR using the provided error code and optional message. + - Sets the return code but does not exit the script. + - Accepts either POSIX error codes (e.g. EACCES) or error numbers (e.g. 13). - **Usage**: - - `errno_warn ` + - `errno_warn [message]` - **Example**: - - `errno_warn EACCES` - - `errno_warn 13` + - `errno_warn EACCES "Failed to access file"` + - `errno_warn 13 "Permission denied"` - **Input Parameters**: - - `errno_code`: The errno code to generate a warning for. + - `errno_code`: The error code to use (POSIX name or number) + - `message`: (Optional) Additional message to include in the warning - **Output**: - - Outputs a warning message to STDERR with caller info including: - - Function name that called errno_warn - - Line number where errno_warn was called - - Source file where errno_warn was called - - Function name that called the function that called errno_warn - - Line number where that function was called - - Source file containing that function call -- **Exceptions**: - - Returns the error number associated with the provided errno code. + - Prints warning messages to STDERR including: + - Optional custom message if provided + - Call stack trace with function name, line number and file + - Error message corresponding to the error code +- **Return Value**: + - Returns the numeric error code ## Definition -* [errno.sh](../errno_sh.md) +* [errno_lib.sh](../errno_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/errval.md b/docs/shdoc/bin/shinclude/functions/errval.md index 6ddd61d1..2c975b86 100644 --- a/docs/shdoc/bin/shinclude/functions/errval.md +++ b/docs/shdoc/bin/shinclude/functions/errval.md @@ -23,13 +23,13 @@ ## Definition -* [errno.sh](../errno_sh.md) +* [errno_lib.sh](../errno_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/escape_string.md b/docs/shdoc/bin/shinclude/functions/escape_string.md new file mode 100644 index 00000000..c454d3d4 --- /dev/null +++ b/docs/shdoc/bin/shinclude/functions/escape_string.md @@ -0,0 +1,28 @@ +## escape_string +# Function: escape_string +`escape_string` - Escape Special Characters in a String +## Description +- **Purpose**: + - Escapes special characters in a string to make it safe for shell commands. +- **Usage**: + - `escaped_str=$(escape_string "Hello & goodbye!" '&"<>;')` +- **Input Parameters**: + - `dirty_string`: The string to escape. + - `chars_to_escape`: The characters to escape. +- **Output**: + - The escaped string. +- **Exceptions**: + - None. + +## Definition + +* [string_lib.sh](../string_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/expand_variable.md b/docs/shdoc/bin/shinclude/functions/expand_variable.md new file mode 100644 index 00000000..c0347e53 --- /dev/null +++ b/docs/shdoc/bin/shinclude/functions/expand_variable.md @@ -0,0 +1,25 @@ +## expand_variable +# Function: expand_variable +`expand_variable` - Expands variables in a given string. +## Description +- **Purpose**: + - This function takes a string and expands variables within it. +- **Usage**: + - `expand_variable "string_with_variables"` +- **Input Parameters**: + - `string_with_variables`: The string containing variables to be expanded. +- **Output**: + - The expanded string with variables replaced by their values. + +## Definition + +* [config_lib.sh](../config_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:19 diff --git a/docs/shdoc/bin/shinclude/functions/function_description.md b/docs/shdoc/bin/shinclude/functions/function_description.md index 4f64ffc2..f4a37a54 100644 --- a/docs/shdoc/bin/shinclude/functions/function_description.md +++ b/docs/shdoc/bin/shinclude/functions/function_description.md @@ -15,13 +15,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/general_help.md b/docs/shdoc/bin/shinclude/functions/general_help.md index b4a59a51..16e4428a 100644 --- a/docs/shdoc/bin/shinclude/functions/general_help.md +++ b/docs/shdoc/bin/shinclude/functions/general_help.md @@ -17,13 +17,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/generate_markdown.md b/docs/shdoc/bin/shinclude/functions/generate_markdown.md index cafe56b3..20aa6af5 100644 --- a/docs/shdoc/bin/shinclude/functions/generate_markdown.md +++ b/docs/shdoc/bin/shinclude/functions/generate_markdown.md @@ -17,13 +17,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/get_function_hash.md b/docs/shdoc/bin/shinclude/functions/get_function_hash.md index 41e1934e..10ecaca6 100644 --- a/docs/shdoc/bin/shinclude/functions/get_function_hash.md +++ b/docs/shdoc/bin/shinclude/functions/get_function_hash.md @@ -2,26 +2,26 @@ # Function: get_function_hash `get_function_hash` - Get the hash of a function's definition. ## Description -- **Purpose**: +- **Purpose**: - Computes the hash of a function's definition for integrity checks. -- **Usage**: +- **Usage**: - `get_function_hash [function_name]` -- **Input Parameters**: +- **Input Parameters**: - `function_name` (string) - The name of the function to hash. -- **Output**: +- **Output**: - The hash of the function's definition. -- **Exceptions**: +- **Exceptions**: - None ## Definition -* [wrapper_funcs.sh](../wrapper_funcs_sh.md) +* [wrapper_lib.sh](../wrapper_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:56 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/get_script_readme_file.md b/docs/shdoc/bin/shinclude/functions/get_script_readme_file.md index 1dd255f8..5a9f9418 100644 --- a/docs/shdoc/bin/shinclude/functions/get_script_readme_file.md +++ b/docs/shdoc/bin/shinclude/functions/get_script_readme_file.md @@ -16,13 +16,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/get_system_readme_file.md b/docs/shdoc/bin/shinclude/functions/get_system_readme_file.md index cd3ab9be..726c4306 100644 --- a/docs/shdoc/bin/shinclude/functions/get_system_readme_file.md +++ b/docs/shdoc/bin/shinclude/functions/get_system_readme_file.md @@ -15,13 +15,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/handle_variable.md b/docs/shdoc/bin/shinclude/functions/handle_variable.md new file mode 100644 index 00000000..82e75016 --- /dev/null +++ b/docs/shdoc/bin/shinclude/functions/handle_variable.md @@ -0,0 +1,35 @@ +## handle_variable +# Function: handle_variable +`handle_variable` - Manage variable assignments based on predefined actions +## Description +- **Purpose**: + - Controls how variables are assigned based on a predefined action table. + - Supports merging, setting, preserving config values, or discarding changes. +- **Usage**: + - `handle_variable "var_name" "value_ref"` +- **Input Parameters**: + - `var_name`: The name of the variable to handle. + - `value_ref`: Name reference to the value to potentially assign. +- **Output**: + - None. Modifies the target variable according to its action rule. +- **Exceptions**: + - Exits with code 2 if the action for the variable is unknown. +- **Required Global Variables**: + - `var_actions`: Associative array mapping variable names to actions: + - "merge": Combines existing value with new value + - "set": Uses the new value + - "config": Preserves the config file value + - "discard": Keeps the original value + +## Definition + +* [type_lib.sh](../type_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/help_functions.md b/docs/shdoc/bin/shinclude/functions/help_functions.md index 76c5fae1..eb5776e9 100644 --- a/docs/shdoc/bin/shinclude/functions/help_functions.md +++ b/docs/shdoc/bin/shinclude/functions/help_functions.md @@ -17,13 +17,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/help_scripts.md b/docs/shdoc/bin/shinclude/functions/help_scripts.md index 02bb3893..39220261 100644 --- a/docs/shdoc/bin/shinclude/functions/help_scripts.md +++ b/docs/shdoc/bin/shinclude/functions/help_scripts.md @@ -17,13 +17,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/init_help_system.md b/docs/shdoc/bin/shinclude/functions/init_help_system.md index 5aed8f4e..81f79c1d 100644 --- a/docs/shdoc/bin/shinclude/functions/init_help_system.md +++ b/docs/shdoc/bin/shinclude/functions/init_help_system.md @@ -9,7 +9,7 @@ - **Scope**: - Global. Modifies the global array `__VENV_FUNCTIONS`. - **Input Parameters**: - - None. Internally iterates over the scripts listed in the `__VENV_SOURCED_LIST` array. + - None. - **Output**: - Populates `__VENV_FUNCTIONS` with function names and their corresponding documentation. - Sorts `__VENV_FUNCTIONS` based on function names. @@ -18,13 +18,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/lastenv.md b/docs/shdoc/bin/shinclude/functions/lastenv.md index 389645e0..307406a4 100644 --- a/docs/shdoc/bin/shinclude/functions/lastenv.md +++ b/docs/shdoc/bin/shinclude/functions/lastenv.md @@ -15,13 +15,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/lenv.md b/docs/shdoc/bin/shinclude/functions/lenv.md index fc4b2d7f..b0b188f4 100644 --- a/docs/shdoc/bin/shinclude/functions/lenv.md +++ b/docs/shdoc/bin/shinclude/functions/lenv.md @@ -14,6 +14,7 @@ - `-t` Sort by last update time - `-r` Reverse the sort order - `-h` Show this help message + - `-x` Enable debug mode - **Output**: - A list of all existing conda virtual environments with their last modification date. - The active environment is marked with an asterisk. @@ -28,13 +29,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/load_pkg_config.md b/docs/shdoc/bin/shinclude/functions/load_pkg_config.md new file mode 100644 index 00000000..efe60048 --- /dev/null +++ b/docs/shdoc/bin/shinclude/functions/load_pkg_config.md @@ -0,0 +1,60 @@ +## load_pkg_config +# Function: load_pkg_config +`load_pkg_config` - Loads package configuration from setup.cf file. +## Description +- **Purpose**: + - This function reads the setup.cf file and sets variables for package installation. +- **Usage**: + - `load_pkg_config` +- **Input Parameters**: + - None. +- **Output**: + - Sets variables from the setup.cf file for package installation. +- **Exceptions**: + - Returns specific error codes if the setup.cf file is not found or invalid. +- **Examples** setup.cf file: + ``` + # Package Configuration File for venvutil + ## Define variables + prefix=$HOME/local/venvutil + exec_prefix=${prefix} + libdir=${exec_prefix}/lib + includedir=${prefix}/include + bindir=${exec_prefix}/bin + datadir=${prefix}/share + sysconfdir=${prefix}/etc + include_dirs=("bin" "docs" "conf") + include_files=("README.md" "LICENSE" "setup.sh" "setup.cf" "manifest.lst") + # Package metadata + Name: venvutil + Description: Virtual Environment Utilities + Version: 0.4.0 + Repository: https://github.com/unixwzrd/venvutil + License: Apache License, Version 2.0 + Support: https://github.com/unixwzrd/venvutil/issues + Contribute: https://patreon.com/unixwzrd + Contribute: https://www.ko-fi.com/unixwzrd + # Dependencies (if any) + Requires: python >= 3.10 + Requires: bash >= 4.0 + Requires: Conda >= 22.11 + Requires: macOS + Requires: Linux + Conflicts: + # Compiler and linker flags (if applicable) + # Cflags: -I${includedir} + # Libs: -L${libdir} -lvenvutil + ``` + +## Definition + +* [config_lib.sh](../config_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:19 diff --git a/docs/shdoc/bin/shinclude/functions/log_message.md b/docs/shdoc/bin/shinclude/functions/log_message.md index 84bfe189..8cf2bb50 100644 --- a/docs/shdoc/bin/shinclude/functions/log_message.md +++ b/docs/shdoc/bin/shinclude/functions/log_message.md @@ -25,13 +25,13 @@ ## Definition -* [errno.sh](../errno_sh.md) +* [errno_lib.sh](../errno_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/nenv.md b/docs/shdoc/bin/shinclude/functions/nenv.md index c7d17780..fd514faa 100644 --- a/docs/shdoc/bin/shinclude/functions/nenv.md +++ b/docs/shdoc/bin/shinclude/functions/nenv.md @@ -8,6 +8,7 @@ - `nenv [-h] PREFIX [EXTRA_OPTIONS]` - **Options**: - `-h` Show this help message + - `-x` Enable debug mode - **Input Parameters**: - `PREFIX` (string) - The prefix to identify the series of environments. - `EXTRA_OPTIONS` (string, optional) - Additional options to pass to the environment creation. @@ -18,13 +19,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/next_step.md b/docs/shdoc/bin/shinclude/functions/next_step.md index 70ae44d5..e5f8c92d 100644 --- a/docs/shdoc/bin/shinclude/functions/next_step.md +++ b/docs/shdoc/bin/shinclude/functions/next_step.md @@ -17,13 +17,13 @@ ## Definition -* [util_funcs.sh](../util_funcs_sh.md) +* [util_lib.sh](../util_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/pact.md b/docs/shdoc/bin/shinclude/functions/pact.md index 66fd34fb..e4acf927 100644 --- a/docs/shdoc/bin/shinclude/functions/pact.md +++ b/docs/shdoc/bin/shinclude/functions/pact.md @@ -8,6 +8,7 @@ - `pact [-h]` - **Options**: - `-h` Show this help message + - `-x` Enable debug mode - **Input Parameters**: - None - **Output**: @@ -17,13 +18,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/parse_manifest_metadata.md b/docs/shdoc/bin/shinclude/functions/parse_manifest_metadata.md new file mode 100644 index 00000000..017a9be3 --- /dev/null +++ b/docs/shdoc/bin/shinclude/functions/parse_manifest_metadata.md @@ -0,0 +1,24 @@ +## parse_manifest_metadata +# Function: parse_manifest_metadata +`parse_manifest_metadata` - Parses manifest metadata. +## Description +- **Purpose**: + - This function reads the manifest file and sets variables for package installation. +- **Usage**: + - `parse_manifest_metadata `# - **Input Parameters**: + - `manifest_file`: The path to the manifest file. +- **Output**: + - Sets variables from the manifest file for package installation. + +## Definition + +* [config_lib.sh](../config_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:19 diff --git a/docs/shdoc/bin/shinclude/functions/pip.md b/docs/shdoc/bin/shinclude/functions/pip.md index 557353db..3c469c43 100644 --- a/docs/shdoc/bin/shinclude/functions/pip.md +++ b/docs/shdoc/bin/shinclude/functions/pip.md @@ -2,27 +2,27 @@ # Function: pip `pip` - Wrapper function for pip commands. ## Description -- **Purpose**: +- **Purpose**: - Wraps pip commands to ensure environment variables are preserved. provides logging for pip commands and the virtual environment affected -- **Usage**: +- **Usage**: - `pip [arguments]` -- **Input Parameters**: +- **Input Parameters**: - `arguments` (string) - Arguments to pass to pip. -- **Output**: +- **Output**: - Executes the pip command with the provided arguments. -- **Exceptions**: +- **Exceptions**: - None ## Definition -* [wrapper_funcs.sh](../wrapper_funcs_sh.md) +* [wrapper_lib.sh](../wrapper_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:56 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/pkg_config_vars.md b/docs/shdoc/bin/shinclude/functions/pkg_config_vars.md new file mode 100644 index 00000000..49e53000 --- /dev/null +++ b/docs/shdoc/bin/shinclude/functions/pkg_config_vars.md @@ -0,0 +1,27 @@ +## pkg_config_vars +# Function: `pkg_config_vars` +`pkg_config_vars` - Sets up the actions for the variables in a package config file. +## Description +- **Purpose**: + - Sets up the actions for the variables in a package config file. +- **Usage**: + - `pkg_config_vars` +- **Input Parameters**: + - None. +- **Output**: + - None. +- **Exceptions**: + - None. + +## Definition + +* [config_lib.sh](../config_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:19 diff --git a/docs/shdoc/bin/shinclude/functions/pop_stack.md b/docs/shdoc/bin/shinclude/functions/pop_stack.md index d28cf6e5..c499e755 100644 --- a/docs/shdoc/bin/shinclude/functions/pop_stack.md +++ b/docs/shdoc/bin/shinclude/functions/pop_stack.md @@ -17,13 +17,13 @@ ## Definition -* [util_funcs.sh](../util_funcs_sh.md) +* [util_lib.sh](../util_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/pop_venv.md b/docs/shdoc/bin/shinclude/functions/pop_venv.md index 70f4049d..d687f2d7 100644 --- a/docs/shdoc/bin/shinclude/functions/pop_venv.md +++ b/docs/shdoc/bin/shinclude/functions/pop_venv.md @@ -15,13 +15,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/process_scripts.md b/docs/shdoc/bin/shinclude/functions/process_scripts.md index 20dacc31..80d4ddd6 100644 --- a/docs/shdoc/bin/shinclude/functions/process_scripts.md +++ b/docs/shdoc/bin/shinclude/functions/process_scripts.md @@ -15,13 +15,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/ptree.md b/docs/shdoc/bin/shinclude/functions/ptree.md index ba0b74dd..61617376 100644 --- a/docs/shdoc/bin/shinclude/functions/ptree.md +++ b/docs/shdoc/bin/shinclude/functions/ptree.md @@ -18,13 +18,13 @@ ## Definition -* [util_funcs.sh](../util_funcs_sh.md) +* [util_lib.sh](../util_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/push_stack.md b/docs/shdoc/bin/shinclude/functions/push_stack.md index 59253799..60b0d4c0 100644 --- a/docs/shdoc/bin/shinclude/functions/push_stack.md +++ b/docs/shdoc/bin/shinclude/functions/push_stack.md @@ -18,13 +18,13 @@ ## Definition -* [util_funcs.sh](../util_funcs_sh.md) +* [util_lib.sh](../util_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/push_venv.md b/docs/shdoc/bin/shinclude/functions/push_venv.md index a4e7027e..0a55252e 100644 --- a/docs/shdoc/bin/shinclude/functions/push_venv.md +++ b/docs/shdoc/bin/shinclude/functions/push_venv.md @@ -15,13 +15,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/renv.md b/docs/shdoc/bin/shinclude/functions/renv.md index ba03f0a1..796b1e61 100644 --- a/docs/shdoc/bin/shinclude/functions/renv.md +++ b/docs/shdoc/bin/shinclude/functions/renv.md @@ -8,6 +8,7 @@ - `renv [-h]` - **Options**: - `-h` Show this help message + - `-x` Enable debug mode - **Input Parameters**: - None - **Output**: @@ -17,13 +18,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/sanitize.md b/docs/shdoc/bin/shinclude/functions/sanitize.md new file mode 100644 index 00000000..2ec40b9f --- /dev/null +++ b/docs/shdoc/bin/shinclude/functions/sanitize.md @@ -0,0 +1,31 @@ +## sanitize +# Function: sanitize +`sanitize` - Removes every character not in the specified allowed set. +## Description +- **Purpose**: + - Removes every character not in the specified allowed set. +- **Usage**: + - `sanitized_value=$(sanitize "Hello? *" 'a-zA-Z0-9._*\- ')` +- **Input Parameters**: + - `dirty_string`: The string to sanitize. + - `allowed_chars`: The characters to keep in the string. +- **Output**: + - The sanitized string. +- **Examples**: + - `sanitize "Hello? *" 'a-zA-Z0-9._*\- '` + - `sanitize "Hello? *" 'a-zA-Z0-9._*\- '` +- **Exceptions**: + - None. + +## Definition + +* [string_lib.sh](../string_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/script_description.md b/docs/shdoc/bin/shinclude/functions/script_description.md index 15af1956..d4c00469 100644 --- a/docs/shdoc/bin/shinclude/functions/script_description.md +++ b/docs/shdoc/bin/shinclude/functions/script_description.md @@ -15,13 +15,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/set_debug.md b/docs/shdoc/bin/shinclude/functions/set_debug.md new file mode 100644 index 00000000..235fc533 --- /dev/null +++ b/docs/shdoc/bin/shinclude/functions/set_debug.md @@ -0,0 +1,30 @@ +## set_debug +# Function set_debug +`set_debug` - Sets the debug level for the script. +## Description +- **Purpose**: + - Sets the debug level for the script. + - Debug level may be given as a value between 0 and 99 + - Debug level may be given as a text log level of the highest level to display: + DEBUG, INFO, WARNING, ERROR, CRITICAL, or SILENT +- **Usage**: + - `set_debug ` +- **Input Parameters**: + - `debug_level`: The debug level to set. +- **Output**: + - Sets the debug level for the script. +- **Exceptions**: + - Warns if invalid level provided and keeps previous level + +## Definition + +* [errno_lib.sh](../errno_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:19 diff --git a/docs/shdoc/bin/shinclude/functions/set_variable.md b/docs/shdoc/bin/shinclude/functions/set_variable.md new file mode 100644 index 00000000..4699be8b --- /dev/null +++ b/docs/shdoc/bin/shinclude/functions/set_variable.md @@ -0,0 +1,31 @@ +## set_variable +# Function: set_variable +`set_variable` - Assign a value to a named variable, respecting existing type or inferring new +## Description +- **Purpose**: + - Assigns a value to a named variable, respecting its existing type or inferring a new type. + - Handles arrays, associative arrays, scalars, and integers. +- **Usage**: + - `set_variable "var_name" "value_ref1"` + - `set_variable "var_name" "value_ref1" "value_ref2"` +- **Input Parameters**: + - `var_name`: The name of the variable to set. + - `value_ref1`: Name reference to first value/array to assign. + - `value_ref2`: (Optional) Name reference to second value/array to merge with first. +- **Output**: + - None. Sets the target variable with the provided value(s). +- **Exceptions**: + - None. Will create new variable if it doesn't exist. + +## Definition + +* [type_lib.sh](../type_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/snum.md b/docs/shdoc/bin/shinclude/functions/snum.md index 607c442d..894ff403 100644 --- a/docs/shdoc/bin/shinclude/functions/snum.md +++ b/docs/shdoc/bin/shinclude/functions/snum.md @@ -15,13 +15,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/sort_2d_array.md b/docs/shdoc/bin/shinclude/functions/sort_2d_array.md index 479e0e3c..a97c0d37 100644 --- a/docs/shdoc/bin/shinclude/functions/sort_2d_array.md +++ b/docs/shdoc/bin/shinclude/functions/sort_2d_array.md @@ -26,13 +26,13 @@ ## Definition -* [util_funcs.sh](../util_funcs_sh.md) +* [util_lib.sh](../util_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/source_util_script.md b/docs/shdoc/bin/shinclude/functions/source_lib.md similarity index 55% rename from docs/shdoc/bin/shinclude/functions/source_util_script.md rename to docs/shdoc/bin/shinclude/functions/source_lib.md index f77dfc5c..97a95d37 100644 --- a/docs/shdoc/bin/shinclude/functions/source_util_script.md +++ b/docs/shdoc/bin/shinclude/functions/source_lib.md @@ -1,12 +1,20 @@ -## source_util_script -# Function: source_util_script - `source_util_script` - Sources a utility script from the specified directory. +## source_lib +# Function: source_lib + `source_lib` - Sources a utility script from the specified directory. ## Description - **Purpose**: - Sources a utility script from the specified directory. It's a helper function used within the `init_env.sh` script to modularity load additional scripts as needed. + - Try to locate the SH_LIB directory which contains included scripts and functions in a + "standard" fashion. places we could look are, in this order: + - $(dirname "${THIS_SCRIPT}") + - $(dirname "${THIS_SCRIPT}")/shinclude + - $HOME/shinclude + - $HOME/bin directory + - $HOME/bin/shinclude directory + - from the environment variable `SH_LIB` - **Usage**: - - `source_util_script "script_name"` + - `source_lib "script_name"` - **Input Parameters**: - `script_name`: The name of the script to source (without the `.sh` extension). - **Output**: @@ -17,13 +25,13 @@ ## Definition -* [init_env.sh](../init_env_sh.md) +* [init_lib.sh](../init_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/specific_function_help.md b/docs/shdoc/bin/shinclude/functions/specific_function_help.md index d8c38c99..6bbe23b2 100644 --- a/docs/shdoc/bin/shinclude/functions/specific_function_help.md +++ b/docs/shdoc/bin/shinclude/functions/specific_function_help.md @@ -17,13 +17,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/specific_script_help.md b/docs/shdoc/bin/shinclude/functions/specific_script_help.md index d7feef48..88fb37eb 100644 --- a/docs/shdoc/bin/shinclude/functions/specific_script_help.md +++ b/docs/shdoc/bin/shinclude/functions/specific_script_help.md @@ -17,13 +17,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/stack_op.md b/docs/shdoc/bin/shinclude/functions/stack_op.md index 71e1cf7e..a8d4d7d8 100644 --- a/docs/shdoc/bin/shinclude/functions/stack_op.md +++ b/docs/shdoc/bin/shinclude/functions/stack_op.md @@ -19,13 +19,13 @@ ## Definition -* [util_funcs.sh](../util_funcs_sh.md) +* [util_lib.sh](../util_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/stringclean.md b/docs/shdoc/bin/shinclude/functions/stringclean.md index 80455566..3b6babef 100644 --- a/docs/shdoc/bin/shinclude/functions/stringclean.md +++ b/docs/shdoc/bin/shinclude/functions/stringclean.md @@ -15,13 +15,13 @@ ## Definition -* [util_funcs.sh](../util_funcs_sh.md) +* [string_lib.sh](../string_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/strip_space.md b/docs/shdoc/bin/shinclude/functions/strip_space.md index 19e9b77f..bb323255 100644 --- a/docs/shdoc/bin/shinclude/functions/strip_space.md +++ b/docs/shdoc/bin/shinclude/functions/strip_space.md @@ -15,13 +15,13 @@ ## Definition -* [util_funcs.sh](../util_funcs_sh.md) +* [string_lib.sh](../string_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/to_upper.md b/docs/shdoc/bin/shinclude/functions/to_upper.md index e8c2a3ab..0338ef31 100644 --- a/docs/shdoc/bin/shinclude/functions/to_upper.md +++ b/docs/shdoc/bin/shinclude/functions/to_upper.md @@ -15,13 +15,13 @@ ## Definition -* [util_funcs.sh](../util_funcs_sh.md) +* [string_lib.sh](../string_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/var_type.md b/docs/shdoc/bin/shinclude/functions/var_type.md index 66114572..e491f01d 100644 --- a/docs/shdoc/bin/shinclude/functions/var_type.md +++ b/docs/shdoc/bin/shinclude/functions/var_type.md @@ -4,29 +4,29 @@ ## Description - **Purpose**: - Retrieves the type of a variable. -- **Usage**: +- **Usage**: - `var_type [-h] "var_name"` -- **Options**: +- **Options**: - `-h` Show this help message -- **Examples**: +- **Examples**: - `var_type "my_variable"` - `var_type=$(var_type "my_variable") -- **Input Parameters**: +- **Input Parameters**: - `var_name`: The name of the variable. -- **Output**: +- **Output**: - The type of the variable as a string. Can be one of `array`, `associative`, `scalar`, or `unknown`. -- **Exceptions**: +- **Exceptions**: - None. ## Definition -* [util_funcs.sh](../util_funcs_sh.md) +* [type_lib.sh](../type_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/venvdiff.md b/docs/shdoc/bin/shinclude/functions/vdiff.md similarity index 83% rename from docs/shdoc/bin/shinclude/functions/venvdiff.md rename to docs/shdoc/bin/shinclude/functions/vdiff.md index 708196f4..a61cd926 100644 --- a/docs/shdoc/bin/shinclude/functions/venvdiff.md +++ b/docs/shdoc/bin/shinclude/functions/vdiff.md @@ -1,4 +1,4 @@ -## venvdiff +## vdiff # Function: venvdiff `venvdiff` - Compare Two Virtual Environments. ## Description @@ -8,6 +8,7 @@ - `venvdiff [-h] [env1] [env2]` - **Options**: - `-h` Show this help message + - `-x` Enable debug mode - **Input Parameters**: - `env1` (string) - The first environment to compare. - `env2` (string) - The second environment to compare. @@ -18,13 +19,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/vdsc.md b/docs/shdoc/bin/shinclude/functions/vdsc.md index 10112a78..1ad56978 100644 --- a/docs/shdoc/bin/shinclude/functions/vdsc.md +++ b/docs/shdoc/bin/shinclude/functions/vdsc.md @@ -15,13 +15,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/vhelp.md b/docs/shdoc/bin/shinclude/functions/vhelp.md index d47eadcc..3dd4c614 100644 --- a/docs/shdoc/bin/shinclude/functions/vhelp.md +++ b/docs/shdoc/bin/shinclude/functions/vhelp.md @@ -17,13 +17,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/vnum.md b/docs/shdoc/bin/shinclude/functions/vnum.md index ada4b15e..8b168ac9 100644 --- a/docs/shdoc/bin/shinclude/functions/vnum.md +++ b/docs/shdoc/bin/shinclude/functions/vnum.md @@ -15,13 +15,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/vpfx.md b/docs/shdoc/bin/shinclude/functions/vpfx.md index ce645434..1caf35bc 100644 --- a/docs/shdoc/bin/shinclude/functions/vpfx.md +++ b/docs/shdoc/bin/shinclude/functions/vpfx.md @@ -15,13 +15,13 @@ ## Definition -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/vren.md b/docs/shdoc/bin/shinclude/functions/vren.md new file mode 100644 index 00000000..6c9141fc --- /dev/null +++ b/docs/shdoc/bin/shinclude/functions/vren.md @@ -0,0 +1,32 @@ +## vren +# Function: vren +`vren` - Rename a Virtual Environment. +## Description +- **Purpose**: + - Renames a virtual environment and makes that environment current. +- **Usage**: + - `vren [-h] [new_name]` +- **Options**: + - `-h` Show this help message + - `-x` Enable debug mode +- **Input Parameters**: + - `old_name` (optional string) - The current name of the environment or + the current environment if not specified. + - `new_name` (string) - The new name of the environment. +- **Output**: + - Renames the specified environment. +- **Exceptions**: + - Errors if the environment does not exist. + +## Definition + +* [venv_lib.sh](../venv_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/functions/write_config.md b/docs/shdoc/bin/shinclude/functions/write_config.md new file mode 100644 index 00000000..9f59486b --- /dev/null +++ b/docs/shdoc/bin/shinclude/functions/write_config.md @@ -0,0 +1,26 @@ +## write_config +# Function: write_config +`write_config` - Write variables in valid Bash syntax (scalar vs array). +## Description +- **Purpose**: + - Writes variables in valid Bash syntax (scalar vs array). +- **Usage**: + - `write_config "/path/to/output.conf" config_variables[@]` +- **Input Parameters**: + - `config_file`: The path to the output file. + - `config_variables`: An array of variable names to write. +- **Output**: + - Writes the variables to the output file in valid Bash syntax. + +## Definition + +* [config_lib.sh](../config_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:19 diff --git a/docs/shdoc/bin/shinclude/functions/write_function_doc.md b/docs/shdoc/bin/shinclude/functions/write_function_doc.md index f4b85a2d..24bbc1e3 100644 --- a/docs/shdoc/bin/shinclude/functions/write_function_doc.md +++ b/docs/shdoc/bin/shinclude/functions/write_function_doc.md @@ -15,13 +15,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/write_page_footer.md b/docs/shdoc/bin/shinclude/functions/write_page_footer.md index 73e8ea7f..651d13ca 100644 --- a/docs/shdoc/bin/shinclude/functions/write_page_footer.md +++ b/docs/shdoc/bin/shinclude/functions/write_page_footer.md @@ -15,13 +15,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/write_script_doc.md b/docs/shdoc/bin/shinclude/functions/write_script_doc.md index 56bfbb35..02e4ef0f 100644 --- a/docs/shdoc/bin/shinclude/functions/write_script_doc.md +++ b/docs/shdoc/bin/shinclude/functions/write_script_doc.md @@ -17,13 +17,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/write_script_function_entry.md b/docs/shdoc/bin/shinclude/functions/write_script_function_entry.md index 9f4c8ba5..ce2a1fcb 100644 --- a/docs/shdoc/bin/shinclude/functions/write_script_function_entry.md +++ b/docs/shdoc/bin/shinclude/functions/write_script_function_entry.md @@ -16,13 +16,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/write_script_readme_header.md b/docs/shdoc/bin/shinclude/functions/write_script_readme_header.md index 4f636e99..583b9244 100644 --- a/docs/shdoc/bin/shinclude/functions/write_script_readme_header.md +++ b/docs/shdoc/bin/shinclude/functions/write_script_readme_header.md @@ -17,13 +17,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/write_system_readme_entry.md b/docs/shdoc/bin/shinclude/functions/write_system_readme_entry.md index c59a662f..28f7ba5e 100644 --- a/docs/shdoc/bin/shinclude/functions/write_system_readme_entry.md +++ b/docs/shdoc/bin/shinclude/functions/write_system_readme_entry.md @@ -16,13 +16,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/write_system_readme_header.md b/docs/shdoc/bin/shinclude/functions/write_system_readme_header.md index b16276bb..aee66aad 100644 --- a/docs/shdoc/bin/shinclude/functions/write_system_readme_header.md +++ b/docs/shdoc/bin/shinclude/functions/write_system_readme_header.md @@ -15,13 +15,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/write_table_footer.md b/docs/shdoc/bin/shinclude/functions/write_table_footer.md index da780fcb..504018e0 100644 --- a/docs/shdoc/bin/shinclude/functions/write_table_footer.md +++ b/docs/shdoc/bin/shinclude/functions/write_table_footer.md @@ -15,13 +15,13 @@ ## Definition -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/functions/zero_pad.md b/docs/shdoc/bin/shinclude/functions/zero_pad.md index 0430664c..0dafb38d 100644 --- a/docs/shdoc/bin/shinclude/functions/zero_pad.md +++ b/docs/shdoc/bin/shinclude/functions/zero_pad.md @@ -19,13 +19,13 @@ ## Definition -* [util_funcs.sh](../util_funcs_sh.md) +* [util_lib.sh](../util_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/help_sys_sh.md b/docs/shdoc/bin/shinclude/helpsys_lib_sh.md similarity index 91% rename from docs/shdoc/bin/shinclude/help_sys_sh.md rename to docs/shdoc/bin/shinclude/helpsys_lib_sh.md index 4d4d431d..4ebf2e02 100644 --- a/docs/shdoc/bin/shinclude/help_sys_sh.md +++ b/docs/shdoc/bin/shinclude/helpsys_lib_sh.md @@ -1,6 +1,6 @@ -# Functions Defined in Script: help_sys.sh +# Functions Defined in Script: helpsys_lib.sh -### [help_sys.sh](/docs/shdoc/bin/shinclude/scripts/help_sys.sh.md) - Help System Functions for Bash Scripts +### [helpsys_lib.sh](/docs/shdoc/bin/shinclude/scripts/helpsys_lib.sh.md) - Help System Functions for Bash Scripts ## List of Functions Defined @@ -35,8 +35,8 @@ Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/init_env_sh.md b/docs/shdoc/bin/shinclude/init_env_sh.md deleted file mode 100644 index 2b213298..00000000 --- a/docs/shdoc/bin/shinclude/init_env_sh.md +++ /dev/null @@ -1,21 +0,0 @@ -# Functions Defined in Script: init_env.sh - -### [init_env.sh](/docs/shdoc/bin/shinclude/scripts/init_env.sh.md) - Initialize Environment and Source Utility Scripts - -## List of Functions Defined - -| Function | Description | -|:--|:--| -| [source_util_script](functions/source_util_script.md) | Sources a utility script from the specified directory. | - ---- - ---- - -Website: [unixwzrd.ai](https://unixwzrd.ai) -Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan -Apache License, Version 2.0 - -Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 diff --git a/docs/shdoc/bin/shinclude/init_lib_sh.md b/docs/shdoc/bin/shinclude/init_lib_sh.md new file mode 100644 index 00000000..2e477965 --- /dev/null +++ b/docs/shdoc/bin/shinclude/init_lib_sh.md @@ -0,0 +1,23 @@ +# Functions Defined in Script: init_lib.sh + +### [init_lib.sh](/docs/shdoc/bin/shinclude/scripts/init_lib.sh.md) - Library Initialization and Environment Setup + +## List of Functions Defined + +| Function | Description | +|:--|:--| +| [check_lib](functions/check_lib.md) | # Function: _source_check | +| [_source_check](functions/_source_check.md) | ## Definition | +| [source_lib](functions/source_lib.md) | Sources a utility script from the specified directory. | + +--- + +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/scripts/config_lib.sh.md b/docs/shdoc/bin/shinclude/scripts/config_lib.sh.md new file mode 100644 index 00000000..d4542421 --- /dev/null +++ b/docs/shdoc/bin/shinclude/scripts/config_lib.sh.md @@ -0,0 +1,62 @@ +# Script: config_lib.sh +`config_lib.sh` - Configuration Management and Variable Handling +# +## Description +- **Purpose**: + - Manages configuration file loading + - Handles variable initialization + - Controls configuration precedence + - Provides configuration validation +# + ## Configuration handling + - Dependency resolution +# + ## Configuration handling + - Modeled to use pkg-config like files. + - Performs variable expansion on assigned values including arrays. + - Variables can be set using the following syntax: + - prefix=$HOME/local/venvutil + - exec_prefix=${prefix} + - libdir=${exec_prefix}/lib + - includedir=${prefix}/include + - bindir=${exec_prefix}/bin + - datadir=${prefix}/share + - sysconfdir=${prefix}/etc +# + ## Variable Handling + - using an associative array with the variable names as keys and an action to use the values. + - Associative array contains (key) VARNAME: (value) Action pairs. + - Actions are the following: + - "set" - set the variable to the value + - "merge" - merge the value into the variable + - "config" - from the config settings + - "discard" - discard and use the default or standard config value. +# +## Usage +- Source this script in your Bash scripts to utilize its functions. + ```bash + source_lib config_lib + ``` +## Input Parameters + - None. +## Output + - Sets variables from the setup.cf file for package installation. +## Exceptions + - Returns specific error codes if the setup.cf file is not found or invalid. +## Initialization + - Ensures the script is sourced only once and initializes necessary variables. + + + +## Defined in Script + +* [config_lib.sh](../config_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:19 diff --git a/docs/shdoc/bin/shinclude/scripts/errno.sh.md b/docs/shdoc/bin/shinclude/scripts/errno_lib.sh.md similarity index 63% rename from docs/shdoc/bin/shinclude/scripts/errno.sh.md rename to docs/shdoc/bin/shinclude/scripts/errno_lib.sh.md index f79ada88..c879e77b 100644 --- a/docs/shdoc/bin/shinclude/scripts/errno.sh.md +++ b/docs/shdoc/bin/shinclude/scripts/errno_lib.sh.md @@ -1,36 +1,36 @@ -# Script: errno.sh -`errno.sh` - Provides POSIX errno codes and utilities for Bash scripts +# Script: errno_lib.sh +`errno_lib.sh` - Provides POSIX errno codes and utilities for Bash scripts ## Description - **Purpose**: - Offers functions to retrieve and manage POSIX error codes within Bash scripts. -- **Usage**: +## Usage - Source this script in your Bash scripts to utilize error code utilities. ```bash - source /path/to/errno.sh + source /path/to/errno_lib.sh ``` -- **Input Parameters**: +## Input Parameters - None. -- **Output**: +## Output - Functions that output error codes and messages. -- **Exceptions**: +## Exceptions - Returns specific error codes if system `errno.h` is not found or invalid errno codes are provided. -- **Initialization**: +## Initialization - Ensures the script is sourced only once and initializes necessary variables. # ## Dependencies -- `util_funcs.sh` (for utility functions like `to_upper`) +- `string_lib.sh` (for utility functions like `to_upper`) ## Defined in Script -* [errno.sh](../errno_sh.md) +* [errno_lib.sh](../errno_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:19 diff --git a/docs/shdoc/bin/shinclude/scripts/help_sys.sh.md b/docs/shdoc/bin/shinclude/scripts/helpsys_lib.sh.md similarity index 75% rename from docs/shdoc/bin/shinclude/scripts/help_sys.sh.md rename to docs/shdoc/bin/shinclude/scripts/helpsys_lib.sh.md index 4e4d3cf6..c9e95d90 100644 --- a/docs/shdoc/bin/shinclude/scripts/help_sys.sh.md +++ b/docs/shdoc/bin/shinclude/scripts/helpsys_lib.sh.md @@ -1,36 +1,36 @@ -# Script: help_sys.sh -`help_sys.sh` - Help System Functions for Bash Scripts +# Script: helpsys_lib.sh +`helpsys_lib.sh` - Help System Functions for Bash Scripts ## Description - **Purpose**: - Provides a dynamic help system for all sourced bash scripts. - It can list available functions, provide detailed information about each function, and list sourced scripts. -- **Usage**: + ## Usage - Source this script in other bash scripts to enable the dynamic help system. ```bash - source help_sys.sh + source helpsys_lib.sh ``` -- **Input Parameters**: +## Input Parameters - None. All input is handled by the individual functions. -- **Output**: +## Output - Enables a help system that can be accessed by calling `help` in the terminal. - Supports generating Markdown documentation. -- **Exceptions**: +## Exceptions - Some functions may return specific error codes or print error messages to STDERR. - Refer to individual function documentation for details. -- **Environment**: +## Environment - **MD_PROCESSOR**: Set to the markdown processor of your choice. If `glow` is in your path, it will use that. ## Defined in Script -* [help_sys.sh](../help_sys_sh.md) +* [helpsys_lib.sh](../helpsys_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:54 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/scripts/init_env.sh.md b/docs/shdoc/bin/shinclude/scripts/init_lib.sh.md similarity index 76% rename from docs/shdoc/bin/shinclude/scripts/init_env.sh.md rename to docs/shdoc/bin/shinclude/scripts/init_lib.sh.md index 77aaf544..ec0dd245 100644 --- a/docs/shdoc/bin/shinclude/scripts/init_env.sh.md +++ b/docs/shdoc/bin/shinclude/scripts/init_lib.sh.md @@ -1,43 +1,44 @@ -# Script: init_env.sh -`init_env.sh` - Initialize Environment and Source Utility Scripts +# Script: init_lib.sh +`init_lib.sh` - Library Initialization and Environment Setup +# ## Description - **Purpose**: - Initializes the environment for bash scripting, particularly in the context of managing virtual environments. It sets up the necessary environment and sources utility scripts required for the proper functioning of other scripts in the system. It is responsible for orchestrating the environment setup in the correct order and can also be used to source additional environment or setup scripts as required, such as `.env.local` files. -- **Usage**: +## Usage - Source this script in other bash scripts to import the necessary environment and utility functions. It also contains a function that can be called to perform environment setup tasks in user scripts. To use it, include the following line in your bash scripts: ```bash - source /path/to/init_env.sh + source /path/to/init_lib.sh ``` -- **Input Parameters**: +## Input Parameters - None. The script operates without requiring any input parameters. -- **Output**: +## Output - Sets up the environment, sources utility scripts, and prepares the system for managing virtual environments. -- **Exceptions**: +## Exceptions - Exits with code 1 if it fails to find any of the required scripts or if any part of the initialization process fails. # ## Dependencies - Utility scripts located in `__VENV_INCLUDE`: - - `util_funcs.sh` - - `help_sys.sh` - - `errno.sh` - - `venv_funcs.sh` - - `wrapper_funcs.sh` + - `util_lib.sh` + - `helpsys_lib.sh` + - `errno_lib.sh` + - `venv_lib.sh` + - `wrapper_lib.sh` - Conda environment ## Defined in Script -* [init_env.sh](../init_env_sh.md) +* [init_lib.sh](../init_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/scripts/string_lib.sh.md b/docs/shdoc/bin/shinclude/scripts/string_lib.sh.md new file mode 100644 index 00000000..afc93be8 --- /dev/null +++ b/docs/shdoc/bin/shinclude/scripts/string_lib.sh.md @@ -0,0 +1,35 @@ +# Script: string_lib.sh +`string_lib.sh` - Consolidated string sanitization & variable expansion library +# +## Description +- **Purpose**: + - Provides functions for: + - Sanitizing strings with a custom "allowed character set" + - Escaping specific characters with a backslash + - Expanding shell variables in a string (handling $VAR or ${VAR}) +## Usage + - Source this script in your Bash scripts to utilize its functions. + ```bash + source_lib string_lib + ``` +## Input Parameters + - None +## Output + - None +## Exceptions + - None + + + +## Defined in Script + +* [string_lib.sh](../string_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/scripts/type_lib.sh.md b/docs/shdoc/bin/shinclude/scripts/type_lib.sh.md new file mode 100644 index 00000000..5bfa8c98 --- /dev/null +++ b/docs/shdoc/bin/shinclude/scripts/type_lib.sh.md @@ -0,0 +1,33 @@ +# Script: type_lib.sh +`type_lib.sh` - Support functions for variable type handling. +## Description +- **Purpose**: + - Offers functions to handle variable types. +## Usage + - Source this script in your Bash scripts to utilize its functions. + ```bash + source /path/to/type_lib.sh + ``` +## Input Parameters + - None. +## Output + - Sets variables from the setup.cf file for package installation. +## Exceptions + - Returns specific error codes if the setup.cf file is not found or invalid. +## Initialization + - Ensures the script is sourced only once and initializes necessary variables. + + + +## Defined in Script + +* [type_lib.sh](../type_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/scripts/util_funcs.sh.md b/docs/shdoc/bin/shinclude/scripts/util_lib.sh.md similarity index 67% rename from docs/shdoc/bin/shinclude/scripts/util_funcs.sh.md rename to docs/shdoc/bin/shinclude/scripts/util_lib.sh.md index 65c9c1cc..a0aae473 100644 --- a/docs/shdoc/bin/shinclude/scripts/util_funcs.sh.md +++ b/docs/shdoc/bin/shinclude/scripts/util_lib.sh.md @@ -1,18 +1,19 @@ -# Script: util_funcs.sh -`util_funcs.sh` - Utility Functions for Bash Scripts +# Script: util_lib.sh +`util_lib.sh` - Utility Functions and Cross-Platform Support +# ## Description - **Purpose**: - - Provides a collection of utility functions to assist in various common tasks within Bash scripting. These functions offer streamlined solutions for string manipulation, number padding, and stack operations, enhancing the efficiency and readability of Bash scripts that incorporate them. -- **Usage**: + - Provides a collection of utility functions to assist in various common tasks within Bash scripting. These functions offer streamlined solutions for number padding, stack operations, and process tree display, enhancing the efficiency and readability of Bash scripts that incorporate them. +## Usage - Source this script within other Bash scripts to make the utility functions available for use: ```bash - source /path/to/util_funcs.sh + source /path/to/util_lib.sh ``` -- **Input Parameters**: +## Input Parameters - None. This script is intended to be sourced by other scripts and does not require input parameters. -- **Output**: +## Output - Provides utility functions that can be called from other Bash scripts. -- **Exceptions**: +## Exceptions - Some functions within the script may return specific error codes depending on their internal logic. Refer to the individual function documentation for detailed exception handling. # ## Dependencies @@ -22,13 +23,13 @@ ## Defined in Script -* [util_funcs.sh](../util_funcs_sh.md) +* [util_lib.sh](../util_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/scripts/venv_funcs.sh.md b/docs/shdoc/bin/shinclude/scripts/venv_lib.sh.md similarity index 76% rename from docs/shdoc/bin/shinclude/scripts/venv_funcs.sh.md rename to docs/shdoc/bin/shinclude/scripts/venv_lib.sh.md index be980457..e193db9c 100644 --- a/docs/shdoc/bin/shinclude/scripts/venv_funcs.sh.md +++ b/docs/shdoc/bin/shinclude/scripts/venv_lib.sh.md @@ -1,11 +1,11 @@ -# Script: venv_funcs.sh - venv_funcs.sh - Virtual Environment Management Functions for Bash Scripts - ## Description - - **Purpose**: - - This script provides a collection of functions to manage conda virtual environments. - - Functions include creating, deleting, switching, and cloning environments, among others. +# Script: venv_lib.sh +`venv_lib.sh` - Virtual Environment Management Functions for Bash Scripts +## Description +- **Purpose**: + - This script provides a collection of functions to manage conda virtual environments. + - Functions include creating, deleting, switching, and cloning environments, among others. # - ## Usage +## Usage - Source this script in other bash scripts to import the virtual environment management functions. - For example, in another script: `source venv_funcs.sh`. # @@ -39,40 +39,42 @@ - `lastenv()`: Retrieve the Last Environment with a Given Prefix. - `benv()`: Create a New Base Virtual Environment. - `nenv()`: Create a New Virtual Environment in a Series. + - `vren()`: Rename a Virtual Environment # -- **Usage Example**: +## Usage Example ```shellscript - source venv_funcs.sh + source venv_lib.sh benv myenv cact myenv ``` # -- **Dependencies**: +## Dependencies - This script depends on the `conda` command-line tool for managing virtual environments. - The `util_funcs.sh` script is also required and should be located in the same directory as this script. # -- **Notes**: +## Notes - This script assumes that the `conda` command is available in the system's PATH. - It is recommended to source this script in other scripts rather than executing it directly. - Make sure to set the appropriate permissions on this script to allow execution. # -- **Author**: [Your Name] -- **Last Modified**: [Date] -# -- **Version**: [Version Number] +## Author + - Michael Sullivan + - https://unixwzrd.ai/ + - https://github.com/unixwzrd + - https://github.com/unixwzrd/venvutil # ## Defined in Script -* [venv_funcs.sh](../venv_funcs_sh.md) +* [venv_lib.sh](../venv_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/scripts/venvutil_lib.sh.md b/docs/shdoc/bin/shinclude/scripts/venvutil_lib.sh.md new file mode 100644 index 00000000..a3e8e980 --- /dev/null +++ b/docs/shdoc/bin/shinclude/scripts/venvutil_lib.sh.md @@ -0,0 +1,70 @@ +# Script: venvutil_lib.sh +`venvutil_lib.sh` - Primary Entry Point for VenvUtil Library System +# +## Description +- **Purpose**: + - Primary entry point and initialization for the VenvUtil library system + - Manages library loading sequence and dependencies + - Provides core environment management functionality +# +## Usage + - Direct source: `source /path/to/venvutil_lib.sh` + - Via helper: `source_lib venvutil_lib` +# +## Library Loading Sequence + 1. config_lib - Configuration management + 2. errno_lib - Error handling and codes + 3. helpsys_lib - Help system functionality + 4. string_lib - String manipulation and display + 5. type_lib - Type checking and validation + 6. util_lib - Utility functions + 7. venv_lib - Virtual environment management + 8. wrapper_lib - Command wrapping and logging +# +## Dependencies + - Bash 4.0 or higher + - Core library files in same directory + - Python package managers (pip/conda) for some functionality +# +## Environment Variables + - `__VENV_SOURCED` - Tracks loaded libraries + - `__VENV_BASE` - Base directory for VenvUtil + - `__VENV_BIN` - Binary directory location + - `__VENV_INCLUDE` - Library include directory +# +## Debug Support + - Set `DEBUG_VENVUTIL=ON` for debug output + - Individual functions support -x flag for debug mode +# +## Return Codes + - 0: Success + - Non-zero: Various error conditions (see errno_lib.sh) +# +## Examples + ```bash + # Direct usage + source /path/to/venvutil_lib.sh + + # Enable debug mode + DEBUG_VENVUTIL=ON source /path/to/venvutil_lib.sh + ``` +# +## Notes + - This is the primary entry point for the VenvUtil system + - All other libraries should be loaded through this file + - Direct sourcing of other libraries is discouraged + + + +## Defined in Script + +* [venvutil_lib.sh](../venvutil_lib_sh.md) +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/scripts/wrapper_funcs.sh.md b/docs/shdoc/bin/shinclude/scripts/wrapper_lib.sh.md similarity index 73% rename from docs/shdoc/bin/shinclude/scripts/wrapper_funcs.sh.md rename to docs/shdoc/bin/shinclude/scripts/wrapper_lib.sh.md index 0f4c1cae..12cb50da 100644 --- a/docs/shdoc/bin/shinclude/scripts/wrapper_funcs.sh.md +++ b/docs/shdoc/bin/shinclude/scripts/wrapper_lib.sh.md @@ -1,19 +1,19 @@ -# Script: wrapper_funcs.sh -`wrapper_funcs.sh` - Python Package Manager Wrapper Functions +# Script: wrapper_lib.sh +`wrapper_lib.sh` - Python Package Manager Wrapper Functions ## Description -- **Purpose**: +- **Purpose**: - Provides enhanced functionality for managing Python package commands by wrapping pip and conda. - Intercepts and logs changes to virtual environments for rollback, auditing, and future use in venvdiff or vdiff. -- **Usage**: +## Usage - Source this script in your command line environment to import the wrapper functions. - - For example, in another script: `source wrapper_funcs.sh`. -- **Features**: + - For example, in another script: `source wrapper_lib.sh`. +## Features - Saves a `pip freeze` before any potentially destructive changes to a virtual environment. - Logs the complete command line to a log file for both conda and pip. - Persists logs in the `$HOME/.venvutil` directory, even after virtual environments are deleted. -- **Dependencies**: +## Dependencies - Requires Bash and the Python package managers pip and conda. -- **Exceptions**: +## Exceptions - Some functions may return specific error codes or print error messages to STDERR. - Refer to individual function documentation for details. @@ -21,13 +21,13 @@ ## Defined in Script -* [wrapper_funcs.sh](../wrapper_funcs_sh.md) +* [wrapper_lib.sh](../wrapper_lib_sh.md) --- Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/string_lib_sh.md b/docs/shdoc/bin/shinclude/string_lib_sh.md new file mode 100644 index 00000000..ced9823f --- /dev/null +++ b/docs/shdoc/bin/shinclude/string_lib_sh.md @@ -0,0 +1,26 @@ +# Functions Defined in Script: string_lib.sh + +### [string_lib.sh](/docs/shdoc/bin/shinclude/scripts/string_lib.sh.md) - Consolidated string sanitization & variable expansion library + +## List of Functions Defined + +| Function | Description | +|:--|:--| +| [to_upper](functions/to_upper.md) | Convert a String to Uppercase | +| [strip_space](functions/strip_space.md) | Remove Leading and Trailing Whitespaces | +| [sanitize](functions/sanitize.md) | Removes every character not in the specified allowed set. | +| [stringclean](functions/stringclean.md) | Sanitize a String by Removing Non-Alphanumeric Characters | +| [escape_string](functions/escape_string.md) | Escape Special Characters in a String | +| [colortext](functions/colortext.md) | Prints a message to STDERR with ANSI color codes. | + +--- + +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/type_lib_sh.md b/docs/shdoc/bin/shinclude/type_lib_sh.md new file mode 100644 index 00000000..cbbcf1b6 --- /dev/null +++ b/docs/shdoc/bin/shinclude/type_lib_sh.md @@ -0,0 +1,23 @@ +# Functions Defined in Script: type_lib.sh + +### [type_lib.sh](/docs/shdoc/bin/shinclude/scripts/type_lib.sh.md) - Support functions for variable type handling. + +## List of Functions Defined + +| Function | Description | +|:--|:--| +| [var_type](functions/var_type.md) | Get the Type of a Variable | +| [set_variable](functions/set_variable.md) | Assign a value to a named variable, respecting existing type or inferring new | +| [handle_variable](functions/handle_variable.md) | Manage variable assignments based on predefined actions | + +--- + +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:20 diff --git a/docs/shdoc/bin/shinclude/util_funcs_sh.md b/docs/shdoc/bin/shinclude/util_lib_sh.md similarity index 59% rename from docs/shdoc/bin/shinclude/util_funcs_sh.md rename to docs/shdoc/bin/shinclude/util_lib_sh.md index 1c334da0..7019b6f2 100644 --- a/docs/shdoc/bin/shinclude/util_funcs_sh.md +++ b/docs/shdoc/bin/shinclude/util_lib_sh.md @@ -1,22 +1,18 @@ -# Functions Defined in Script: util_funcs.sh +# Functions Defined in Script: util_lib.sh -### [util_funcs.sh](/docs/shdoc/bin/shinclude/scripts/util_funcs.sh.md) - Utility Functions for Bash Scripts +### [util_lib.sh](/docs/shdoc/bin/shinclude/scripts/util_lib.sh.md) - Utility Functions and Cross-Platform Support ## List of Functions Defined | Function | Description | |:--|:--| -| [strip_space](functions/strip_space.md) | Remove Leading and Trailing Whitespaces | | [zero_pad](functions/zero_pad.md) | Pad a Single-Digit Number with a Leading Zero | | [next_step](functions/next_step.md) | Increment a Given Sequence Number by 1 and Pad it with a Zero if Needed | | [sort_2d_array](functions/sort_2d_array.md) | Sort a Two-Dimensional Array | | [push_stack](functions/push_stack.md) | Push a Value onto a Named Stack | | [pop_stack](functions/pop_stack.md) | Pop a Value from a Named Stack | | [stack_op](functions/stack_op.md) | Perform Stack Operations | -| [stringclean](functions/stringclean.md) | Sanitize a String by Removing Non-Alphanumeric Characters | -| [to_upper](functions/to_upper.md) | Convert a String to Uppercase | | [ptree](functions/ptree.md) | Display the Process Tree for a Given PID | -| [var_type](functions/var_type.md) | Get the Type of a Variable | --- @@ -24,8 +20,8 @@ Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/venv_funcs_sh.md b/docs/shdoc/bin/shinclude/venv_lib_sh.md similarity index 80% rename from docs/shdoc/bin/shinclude/venv_funcs_sh.md rename to docs/shdoc/bin/shinclude/venv_lib_sh.md index 7703a68a..9dd56579 100644 --- a/docs/shdoc/bin/shinclude/venv_funcs_sh.md +++ b/docs/shdoc/bin/shinclude/venv_lib_sh.md @@ -1,6 +1,6 @@ -# Functions Defined in Script: venv_funcs.sh +# Functions Defined in Script: venv_lib.sh -### [venv_funcs.sh](/docs/shdoc/bin/shinclude/scripts/venv_funcs.sh.md) - Virtual Environment Management Functions for Bash Scripts +### [venv_lib.sh](/docs/shdoc/bin/shinclude/scripts/venv_lib.sh.md) - Virtual Environment Management Functions for Bash Scripts ## List of Functions Defined @@ -23,7 +23,8 @@ | [denv](functions/denv.md) | Delete a Virtual Environment. | | [renv](functions/renv.md) | Revert to Previous Virtual Environment. | | [ccln](functions/ccln.md) | Clone current Virtual Environment | -| [venvdiff](functions/venvdiff.md) | Compare Two Virtual Environments. | +| [vren](functions/vren.md) | Rename a Virtual Environment. | +| [vdiff](functions/vdiff.md) | Compare Two Virtual Environments. | --- @@ -31,8 +32,8 @@ Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:55 +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/venvutil_lib_sh.md b/docs/shdoc/bin/shinclude/venvutil_lib_sh.md new file mode 100644 index 00000000..d7c81c24 --- /dev/null +++ b/docs/shdoc/bin/shinclude/venvutil_lib_sh.md @@ -0,0 +1,20 @@ +# Functions Defined in Script: venvutil_lib.sh + +### [venvutil_lib.sh](/docs/shdoc/bin/shinclude/scripts/venvutil_lib.sh.md) - Primary Entry Point for VenvUtil Library System + +## List of Functions Defined + +| Function | Description | +|:--|:--| + +--- + +--- + +Website: [unixwzrd.ai](https://unixwzrd.ai) +Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) +Copyright (c) 2025 Michael Sullivan +Apache License, Version 2.0 + +Generated Markdown Documentation +Generated on: 2025-02-11 at 00:27:21 diff --git a/docs/shdoc/bin/shinclude/wrapper_funcs_sh.md b/docs/shdoc/bin/shinclude/wrapper_lib_sh.md similarity index 71% rename from docs/shdoc/bin/shinclude/wrapper_funcs_sh.md rename to docs/shdoc/bin/shinclude/wrapper_lib_sh.md index 6f82e8e5..0725402e 100644 --- a/docs/shdoc/bin/shinclude/wrapper_funcs_sh.md +++ b/docs/shdoc/bin/shinclude/wrapper_lib_sh.md @@ -1,6 +1,6 @@ -# Functions Defined in Script: wrapper_funcs.sh +# Functions Defined in Script: wrapper_lib.sh -### [wrapper_funcs.sh](/docs/shdoc/bin/shinclude/scripts/wrapper_funcs.sh.md) - Python Package Manager Wrapper Functions +### [wrapper_lib.sh](/docs/shdoc/bin/shinclude/scripts/wrapper_lib.sh.md) - Python Package Manager Wrapper Functions ## List of Functions Defined @@ -17,8 +17,8 @@ Website: [unixwzrd.ai](https://unixwzrd.ai) Github Repo: [venvutil](https://github.com/unixwzrd/venvutil) -Copyright (c) 2024 Michael Sullivan +Copyright (c) 2025 Michael Sullivan Apache License, Version 2.0 Generated Markdown Documentation -Generated on: Generated: 2025 01 03 at 02:39:56 +Generated on: 2025-02-11 at 00:27:21 diff --git a/manifest.lst b/manifest.lst index b8cffc9f..371fd0a2 100644 --- a/manifest.lst +++ b/manifest.lst @@ -1,48 +1,64 @@ # This file uses pipe-separated fields +c | . | | transcript.md | | | | | +c | bin | | numpy-1.26-reinst.sh | | | | | +c | bin/shinclude | | init_env.sh | | | | | +c | docs/shdoc/bin/shinclude | | init_env_sh.md | | | | | c | modules | | requirements-stdt.txt | | | | | -d | | | bin | 755 | | | 1056 | +d | | | bin | 755 | | | 1312 | d | | | conf | 755 | | | 352 | -d | | | docs | 755 | | | 544 | -d | bin | bin | shinclude | 755 | | | 256 | +d | | | docs | 755 | | | 672 | +d | | | modules | 755 | | | 96 | +d | bin | bin | shinclude | 755 | | | 384 | d | docs | docs | shdoc | 755 | | | 192 | d | docs/shdoc | docs/shdoc | bin | 755 | | | 128 | -d | docs/shdoc/bin | docs/shdoc/bin | shinclude | 755 | | | 352 | -d | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | functions | 755 | | | 2048 | -d | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | scripts | 755 | | | 256 | +d | docs/shdoc/bin | docs/shdoc/bin | shinclude | 755 | | | 480 | +d | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | functions | 755 | | | 2528 | +d | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | scripts | 755 | | | 384 | f | | | LICENSE | 644 | | | 11362 | fbb090d446bc51f5b8611e8c59bddf5447f155e2 -f | | | README.md | 644 | | | 16044 | d5491b1d2c3c353d096a32bf311ebfc8a03c7717 -f | | | manifest.lst | 644 | | | 17641 | 26869f878da4efb03319f10d8f6d7a6b36c8f8c2 -f | | | setup.cf | 644 | | | 799 | b61b02ff439616b7f46a85b4e3ae3078f3282073 -f | | | setup.sh | 755 | | | 20706 | e37ac02cbcf2826707c1463fbc09b5bba50c9f9d -f | bin | bin | buildvenvs | 755 | | | 14995 | acd98e35e3b7052dd1e068729169855f204efd6c +f | | | README.md | 644 | | | 19507 | 491669ee7800912f8afb7864d8896f7391f0ff02 +f | | | manifest.lst | 644 | | | 23103 | 472eb940bb352ba3e448f211d8f75543307ba043 +f | | | requirements-build.txt | 644 | | | 185 | 5befca95d85d0ff6665950e90a120accc77f2dd4 +f | | | requirements.txt | 644 | | | 82 | 7ce33b2cefe2dafe0ec69a943e5321206c8b3697 +f | | | setup.cf | 644 | | | 985 | cb71a80732c5a5992f9f5b9919387bf9612f9144 +f | | | setup.sh | 755 | | | 20121 | 98e8c1ab2ccc2c7698e907cfe1a633e9c477fe7d +f | bin | bin | buildvenvs | 755 | | | 15033 | f6b06def5460f5a76d338bc3f0e37482b963b0f6 f | bin | bin | chunkfile.py | 755 | | | 9656 | a47edafccbf88d398f7d5a25f87b2c2713767aab +f | bin | bin | compare_test | 755 | | | 1446 | 2af66cb27d06860e6cfe78574b80aea07a30e322 +f | bin | bin | extract_chat.py | 755 | | | 32669 | bbbf6afa034799472a6884b0106cd805ec66c0a3 f | bin | bin | filetree.py | 755 | | | 12422 | af3965837e85ca485c91e55f366fd0c34810b307 f | bin | bin | filter-vm_stat | 755 | | | 2606 | 7eccba1613a50f1be68e3bb026328766a349ef61 -f | bin | bin | generate_manifest.sh | 755 | | | 5707 | 1f77e3ae38b7cd1a9c51951001205018175530b7 -f | bin | bin | genmd | 755 | | | 49545 | 8ee360e4bc7a5c8e4d35d14e1589cb0b71d8c386 +f | bin | bin | generate_manifest.sh | 755 | | | 6718 | 62b5e8a545a8df4060fb22170ccbae9a8d9eeb70 +f | bin | bin | genmd | 755 | | | 48543 | 276014debab2e7eea1514c76df7eedb1a443a5c4 f | bin | bin | ld | 755 | | | 2697 | 61e28d8e54ae13b00d6c8f7e0cb4e01bc768b798 -f | bin | bin | numpy-1.26-reinst.sh | 755 | | | 343 | a1fcc77cd28f16dd6527ffa3c932a6206cdc9721 -f | bin | bin | numpybench | 755 | | | 3625 | dd5b984895064eb0b6f1ee7da318ac766732da1b +f | bin | bin | numpy-comp.sh | 755 | | | 1760 | dc90c4c300f335c5b28ee96d93d53609f054bee1 +f | bin | bin | numpy_torture.py | 755 | | | 5006 | b74789f9f4739141bae59d35b7ea1e210ff65b12 +f | bin | bin | numpybench | 755 | | | 3937 | ff389a083a1ce3fa2ff1f48bbd206bd178991ed5 f | bin | bin | numpyprof | 755 | | | 1579 | fe821ae6a607ff116a9ae9f9abc8483125cdfcac f | bin | bin | numpytime | 755 | | | 1423 | 14349c95f754bc9260a2bf6d4a84b1b6f1126336 -f | bin | bin | purgevenv | 755 | | | 905 | 5dccd04b18d676390fb85e45ff268c5ee0ce8d7d -f | bin | bin | runbench | 755 | | | 765 | 66887eaff9734443a19ac2e4b67283d265eb00aa +f | bin | bin | purgevenv | 755 | | | 917 | 7203a065dc5ed9fae2581f4debfaf76b4c411fbd +f | bin | bin | rename-chat.py | 755 | | | 6661 | 52366026c60ba56ebb31d202e5cc3915648ddee9 +f | bin | bin | runbench | 755 | | | 777 | b24f94142dbbbcb1fd0c8f6b5ccf82b72d8dbcee f | bin | bin | stressgpu | 755 | | | 395 | 488c9966529abe9c59adb64b6b82106581c69d9f f | bin | bin | tensorgpu | 755 | | | 1209 | 31437adec4daebb8e9aaf3b0fee135f1b5b49e78 -f | bin | bin | tokencount.py | 755 | | | 2839 | 072d66dc0c56b4cff52c046d3478cf1c89712132 +f | bin | bin | tokencount.py | 755 | | | 2831 | 104124637338b633bfc4369e5c919cf39e8efbb9 f | bin | bin | torch-test | 755 | | | 656 | c25bc144b14725ac69d6918b17876b78cfbd30bc +f | bin | bin | torch_torture.py | 755 | | | 7541 | ca2f8afa27450fc74248258df2dfc28e41d0659d f | bin | bin | torchamp | 755 | | | 724 | 88e556163c153b72fa1cfe1bfdd264ee3104e623 f | bin | bin | torchbench | 755 | | | 1435 | a9e03bb113953edb49d424c1f627ed873de048c3 f | bin | bin | torchprof | 755 | | | 1544 | 0e1d1482cd52f587fc0e8d92082ad17e077479bd f | bin | bin | torchtime | 755 | | | 1389 | 54a4dcdd6ac16189f2be61f8d71ed46ab3fdd312 f | bin | bin | vinfo | 755 | | | 4789 | 9997bdc1649925c2ee46d15235d504d7bba64a03 f | bin | bin | warehouse.sh | 755 | | | 3211 | 7ef782147dacf52bcdcf6f1cd7464a1fe8b14e11 -f | bin/shinclude | bin/shinclude | errno.sh | 755 | | | 12986 | 7fa15e93c3f002ad748d8cd3535050b96cf4e8ac -f | bin/shinclude | bin/shinclude | help_sys.sh | 755 | | | 33004 | 75bbc9a8936d751e512637b5cefd0b1f7f083c34 -f | bin/shinclude | bin/shinclude | init_env.sh | 755 | | | 4003 | a000a59aa32cb60a5b535160a1c8361e3849fd77 -f | bin/shinclude | bin/shinclude | util_funcs.sh | 755 | | | 14664 | 396c22b209f423b346fa719cf65a25d59e6f4d5f -f | bin/shinclude | bin/shinclude | venv_funcs.sh | 755 | | | 24897 | bcd89a8f449e7704cecbc630ad2235cfe7379cde -f | bin/shinclude | bin/shinclude | wrapper_funcs.sh | 755 | | | 9082 | 05829dec084c64e2a900df2aa20dbb2048d91896 +f | bin/shinclude | bin/shinclude | config_lib.sh | 755 | | | 13329 | dc1e1fdcc3618f5b245ec371ea558956e551808a +f | bin/shinclude | bin/shinclude | errno_lib.sh | 755 | | | 15802 | 44a30c69c34eb235f4f34c6d2f69dea323fe4a32 +f | bin/shinclude | bin/shinclude | helpsys_lib.sh | 755 | | | 33245 | 702bf632896fcecc4659e90fc10c41363c7e57d1 +f | bin/shinclude | bin/shinclude | init_lib.sh | 755 | | | 6724 | e822fc9d70227ec60804a2946892fb21135ad619 +f | bin/shinclude | bin/shinclude | string_lib.sh | 755 | | | 8707 | 0b538dc5f5991cc496c976da36e864c613c25944 +f | bin/shinclude | bin/shinclude | type_lib.sh | 755 | | | 8175 | 71ae47b30f6d06e3f9de2e29174ff0e59e6cba3f +f | bin/shinclude | bin/shinclude | util_lib.sh | 755 | | | 12627 | 29ffe800c02b0486225f12cd79e9eb2f5e16a14a +f | bin/shinclude | bin/shinclude | venv_lib.sh | 755 | | | 30367 | 9a96d837b738917439becf2d5639261a1b4de300 +f | bin/shinclude | bin/shinclude | venvutil_lib.sh | 755 | | | 3155 | 9a77e8caa2fe8c340b08b5dfe613ecb7464fad18 +f | bin/shinclude | bin/shinclude | wrapper_lib.sh | 755 | | | 10889 | 6957028637e368e3411a5f307e9d80b017831bca f | conf | conf | config-a | 755 | | | 10167 | 3b7bbdf007f090ccecba521ef654708ec1e4ec8c f | conf | conf | config-a.sh | 755 | | | 10167 | 3b7bbdf007f090ccecba521ef654708ec1e4ec8c f | conf | conf | config-b.sh | 750 | | | 8038 | 8f3a0adf7b6004ae381846c3ba64b318ad8ed707 @@ -53,97 +69,128 @@ f | conf | conf | config-webui-macOS.sh | 750 | | | 7485 | 8ba93a91dfc5cc2ebc6 f | conf | conf | config-webui.sh | 750 | | | 6114 | f5aeb3d99bafb6feec604f3544c8638cad240785 f | conf | conf | help_sys.conf | 640 | | | 419 | 29768a4c81029cba3a20ba8b147ad03b4013a9c5 f | docs | docs | BuildGraph.txt | 644 | | | 1562 | 79c4ad4cd934656babe97b3786dfb0311df89bdb -f | docs | docs | Documentation_Generation.md | 644 | | | 1811 | f40fbc97bf59ddffefc2e9b7b64808f8eb7ad880 +f | docs | docs | Documentation_Generation.md | 644 | | | 1811 | e8d40cdbf7e225fc94ab87d803cc1ea1140afdb8 f | docs | docs | Function_Doc_Templ.md | 644 | | | 1436 | 7948eb4a191d7ce56be65a10be0cbd38095ef1da +f | docs | docs | Metrics_Layout.md | 644 | | | 3153 | 37be3642c76c64f47f01ebe2e0addccabd441c91 f | docs | docs | Script_Doc_Templ.md | 644 | | | 3185 | 623eabdfb544a4f8cc00201d3939353df5961bf8 -f | docs | docs | Standards.md | 644 | | | 2360 | 02f72dbe8d3ac5dd0ed8e9c7bcc77b72d3b67c4d +f | docs | docs | Standards.md | 644 | | | 2178 | 462100e67a60014c05bd60edb55564ed0bf45ab0 +f | docs | docs | chat_tools.md | 644 | | | 2731 | c65cd177ab5a78b2bff8ef0a1c6e74d7be0657ab f | docs | docs | chunk-offsets.md | 644 | | | 6475 | 7d053e1d9e4231ab5a0f2a62daec2c1a35231981 f | docs | docs | chunkfile.md | 644 | | | 2753 | 24e3d9f55746d43954899e65f5f7ca7c07713415 -f | docs | docs | filetree.md | 644 | | | 4662 | b4a1a3e6dd540dfde4c2e80363c1e3c87c6161f3 +f | docs | docs | filetree.md | 644 | | | 4662 | b8b12e3e99c04c6431edf5afecba1294381cf848 f | docs | docs | generate_manifest.md | 644 | | | 3872 | 74ec5e5f34d840fbb32d2d03fa23151e7963313f f | docs | docs | genmd.md | 644 | | | 15931 | 05debeaf580617d4d0d60670a9a258a9952f97cc -f | docs | docs | installer-manifest.md | 644 | | | 6299 | 1504cfa14990e2fbd9c56e8022f864c01d24e6a5 +f | docs | docs | installer-manifest.md | 644 | | | 7085 | 470ae2c740a9414273fc9aacf01d154d02f93f45 +f | docs | docs | migration_guide.md | 644 | | | 4393 | 20469c8f541cb8e8a7e6b3593c75669e134b62ea f | docs | docs | numpybench.md | 644 | | | 2627 | 1e3ad54d8f4f2aaf34bdce6583c0d6901e94444e +f | docs | docs | performance_testing.md | 644 | | | 4664 | ef4e3e2e37838664cca906024865c97eef76c457 f | docs | docs | warehouse.md | 644 | | | 3039 | c7fe4ad3e74302516dafee83d8dd8cdabaa47886 f | docs/shdoc | docs/shdoc | AUTO_GENERATED_DO_NOT_MODIFY_OR_PLACE_FILES_HERE | 644 | | | 0 | da39a3ee5e6b4b0d3255bfef95601890afd80709 -f | docs/shdoc | docs/shdoc | README.md | 644 | | | 1018 | 41721a32094d9d5943ed1261260a60075e782da1 -f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | errno_sh.md | 644 | | | 1160 | df2d21a9821f7a4b916de52241debc0c23586ab2 -f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | help_sys_sh.md | 644 | | | 2874 | bdacbb44e2d059dc3b8dacee15a0aba2dcd4ae8f -f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | init_env_sh.md | 644 | | | 610 | d6af95249d6d9bba80fd0b7a7d9695ae049a30ca -f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | util_funcs_sh.md | 644 | | | 1381 | 3320e95e3cea4faff7fbba4cfaecef4dff96fec9 -f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | venv_funcs_sh.md | 644 | | | 1868 | 94b1e16fb1a487ca98261a8b0a754968065f78cb -f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | wrapper_funcs_sh.md | 644 | | | 899 | f22f23280af61d7e4669c3481e30562484b9ef4e -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | __set_venv_vars.md | 644 | | | 666 | 2ffee4448652556ee6a541382f8d4a35ba3f2799 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | __venv_conda_check.md | 644 | | | 840 | 69d45c2c8f548a2608812d0fae1205e2d0ef6162 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | benv.md | 644 | | | 1002 | 34088c2cff7591d527243f9e7e8d2032d882deb1 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | cact.md | 644 | | | 775 | 8e4c99b4c6df98650a10c4ca3cdfef651c1abb8b -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | ccln.md | 644 | | | 1019 | 3151c8e74ebda614f9585b53ea1ee5e58ac1326c -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | dact.md | 644 | | | 647 | 0988d41d04ac366685cdbc4249982faf4510e1c7 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | denv.md | 644 | | | 755 | e7be65fd48a3a00a2a1b9247eee36c93af122892 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | do_wrapper.md | 644 | | | 845 | 45afa41b9db03a0fae7e7a205162267fdd851194 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | docs_base_path.md | 644 | | | 675 | c394143f606d142912aba4f63878330b25acfa89 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | errfind.md | 644 | | | 844 | c49d3fc207430dd3a313204fddff044f6db9fb2c -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | errno.md | 644 | | | 1063 | 7c4f2c068b1d5d9926775be7270348843a9b9061 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | errno_exit.md | 644 | | | 1293 | 1a913812cff41a771b2a6835d59212a03248d493 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | errno_warn.md | 644 | | | 1337 | 0003df7658261c7eb1100dc21fd9de9abf3f532a -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | errval.md | 644 | | | 1068 | fe2be2484f739375a869c7cad2a05239aa383f01 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | function_description.md | 644 | | | 759 | b44d057c94ea50def578f19611154180c87fd0f8 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | general_help.md | 644 | | | 688 | 9f5ad5beea86ad71965972876c7d8d8c204b6178 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | generate_markdown.md | 644 | | | 744 | 639a14bbe4431c5cec9616700872255bcae95a62 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | get_function_hash.md | 644 | | | 765 | beda97022c90eb24b6cec3b711754b1ddf0e6fd8 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | get_script_readme_file.md | 644 | | | 834 | c4e5735419974ed961789a4d198076d37593fa9e -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | get_system_readme_file.md | 644 | | | 790 | fc1b4b47ccd7bbcdb6a4c35e749fef280f862877 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | help_functions.md | 644 | | | 872 | 3cabf596b08dbb11f375cc7d1099eaaab99c130a -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | help_scripts.md | 644 | | | 655 | aafc2d93f052e34259fc2c22978b225023b33ddb -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | init_help_system.md | 644 | | | 1071 | 6852f8b06ab61e234e6d0c300f9a3e2aa0e6332e -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | lastenv.md | 644 | | | 713 | 41b75d692c3ebb3bff41697b6ceb13a890029c12 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | lenv.md | 644 | | | 1578 | d776a61f146fb29aa7910c556a44bda49592639b -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | log_message.md | 644 | | | 1291 | ac6f24de61baf087885238d7717cc3a8700306d6 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | nenv.md | 644 | | | 1003 | 423cf90ec061790f752cb20dde6a5b20866588c6 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | next_step.md | 644 | | | 968 | 56a1aefbbc5aa650046262d7b261a3e9ec85a222 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | pact.md | 644 | | | 698 | bc7b40b253c8374ca9e3981d6a5bc031546d7bd6 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | pip.md | 644 | | | 776 | 8022adb609857650a6b8f22dbff0a87985de3519 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | pop_stack.md | 644 | | | 795 | 916b4f00367eede629b7f1e66ac2e0ac54e17fb1 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | pop_venv.md | 644 | | | 671 | a514139dfc5160a0aacea1221e8cf6e59de02e25 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | process_scripts.md | 644 | | | 793 | c5dd7c142e831fbdffe1b3f31172306deae656e8 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | ptree.md | 644 | | | 828 | dd39cdedd0959c21e3fa35833df511aeed25c66a -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | push_stack.md | 644 | | | 823 | 9e3a70953c21a8e739ee9512c555c30e8454961b -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | push_venv.md | 644 | | | 668 | 92e98f321272913ede8e24797091fc5690c160c1 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | renv.md | 644 | | | 817 | 25f0546d7e0bd3c9e0d707b196560ae27af40af2 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | script_description.md | 644 | | | 739 | 99e3f54d505bd3b425031c48708db998278e43b9 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | snum.md | 644 | | | 675 | 88992604fbce35a75355deb7f30383b0f762884f -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | sort_2d_array.md | 644 | | | 1334 | 43244058820b8ace10bcf73db9af8a657a211c0d -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | source_util_script.md | 644 | | | 1058 | 80ff184bca07c12d2f62da03f548fcd38ded712a -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | specific_function_help.md | 644 | | | 862 | 54f011d8a2b72d49d7b10814a6722ce577393d42 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | specific_script_help.md | 644 | | | 849 | 831c3666dec9ec3cb9fa31823558058f2bea43cf -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | stack_op.md | 644 | | | 961 | 103209fa1a8a101eeac0254762b3ffd2b65c26fd -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | stringclean.md | 644 | | | 769 | 960fd47bb5dd625bb8a723549b381fd2ad8adbb1 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | strip_space.md | 644 | | | 751 | 0d53ca0cf5bfec2218fbbd287d91cdaf6f91b69a -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | to_upper.md | 644 | | | 655 | eca94d357ea31fa786deb552ce41ff74bbd2cd7a -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | var_type.md | 644 | | | 850 | 6803fbfb4980acf52f5b24d29542fef49ffc9f7b -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | vdsc.md | 644 | | | 641 | bdff21380fa5aa7a2ee8be0c3f38541b58f470d5 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | venvdiff.md | 644 | | | 860 | 43323a50d3cb3b877bace7f3403ac5fe7dcdfa6b -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | vhelp.md | 644 | | | 794 | df903c1ec3fb0553800b5fd6a664e95ede04cc77 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | vnum.md | 644 | | | 634 | 53c0caae539e4fc026255d128f2c4c15b6fa2445 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | vpfx.md | 644 | | | 626 | 4b6b001356c419bc4d3afe4592571bf3a0a43773 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_function_doc.md | 644 | | | 832 | df571bc2b115fea3d60284ffb2b412d35fb4619c -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_page_footer.md | 644 | | | 757 | b27abe4cef36729b7287bd5ccf72cf6743b4ed73 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_script_doc.md | 644 | | | 916 | f9ff9d27829451d0fa1000068631cc01ba0bd924 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_script_function_entry.md | 644 | | | 891 | 0043c2bf03d9dcf64740f910130eea008ec8ca34 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_script_readme_header.md | 644 | | | 950 | 4c85649c4117ccfb43ccea94c7f5c080dddaca99 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_system_readme_entry.md | 644 | | | 882 | 6f7db5d02779496e17865643d4e05e4000e9ee14 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_system_readme_header.md | 644 | | | 783 | bdc2f4792a6c559045f456aef59af718867f5f32 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_table_footer.md | 644 | | | 742 | f0c9b4294cffedc35d4ff6ac964c299c06ba40c2 -f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | zero_pad.md | 644 | | | 1046 | 1c363c76631067e4871f1656dd7e0e11fd56671a -f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | errno.sh.md | 644 | | | 1053 | d5fdab124016daf5ac39b9b2882af4ccf31f55d9 -f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | help_sys.sh.md | 644 | | | 1267 | 2de49186bf548b206d2885dbe3914df93a0b871d -f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | init_env.sh.md | 644 | | | 1760 | 9fce616dcff47b7b37d622d5b4809bcac0c67e4d -f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | util_funcs.sh.md | 644 | | | 1436 | 34a98ae43aed79b34d0afe32f3c8ac370a8b7fb4 -f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | venv_funcs.sh.md | 644 | | | 2831 | 495e6410ca78604e0964b35ac5df2126a14dfb41 -f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | wrapper_funcs.sh.md | 644 | | | 1363 | b48602981d3ac54d8b3af77f4af72c9fd873e7fb +f | docs/shdoc | docs/shdoc | README.md | 644 | | | 1433 | 8a3a0d5ee005931fe2b0349739b082f3f4eaf42e +f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | config_lib_sh.md | 644 | | | 998 | bbb91adf78508ee3314ae856690f821876fc6967 +f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | errno_lib_sh.md | 644 | | | 1300 | a649aec260535250cd443441f81bf053cff877ff +f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | helpsys_lib_sh.md | 644 | | | 2872 | 06427c8da4128daef6c0ec29e2817b9976a3c845 +f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | init_lib_sh.md | 644 | | | 711 | e7966b636bf2b3976b9f9fae9c88e064b46ae9ad +f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | string_lib_sh.md | 644 | | | 1040 | 8c18e77b05fd22cec9a9bcfd10447abdacca57fa +f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | type_lib_sh.md | 644 | | | 783 | cd1fa46e0f562d8dc0eaf90f70fb73accfee15eb +f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | util_lib_sh.md | 644 | | | 1047 | f6f9f6e581d2d344adce12e7d08442ee012e8633 +f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | venv_lib_sh.md | 644 | | | 1907 | e26d2c2dad5dce6480326d68ae9842b0ad4d418a +f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | venvutil_lib_sh.md | 644 | | | 494 | 136ca0debb50d161b40e337a4c9cfab54e67a855 +f | docs/shdoc/bin/shinclude | docs/shdoc/bin/shinclude | wrapper_lib_sh.md | 644 | | | 882 | 1f6cd6d1cb3af0fcb05368e3d6010eea655d9da8 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | __set_venv_vars.md | 644 | | | 651 | 3f0bc7939e517fcc1c835a1e0e543a2396e7ef4d +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | __venv_conda_check.md | 644 | | | 820 | d06e7c0e419c996852213d29be5ebf56058afd35 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | _deprecated.md | 644 | | | 656 | 5e11f64921d64f0c56f1b9a3a18452b4b8d3923a +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | _source_check.md | 644 | | | 315 | 46723a153a301fb2226fd83c5d26eaf245e97317 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | benv.md | 644 | | | 1320 | 3782cdb2a6e26a3ec51ab910c5ce5c3162e4b786 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | cact.md | 644 | | | 789 | 017205307a7de2feb50c250a05c2fcc149ae3c7c +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | ccln.md | 644 | | | 1033 | d72b0b28e464b530ab99b3808ef6478b098d8544 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | check_lib.md | 644 | | | 813 | f16e8b5b9dd0e15bd37382c7aa6d78632ec65719 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | colortext.md | 644 | | | 887 | 1362de2b3d0fdfa547311a62e8c22dfd82aef93e +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | dact.md | 644 | | | 661 | 4aef1b22f31e48f57dee33efdb678c7ba4944a5b +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | denv.md | 644 | | | 769 | a38178774f75772d70025ba8c114c0b88073d3d5 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | do_wrapper.md | 644 | | | 825 | dc71fef3bd98b725c10f6bc5ff09c5ef5f38a1c1 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | docs_base_path.md | 644 | | | 670 | 61671854dd360686fdbc5cb6d9c99e31e81b7bf3 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | errfind.md | 644 | | | 841 | a0a7029d09310ea8ef4834c08094ca65206deefd +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | errno.md | 644 | | | 1060 | 95bd4a1d09c0baf18f09db75656bb360fc7a99c2 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | errno_exit.md | 644 | | | 1270 | 48509b8e9213bb2aca6bdd923bb5bb86c93ea547 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | errno_warn.md | 644 | | | 1268 | d5901293e35ee329d8ca9f6bf6ba6c659b412cf3 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | errval.md | 644 | | | 1065 | 137a069d93916cefa2880ca5ff47a1e7aeaf4845 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | escape_string.md | 644 | | | 770 | 75a92ebd3f9bc794113149d8de021475ce6b0b89 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | expand_variable.md | 644 | | | 741 | 87d4f4abd1266c75089044b0dbc1459d62747556 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | function_description.md | 644 | | | 754 | 8f84038e4fe692fdab430da699e2d09099658e36 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | general_help.md | 644 | | | 683 | 03000e9c9beba5510d0e24b20c5d1c4807ac0ff8 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | generate_markdown.md | 644 | | | 739 | a4cafa3b153ad2a779d41462d25499059d52ac6b +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | get_function_hash.md | 644 | | | 745 | 2dcc08a9283e960cd6f9169bb90804ff24ef88cc +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | get_script_readme_file.md | 644 | | | 829 | 615265b71ba496be4372ab8a7a0d4ce16e408bff +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | get_system_readme_file.md | 644 | | | 785 | aff6b0c39bd7aaf8f8ae35e218ce766ed7d1cf47 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | handle_variable.md | 644 | | | 1256 | 5950cac878b299cb77e71442a89708e17d9c78e3 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | help_functions.md | 644 | | | 867 | 7ab6fac45412e1d78fa4da8fe840c41015489757 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | help_scripts.md | 644 | | | 650 | 41ae2a6912ec28083d85fb4ce8fed49e16d16c6d +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | init_help_system.md | 644 | | | 986 | ce5a30bfbf94bcfe3011725e864fdae4102e40e3 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | lastenv.md | 644 | | | 698 | 5240d7b7b4f46d4e994f46dbe78d70c35390c0e0 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | lenv.md | 644 | | | 1596 | 59ba9fa60f66081dc7e7918932d7082c47d7da02 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | load_pkg_config.md | 644 | | | 1857 | e0dbd1565de5f2219adb436bdb1d2687d7926319 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | log_message.md | 644 | | | 1288 | 17ca928e242f37c3341982e51e7b79ab10905476 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | nenv.md | 644 | | | 1017 | 872dad88360ba803e2c9cf4e1fd9fbabbd146a99 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | next_step.md | 644 | | | 953 | b8b82252d4207cab6efda01a3aed3b644eb9fd08 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | pact.md | 644 | | | 712 | 7f6ef7854e0072b76e67e9b1f3a64a18f09215d0 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | parse_manifest_metadata.md | 644 | | | 754 | 810c7385c635b86281dc38caf62ff33fb3e1bdb0 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | pip.md | 644 | | | 756 | 2c2250c2bebe761184c0adb76a5da70e933f13e7 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | pkg_config_vars.md | 644 | | | 654 | c1f48c9d42426df2fa32aa23525c9166253fb18d +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | pop_stack.md | 644 | | | 780 | a1e3e13ba4b70d5ddab9c392e5391714fdcfada1 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | pop_venv.md | 644 | | | 656 | 21ce73dea9a3ca35f9c8691da715564407514994 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | process_scripts.md | 644 | | | 788 | 5aa06123a791f6affc5ea5ef27bbc60b66430fae +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | ptree.md | 644 | | | 813 | b5bba9575445ce92d1e81f0f5c14ee496fe990a7 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | push_stack.md | 644 | | | 808 | 36baccd095a551c3d6572ea13c72f1888e0e8f82 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | push_venv.md | 644 | | | 653 | 054e57a408e61a9a7f04cf6caf1ba8724b9fb3fe +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | renv.md | 644 | | | 831 | 951960a238d098cfee864e2e80f70bd6d88ae0a7 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | sanitize.md | 644 | | | 873 | 1fc69df92402d5125c09abb5974441c66feccf89 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | script_description.md | 644 | | | 734 | 47e3515c51838b10307d574780f200645a13fce8 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | set_debug.md | 644 | | | 898 | 2eda4c837af98c793bc539bbcd96696fa7365583 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | set_variable.md | 644 | | | 1110 | ccb8a730ef25bd199d32b3161ca3542b837502df +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | snum.md | 644 | | | 660 | 8cc8ae2c01ec80081d417a126169a8b1d568347b +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | sort_2d_array.md | 644 | | | 1319 | b62b63c8de28f0e79024d203dd43206c392de372 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | source_lib.md | 644 | | | 1377 | 29805bbfcf34c7171e2f2dcf187a1d351faf939c +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | specific_function_help.md | 644 | | | 857 | 6309fa4ebd74e54a27b301229803431df9b85942 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | specific_script_help.md | 644 | | | 844 | afaecd7cdaa40110823940ffcda4ed17b07f85e6 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | stack_op.md | 644 | | | 946 | 778ee58a74cfaa6f7c74bf0e9ff01ca9735e0291 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | stringclean.md | 644 | | | 758 | 403613876a69e316869d8674fdab7fcbde215a49 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | strip_space.md | 644 | | | 740 | 4405e2746d51639b836d9cd46e901807fa36d9a1 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | to_upper.md | 644 | | | 644 | f5e65ca005f15cb931f5e2d54caa24091722f42d +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | var_type.md | 644 | | | 829 | b20430ee878658df38c1d2a4267961efd61de83c +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | vdiff.md | 644 | | | 871 | ff19819f0095c869af7513f992f45c3e74c167cc +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | vdsc.md | 644 | | | 626 | b1f8b44e68f8eddb8e133e6a7dad44f0e45db6eb +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | vhelp.md | 644 | | | 789 | 5b86e30b740a0eb7269facbdd42d755d1b4978bf +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | vnum.md | 644 | | | 619 | 94b4fbc47305a0f8a67f4ac241a2bf9773dabc0f +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | vpfx.md | 644 | | | 611 | ff4a8bd972457ed284646b05e7d57ab8350195f1 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | vren.md | 644 | | | 907 | 7165e02e8f83b55948b9d4eded217f7da2e585fb +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_config.md | 644 | | | 795 | 58f36eb8f85e1d6c0e6569975a55dddf06b1f450 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_function_doc.md | 644 | | | 827 | a35ff49e63c6773f48b55cc52486ef7852c6671d +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_page_footer.md | 644 | | | 752 | 636c2711cd85264b1ea4b325a579902042e327b0 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_script_doc.md | 644 | | | 911 | 9ba1f77eda885f28bee1047bdef99b5736f8d907 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_script_function_entry.md | 644 | | | 886 | f6dc73f5d2af0c1943ea2699d35bdb00d03a5008 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_script_readme_header.md | 644 | | | 945 | 8172ce0b029b5ca8fcbb2218b4e836283bb9c5b6 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_system_readme_entry.md | 644 | | | 877 | 62af5b8a7a7f326ab1863c21062bb725c7150597 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_system_readme_header.md | 644 | | | 778 | c846da1bb641a33efd07ab00e4e3a7979ed03b32 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | write_table_footer.md | 644 | | | 737 | 8d9bfad6bd161285ad3aea1f612f634e56edfd69 +f | docs/shdoc/bin/shinclude/functions | docs/shdoc/bin/shinclude/functions | zero_pad.md | 644 | | | 1031 | 52ba6572d724ec7494e31bbe1a65e4b5783d8617 +f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | config_lib.sh.md | 644 | | | 1885 | 91e1ee07ceb9fa46e7387f63548a55ed21f1e1bc +f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | errno_lib.sh.md | 644 | | | 1042 | df1a5972f00a5bd2cfb6ede73ae30fbc59b5dade +f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | helpsys_lib.sh.md | 644 | | | 1248 | b5b2cc060db0d0143e89b85887500586b56ebbc7 +f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | init_lib.sh.md | 644 | | | 1727 | 44686c014cbba50cbc16274be36bb2e054c479e3 +f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | string_lib.sh.md | 644 | | | 856 | 5929bec92073835796040a248152b4eea859b7d9 +f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | type_lib.sh.md | 644 | | | 898 | d275ccff881bf699f9c072d4e214ab56941f84ca +f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | util_lib.sh.md | 644 | | | 1408 | effd7d6a9a0173685d7b90e32bdb9ea55d07ca11 +f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | venv_lib.sh.md | 644 | | | 2900 | 12e723634d10db76239582565d988d359ab79560 +f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | venvutil_lib.sh.md | 644 | | | 2028 | efdc3fd6fe4699b87b863faec3f2b57f0e3c6c3c +f | docs/shdoc/bin/shinclude/scripts | docs/shdoc/bin/shinclude/scripts | wrapper_lib.sh.md | 644 | | | 1322 | 566a7ddd7b4e97600075620bb526e027bfb4c988 +f | modules | modules | conda-install.sh | 644 | | | 1494 | b9d60252174be34023013a06897184db09e68664 l | bin | chunkfile.py | chunkfile | 755 | | | 12 | +l | bin | extract_chat.py | extract-chat | 755 | | | 15 | l | bin | filetree.py | filetree | 755 | | | 11 | l | bin | generate_manifest.sh | generate_manifest | 755 | | | 20 | +l | bin | numpy-comp.sh | numpy-comp | 755 | | | 13 | +l | bin | rename-chat.py | rename-chat | 755 | | | 14 | l | bin | tokencount.py | tokencount | 755 | | | 13 | l | bin | warehouse.sh | recall | 755 | | | 12 | l | bin | warehouse.sh | warehouse | 755 | | | 12 | diff --git a/modules/conda-install.sh b/modules/conda-install.sh index cb9aea70..aaf0c9a1 100644 --- a/modules/conda-install.sh +++ b/modules/conda-install.sh @@ -1,10 +1,10 @@ - #!/bin/sh +#!/usr/bin/env bash # Conda install __CONDA() { cd ${__BUILD_DIR} - [ -e ./tmp ] || mkdir tmp + mkdir tmp cd tmp curl https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh -o miniconda.sh # Do a non-destructive Conda install whcih will preserve existing VENV's diff --git a/pre-commit b/pre-commit new file mode 120000 index 00000000..d8f66974 --- /dev/null +++ b/pre-commit @@ -0,0 +1 @@ +../main/.git/hooks/pre-commit \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index 55a2fc2a..cdaab7ad 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,9 @@ # Required by venvutil rich nltk -tiktoken \ No newline at end of file +tiktoken +mistune +numpy==1.26.* +torch +ftfy +mistune \ No newline at end of file diff --git a/setup.cf b/setup.cf index c19b9803..50c640dc 100644 --- a/setup.cf +++ b/setup.cf @@ -3,16 +3,18 @@ # Define variables prefix=$HOME/local/venvutil exec_prefix=${prefix} -libdir=${exec_prefix}/lib +libdir="${exec_prefix}/lib" includedir=${prefix}/include bindir=${exec_prefix}/bin datadir=${prefix}/share sysconfdir=${prefix}/etc +include_files=("README.md" "LICENSE" "setup.sh" "setup.cf" "manifest.lst" "requirements.txt" "requirements-build.txt") +include_dirs=("bin" "docs" "conf" "modules") # Package metadata Name: venvutil -Description: Virtual Environment Utilities -Version: 0.4.0 +Description: "${TERM} Virtual Environment Utilities" +Version: 20250206-00_R1 Repository: https://github.com/unixwzrd/venvutil License: Apache License, Version 2.0 Support: https://github.com/unixwzrd/venvutil/issues @@ -21,11 +23,11 @@ Contribute: https://www.ko-fi.com/unixwzrd # Dependencies (if any) Requires: python >= 3.10 -Requires: bash >= 4.0 +Requires: bash > 4.0 Requires: Conda >= 22.11 Requires: macOS Requires: Linux -Conflicts: +# Conflicts: # Compiler and linker flags (if applicable) # Cflags: -I${includedir} diff --git a/setup.sh b/setup.sh index d8fb938b..0d0923e9 100755 --- a/setup.sh +++ b/setup.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# venvutil_setup.sh - Setup and configure venvutil. +# setup.sh - Setup and configure venvutil. # # This script installs and configures the venvutil tools and utilities. # It supports installation and removal operations, although the removal and rollback functionality @@ -56,31 +56,6 @@ # Apache License, Version 2.0 # -[ "${DEBUG_SETUP:-""}" = "ON" ] && set -x -set -euo pipefail - -# Initialize script variables -THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[$((${#BASH_SOURCE[@]} -1))]}") -MY_NAME="$(basename "${THIS_SCRIPT}")" -MY_PATH="$(dirname "${THIS_SCRIPT}")" - -[[ "${BASH_VERSINFO[0]}" -lt 4 ]] \ - && echo "($MY_NAME) ERROR: This script requires Bash version 4 or higher." >&2 \ - && exit 75 - -# Default values -PKG_NAME="DEFAULT" -PKG_VERSION="" -INSTALL_BASE="" -INSTALL_CONFIG="$HOME/.${PKG_NAME}" -INSTALL_MANIFEST="manifest.lst" -ACTION="" -VERBOSE=false - -declare -g -A pkg_config_values=() -declare -g -a pkg_config_set_vars=() -declare -g -a pkg_config_desc_vars=() - # Logging function log_message() { local message_level="$1"; shift @@ -95,12 +70,12 @@ log_message() { # Print message to STDERR and log file if [ "$VERBOSE" = true ]; then - echo "($MY_NAME) [$message_level] $message_out" 2>&1 | tee -a "$INSTALL_CONFIG/install.log" >&2 + echo "($__SETUP_NAME) [$message_level] $message_out" 2>&1 | tee -a "$INSTALL_CONFIG/install.log" >&2 return 0 fi # Write message to log file - echo "($MY_NAME) [$message_level] $message_out" >> "$INSTALL_CONFIG/install.log" 2>&1 + echo "($__SETUP_NAME) [$message_level] $message_out" >> "$INSTALL_CONFIG/install.log" 2>&1 } # Function to display help extracted from the script @@ -134,7 +109,7 @@ parse_arguments() { case $opt in d) INSTALL_BASE="$OPTARG" ;; v) VERBOSE=true ;; - h) display_help_and_exit "Usage: $MY_NAME [options] {install|remove|rollback}" ;; + h) display_help_and_exit "Usage: $__SETUP_NAME [options] {install|remove|rollback}" ;; \?) echo "Invalid option -$OPTARG" >&2; exit 1 ;; :) echo "Option -$OPTARG requires an argument." >&2; exit 1 ;; esac @@ -144,16 +119,36 @@ parse_arguments() { # Ensure at least one action is specified ACTION="${1:-}" if [ -z "$ACTION" ]; then - display_help_and_exit "Usage: $MY_NAME [options] {install|remove|rollback}" + display_help_and_exit "Usage: $__SETUP_NAME [options] {install|remove|rollback}" fi return 0 } -# Initialization +get_os_config() { + # Find host OS and architecture + declare -g OS ARCH + OS=$(uname -s) + [ "$OS" == "Darwin" ] && OS="MacOSX" + [ "$OS" == "Linux" ] && OS="Linux" + ARCH=$(uname -m) + ARCH=${ARCH//aarch64/arm64} + return 0 +} + +# Installation Initialization initialization() { - load_pkg_config + get_os_config + + # check to see if function exists for pkg_config_vars + if ! declare -f pkg_config_vars &>/dev/null; then + log_message "ERROR" "pkg_config_vars function not found configuration not loaded." + exit 2 + fi + + pkg_config_vars + load_pkg_config "${__SETUP_BASE}/setup.cf" # Set PKG_NAME early to load config PKG_NAME=${Name:-${PKG_NAME:-"DEFAULT"}} @@ -165,166 +160,27 @@ initialization() { create_pkg_config_dir # Set default manifest path - INSTALL_MANIFEST="$MY_PATH/manifest.lst" + INSTALL_MANIFEST="$__SETUP_BASE/manifest.lst" log_message "INFO" "Configuring $PKG_NAME for initialization..." # Parse manifest metadata - parse_manifest_metadata + parse_manifest_metadata "${INSTALL_MANIFEST}" return 0 } -expand_variables() { - local input="$1" - - # Validate the input: Allow variable references and valid values - if [[ ! "$input" =~ ^[A-Za-z0-9_\$\{\}]+([[:space:]]*[-+*/]?[[:space:]]*[A-Za-z0-9_\$\{\}]+)*$ ]]; then - return 1 - fi - - # Sanitize the input by escaping special characters if necessary - # For example, you might want to escape quotes or backslashes - sanitized_input=$(echo "$input" | sed 's/[&;|<>]/\\&/g') - - # Use eval to expand variables safely, handling both ${var} and $var notation - eval "echo \"$sanitized_input\"" -} - -# Load package configuration from .cf file -load_pkg_config() { - local config_file="$MY_PATH/setup.cf" - if [ ! -f "$config_file" ]; then - echo "ERROR: Configuration file $config_file not found." >&2 - exit 2 - fi - - local key="" - local value="" - while IFS= read -r line || [[ -n "$line" ]]; do - # Trim leading and trailing whitespace - line="$(sed 's/^[[:space:]]*//; s/[[:space:]]*$//' <<< "$line")" - - # Skip comments and blank lines - [[ -z "$line" ]] && continue - [[ "$line" =~ ^# ]] && continue - - # Check for Key=Value pattern - if [[ "$line" =~ ^([A-Za-z_]+)=(.*)$ ]]; then - key="${BASH_REMATCH[1]}" - if ! value=$(expand_variables "${BASH_REMATCH[2]}"); then - echo "($MY_NAME) WARNING: Invalid variable assignment: '$line' - skipping." >&2 - continue - fi - # Set shell variable - declare -g "$key"="$value" - # Append or initialize array entry - if [[ -z "${pkg_config_values[$key]:-}" ]]; then - pkg_config_values[$key]="$value" - pkg_config_set_vars+=("$key") - else - pkg_config_values[$key]+=$'\n'"$value" - fi - continue - fi - - # Check for Key: Value pattern - if [[ "$line" =~ ^([A-Za-z_]+):[[:space:]]*(.*)$ ]]; then - key="${BASH_REMATCH[1]}" - value="${BASH_REMATCH[2]}" - # Set shell variable - declare -g "$key"="$value" - # Append or initialize array entry - if [[ -z "${pkg_config_values[$key]:-}" ]]; then - pkg_config_values[$key]="$value" - pkg_config_desc_vars+=("$key") - else - pkg_config_values[$key]+=$'\n'"$value" - fi - continue - fi - - # If a line doesn't match either pattern, assume it’s an additional value for the last key - # If no previous key is known, just ignore. - if [[ -n "$key" ]]; then - # Treat as another line for the current key - declare -g "$key"="${!key:-}\n${line}" - pkg_config_values[$key]+=$'\n'"$line" - fi - done < "$config_file" -} - # Create package configuration directory create_pkg_config_dir() { if [ ! -d "${INSTALL_CONFIG}" ]; then - mkdir -p "${INSTALL_CONFIG}" - mkdir -p "$INSTALL_CONFIG/log" "$INSTALL_CONFIG/freeze" + mkdir -p "$INSTALL_CONFIG/log" "$INSTALL_CONFIG/freeze" log_message "INFO" "Created ${INSTALL_CONFIG} directories..." fi # Create application configuration directory return 0 } -# Parse manifest metadata -parse_manifest_metadata() { - if [ ! -f "$INSTALL_MANIFEST" ]; then - echo "ERROR: Manifest file $INSTALL_MANIFEST not found." >&2 - exit 2 - fi - echo "Parsing manifest metadata..." >&2 - local manifest_file="$INSTALL_MANIFEST" - while IFS='| ' read -r line || [[ -n "$line" ]]; do - [[ "$line" =~ ^#.*$ ]] && continue # Skip comments - [[ -z "$line" ]] && break # Stop at first blank line (end of metadata) - if [[ "$line" =~ ^[A-Za-z_]+=.*$ ]]; then - eval "$line" - fi - done < "$manifest_file" - return 0 -} - -install_conda() { - log_message "INFO" "Installing conda..." - # Check if conda is already installed - if command -v conda &> /dev/null; then - log_message "INFO" "Conda is already installed. Skipping installation." - return 0 - fi - - # Find host OS and architecture - local OS ARCH - OS=$(uname -s) - [ "$OS" == "Darwin" ] && OS="MacOSX" - [ "$OS" == "Linux" ] && OS="Linux" - ARCH=$(uname -m) - ARCH=${ARCH//aarch64/arm64} - - # Download and install conda - log_message "INFO" "Downloading and installing Conda..." - local INSTALLER_URL="https://repo.anaconda.com/miniconda/Miniconda3-latest-${OS}-${ARCH}.sh" - curl -k -O "$INSTALLER_URL" - # do a non-destructive install - bash "Miniconda3-latest-${OS}-${ARCH}.sh" -b -u - rm "Miniconda3-latest-${OS}-${ARCH}.sh" - # Activate the Conda installation - # shellcheck disable=SC1091 - source "${HOME}/miniconda3/bin/activate" - # Initialize conda for our shell - conda init "$(basename "${SHELL}")" - log_message "INFO" "Conda installed successfully, checking for updates..." - conda update -n base -c defaults conda -y - # Because Red Hat Enterprise Linux defines, sets it, but doesn't export it BASHSOURCED... - # Prevents /etc/bashrc from being sourced again on Red Hat Enterprise Linux. - export BASHSOURCED=Y - # So we don't recurse. - export CONDA_INSTALL_COMPLETE=Y - SHELL=$(which "$(basename "$SHELL")") - # Wheeeeee!!!!!! - exec "$SHELL" -l -c "${THIS_SCRIPT} ${ACTION}" - return 0 -} - # Write package information write_pkg_info() { install_log="${INSTALL_CONFIG}/${PKG_NAME}.pc" @@ -348,13 +204,13 @@ write_pkg_info() { check_deps() { # Check for Bash version 4+ if [ "${BASH_VERSINFO[0]}" -lt 4 ]; then - log_message "ERROR" "$MY_NAME requires Bash version 4 or higher." + log_message "ERROR" "$__SETUP_NAME requires Bash version 4 or higher." exit 75 fi # Check Operating System (Linux or macOS) if [ "$(uname -s)" != "Darwin" ] && [ "$(uname -s)" != "Linux" ]; then - log_message "ERROR" "$MY_NAME is only supported on macOS and Linux." + log_message "ERROR" "$__SETUP_NAME is only supported on macOS and Linux." exit 75 fi @@ -378,17 +234,11 @@ check_deps() { # Pre-installation tasks pre_install() { # Check if pre-installation tasks have already been completed - # Stop recursion before it starts, this is re-entrant. - if [ "${CONDA_INSTALL_COMPLETE:-""}" == "Y" ]; then - return 0 - fi log_message "INFO" "Pre-installation tasks..." # Custom pre-install tasks can be added here check_deps # This may be used for verification or rollback/removal later. write_pkg_info - install_conda - unset CONDA_INSTALL_COMPLETE } install_assets() { @@ -410,7 +260,7 @@ install_assets() { IFS=$'| ' read -r asset_type destination source name permissions owner group size checksum <<< "$line" destination="${INSTALL_BASE}/${destination}" - source_path="${MY_PATH}/${source}/${name}" + source_path="${__SETUP_BASE}/${source}/${name}" dest_path="${destination}/${name}" mkdir -p "$destination" @@ -427,18 +277,24 @@ install_assets() { chmod "$permissions" "$dest_path" ;; h) # Create hard link + # shellcheck disable=SC2164 cd "$destination" ln "$source" "$name" + # shellcheck disable=SC2164 cd - > /dev/null ;; l) # Create symbolic link + # shellcheck disable=SC2164 cd "$destination" ln -sf "$source" "$name" + # shellcheck disable=SC2164 cd - > /dev/null ;; c) # Remove the asset + # shellcheck disable=SC2164 cd "$destination" rm -rf "$name" + # shellcheck disable=SC2164 cd - > /dev/null ;; *) @@ -452,10 +308,12 @@ post_install_user_message() { log_message "INFO" "Provide user instructions..." # Custom post-install message can be added here cat <<_EOT_ + The package $PKG_NAME has been installed to $INSTALL_BASE. To use the package, the following line was added to your .bashrc file: if [[ ! "\$PATH" =~ "$INSTALL_BASE/bin:" ]]; then export PATH="$INSTALL_BASE/bin:\$PATH"; fi + if [[ -f "${INSTALL_BASE}/bin/shinclude/venvutil_lib.sh" ]]; then source \"${INSTALL_BASE}/bin/shinclude/venvutil_lib.sh\"; fi If you wish to use it in the current shell, run the following command: @@ -464,19 +322,19 @@ post_install_user_message() { or exit the terminal and start a new one. To verify the installation files for correct location and file integrity run the following command: - $MY_NAME verify (not implemented yet) + $__SETUP_NAME verify (not implemented yet) If you wish to uninstall the packages associated with $PKG_NAME, run the following command: - $MY_NAME uninstall (not implemented yet) + $__SETUP_NAME uninstall (not implemented yet) This will only remove the files associated with the package, not the Conda installation, its installed packages or any other dependencies. If you wish to uninstall everything associated with the package, run the following command: - $MY_NAME remove_all (not implemented yet) + $__SETUP_NAME remove_all (not implemented yet) The documentation may be found in the $INSTALL_BASE/README.md file. Please contact the package maintainers for any issues or feature requests or file them on @@ -500,6 +358,77 @@ write_pkg_config() { return 0 } +install_python_packages() { + log_message "INFO" "Installing Python packages..." + log_message "INFO" "Creating virtual environment..." + benv venvutil + log_message "INFO" "Installing NLTK data..." + pip install -r "$__SETUP_BASE/requirements.txt" 2>&1 | tee -a "$INSTALL_CONFIG/install.log" >&2 + python <<_EOT_ +import nltk +nltk.download('punkt') +nltk.download('stopwords') +_EOT_ + log_message "INFO" "NLTK data installed successfully." + return 0 +} + +restart_shell() { + log_message "INFO" "Restarting shell..." + # Because Red Hat Enterprise Linux defines, sets it, but doesn't export it BASHSOURCED... + # Prevents /etc/bashrc from being sourced again on Red Hat Enterprise Linux. + export BASHSOURCED=Y + # So we don't recurse. + export CONDA_INSTALL_COMPLETE=Y + SHELL=$(which "$(basename "$SHELL")") + # Wheeeeee!!!!!! + exec "$SHELL" -l -c "${__SETUP_BASE}/${__SETUP_NAME} ${ACTION}" + return 0 +} + +run_conda_installer() { + log_message "INFO" "Running conda installer..." + bash "Miniconda3-latest-${OS}-${ARCH}.sh" -b -u + rm "Miniconda3-latest-${OS}-${ARCH}.sh" + # Activate the Conda installation + # shellcheck disable=SC1091 + source "${HOME}/miniconda3/bin/activate" + + # Initialize conda for our shell + conda init "$(basename "${SHELL}")" + log_message "INFO" "Conda installed successfully, checking for updates..." + conda update -n base -c defaults conda -y + + return 0 +} + +get_conda_installer() { + log_message "INFO" "Getting conda installer..." + # Find host OS and architecture + local OS ARCH + OS=$(uname -s) + [ "$OS" == "Darwin" ] && OS="MacOSX" + [ "$OS" == "Linux" ] && OS="Linux" + ARCH=$(uname -m) + ARCH=${ARCH//aarch64/arm64} + local INSTALLER_URL="https://repo.anaconda.com/miniconda/Miniconda3-latest-${OS}-${ARCH}.sh" + curl -k -O "$INSTALLER_URL" + return 0 +} + +install_conda() { + # Stop recursion before it starts, this is re-entrant. + if [ "${CONDA_INSTALL_COMPLETE:-""}" == "Y" ]; then + unset CONDA_INSTALL_COMPLETE + return 0 + fi + log_message "INFO" "Installing conda..." + get_conda_installer + run_conda_installer + restart_shell + return 0 +} + # Update .bashrc update_bashrc() { log_message "INFO" "Updating .bashrc for package $PKG_NAME in PATH..." @@ -507,33 +436,25 @@ update_bashrc() { # Expressions don't expand in single quotes, use double quotes for that. # shellcheck disable=SC2016 local path_line="if [[ ! \"\$PATH\" =~ \"$INSTALL_BASE/bin:\" ]]; then export PATH=\"$INSTALL_BASE/bin:\$PATH\"; fi" + local source_line="if [[ -f "${INSTALL_BASE}/bin/shinclude/venvutil_lib.sh" ]]; then source \"${INSTALL_BASE}/bin/shinclude/venvutil_lib.sh\"; fi" - if ! grep -Fxq "$path_line" "$bashrc"; then - echo "$path_line" >> "$bashrc" + for line in "${path_line}" "${source_line}"; do + if ! grep -Fxq "$line" "$bashrc"; then + echo "$line" >> "$bashrc" + fi log_message "INFO" "Updated $bashrc added package $PKG_NAME bin directory." - fi + log_message "INFO" "Updated $bashrc added package $PKG_NAME initialization." + done return 0 } -install_python_packages() { - log_message "INFO" "Installing NLTK data..." - pip install -r "$MY_PATH/requirements.txt" 2>&1 | tee -a "$INSTALL_CONFIG/install.log" >&2 - python <<_EOT_ -import nltk -nltk.download('punkt') -nltk.download('stopwords') -_EOT_ - log_message "INFO" "NLTK data installed successfully." -} - post_install() { log_message "INFO" "Post-installation tasks..." - install_python_packages - # Example: Update .bashrc if necessary update_bashrc + install_conda + install_python_packages write_pkg_config post_install_user_message - log_message "INFO" "Installation for $PKG_NAME complete." return 0 } @@ -543,6 +464,7 @@ install() { pre_install install_assets post_install + log_message "INFO" "Installation for $PKG_NAME complete." return 0 } @@ -550,6 +472,7 @@ install() { pre_remove() { log_message "INFO" "Pre-removal tasks..." # Custom pre-remove tasks can be added here + return 0 } # Removal function @@ -588,14 +511,7 @@ remove_assets() { esac done - post_remove -} - -# Post-removal tasks -post_remove() { - log_message "INFO" "Post-removal tasks..." - # Example: Remove package bin directory from .bashrc - remove_bashrc_entries + return 0 } # Remove entries from .bashrc @@ -604,18 +520,22 @@ remove_bashrc_entries() { # Expressions don't expand in single quotes, use double quotes for that. # shellcheck disable=SC2016 local path_line="if [[ ! \"\$PATH\" =~ \"$INSTALL_BASE/bin:\" ]]; then export PATH=\"$INSTALL_BASE/bin:\$PATH\"; fi" + local source_line="if [[ -f "${INSTALL_BASE}/bin/shinclude/venvutil_lib.sh" ]]; then source \"${INSTALL_BASE}/bin/shinclude/venvutil_lib.sh\"; fi" - if grep -Fxq "$path_line" "$bashrc"; then - sed -i.bak "/$path_line/d" "$bashrc" - log_message "INFO" "Removed package bin directory from $bashrc." - fi + for line in "${path_line}" "${source_line}"; do + if grep -Fxq "$line" "$bashrc"; then + sed -i.bak "/$line/d" "$bashrc" + log_message "INFO" "Removed package bin directory from $bashrc." + fi + done + return 0 } -# Rollback function -rollback() { - log_message "CRITICAL" "Rollback initiated..." - # Implement rollback logic based on actions logged during installation - # For example, read a log file and undo actions +# Post-removal tasks +post_remove() { + log_message "INFO" "Post-removal tasks..." + # Example: Remove package bin directory from .bashrc + remove_bashrc_entries } remove() { @@ -623,10 +543,18 @@ remove() { pre_remove remove_assets post_remove + log_message "INFO" "Package: $PKG_NAME removed." return 0 } +# Rollback function +rollback() { + log_message "CRITICAL" "Rollback initiated..." + # Implement rollback logic based on actions logged during installation + # For example, read a log file and undo actions +} + verify() { log_message "INFO" "Verifying package: $PKG_NAME tasks..." # Implement verification logic here @@ -656,9 +584,62 @@ main() { ;; *) echo "Invalid action: $ACTION" - display_help_and_exit "Usage: $MY_NAME [options] {install|remove|rollback}" + display_help_and_exit "Usage: $__SETUP_NAME [options] {install|remove|rollback}" ;; esac } +## Initialization +[ "${DEBUG_SETUP:-""}" == "ON" ] && set -x +set -uo pipefail + +[[ "${BASH_VERSINFO[0]}" -lt 4 ]] \ + && echo "($__SETUP_NAME) ERROR: This script requires Bash version 4 or higher." >&2 \ + && exit 75 + +# Determine the real path of the script +[ -L "${BASH_SOURCE[0]}" ] && THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[0]}") || THIS_SCRIPT="${BASH_SOURCE[0]}" +# Extract script name, directory, and arguments +# Initialize script variables +THIS_SCRIPT=$(readlink -f "${BASH_SOURCE[$((${#BASH_SOURCE[@]} -1))]}") +__SETUP_NAME="$(basename "${THIS_SCRIPT}")" +__SETUP_BASE="$(dirname "${THIS_SCRIPT}")" +__SETUP_BIN="${__SETUP_BASE}/bin" +__SETUP_INCLUDE="${__SETUP_BIN}/shinclude" + +# Default values +PKG_NAME="DEFAULT" +PKG_VERSION="" +INSTALL_BASE="" +INSTALL_CONFIG="$HOME/.${PKG_NAME}" +INSTALL_MANIFEST="manifest.lst" +ACTION="" +VERBOSE=false + +declare -g -A pkg_config_values=() +declare -g -a pkg_config_set_vars=() +declare -g -a pkg_config_desc_vars=() + +SH_LIB="${SH_LIB:-""}" +for try in "${SH_LIB}" "$(dirname "${THIS_SCRIPT}")/shinclude" "${__SETUP_INCLUDE}" "${HOME}/bin/shinclude"; do + [ -f "${try}/init_lib.sh" ] && { SH_LIB="${try}"; break; } +done +[ -z "${SH_LIB}" ] && { + cat<<_EOT_ >&2 +ERROR ($__SETUP_NAME): Could not locate \`init_lib.sh\` file. +ERROR ($__SETUP_NAME): Please set install init_lib.sh which came with this repository in one of + the following locations: + - $(dirname "${THIS_SCRIPT}")/bin/shinclude + - $HOME/shinclude + - $HOME/bin/shinclude + or set the environment variable SH_LIB to the directory containing init_lib.sh + +_EOT_ + exit 2 # (ENOENT: 2): No such file or directory +} + +echo "INFO ($__SETUP_NAME): Using SH_LIB directory - ${SH_LIB}" >&2 +# shellcheck source=/dev/null +source "${SH_LIB}/venvutil_lib.sh" + main "$@" diff --git a/transcript.md b/transcript.md deleted file mode 100644 index ec0b6304..00000000 --- a/transcript.md +++ /dev/null @@ -1,411 +0,0 @@ -# Why Do Narcissistic Parents RUIN Sibling Bonds? - -our parents leave us too early our -spouse and our children come along too -late our siblings are the only ones who -are there for us our entire ride I love -this quote because our siblings are -supposed to be there always we expect to -grow old together possibly raise our -children together but what happens when -sibling relationships have been tampered -with is it possible that some parents -purposefully destroy their children's -sibling relationships I'm here to say -yes unfortunately this happens and it -happens way too often it's possible my -parent did this to me and today we're -going to talk about it let's dive into -the dark very common topic of sibling -alienation hi I'm Maddie and welcome -back or welcome to my project here I -talk about all things related to a -certain form of severe child -psychological currently known as -parental alienation I went through this -starting around the age of or nine when -my parents were still married towards -the end of their marriage this type of -abuse often happens during a custody -dispute or a divorce and it really -involves one angry parent manipulating -their child to reject their other loving -parent without true justification this -is a healthy loving child bond that is -purposefully destroyed because this -parent is angry and wants to hurt their -ex wants to isolate and control their -child it's very devastating form of that -leaves long lasting effects and that's -why I speak out about it on my project I -usually talk about parental alienation -but did you know that this type of -psychological abuse can take many -different forms not only do we have -parental alienation there is also -sibling alienation extended family -alienation grandparent alienation for -example there's friend alienation -communal alienation but today we're -going to talk about sibling alienation -this is close to home for me but I do -not represent all alienated kids or -speak for all alienated kids I'm also -not an expert I'm just speaking for my -own EXP experiences first let's define -sibling alienation sibling alienation -involves a third party often apparent -manipulating one or both of their -children to reject each other -essentially it's an extreme form of -triangulation when why and how does -sibling alienation occur well sibling -alienation often occurs when a parent -may feel threatened angry insecure or -fearful and they really want to seek -control and power this path illogical -parent will seek control and power by -controlling their children's -relationships with each other this -parent will isolate the siblings which -will weaken them and keep them from -talking about issues in the family for -example this parent will also act -sometimes out of jealousy they can't -stand their children being closer to -each other than they are with them how -will a parent pit siblings against each -other well they're are very ious -different mechanisms alienation tactics -parent will use let's get into a few of -them number one bad mouthing this can be -very covert or this can be more overt -for example a covert way of mad bad -mouthing one sibling to another may be -have you ever thought that your sister -might you know just be really judgmental -and mean or do you think that your -brother has ever lied to you something -moreover might be crying and saying your -sister your brother is just so hurtful -in front of their other sibling which is -going to make that sibling naturally -feel bad for their parent and resentful -or confused at their sibling they might -even lash out at their sibling for -hurting their parent this is very common -to me I actually lived most most of my -life being the main target of my -parents' alienation so I was of course -alienated to different degrees from all -of my siblings I have four younger -brothers I was also the only child -targeted with the parental alienation so -my brothers always had close and happy -loving relationships with my dad -basically there's one of two ways that -parents can alienate their child they -can pit one sibling against the other or -they can pit both siblings against each -other I hope that makes sense I want to -be really clear to lose one's sibling -for no justifiable reason while being -blamed and made to believe it's your -fault is a type of pain I wouldn't wish -on anyone of course from family to -family sibling alienation will often -differ it might even change over time -according to the pathological parents -whims and according to the role that -each of the child are playing in the -family at that moment in time I've -noticed in my family and many others -that it seems whatever role the child -has been given by their pathological -parent where whether that's the favored -child the scapegoated child The Lost -Child the Forgotten child you know -whatever it may be the role that child -is meant to play is going to be -perceived by the siblings as well I'm -going to use my family as an example -here so I was the scapegoat a child in -my family the black sheep could do -nothing right the mess up of the family -you know blamed for everything and all -the problems of the family and what I -noticed growing up from a young age -after the divorce really is that all of -my brothers to various degrees took on -my mom's mentality about me as far as -the scapegoating went so they all began -to slowly adopt my pathological parents -view of who I am which is a very skewed -View and it's it's not true they began -to treat me like the scapegoated child -in the family in addition to my mom -treating me like the scapegoated child -in the family and this does extend to -the favorite a child in my family too he -was he was also treated very well by the -other siblings in contrast so I noticed -that in these family systems the -siblings seem to adopt the perceptions -and the attitudes of that pathological -parent towards the other siblings it's -like the pathological parent sets the -tone for the family here's how you treat -this kid here's how you treat this kid -and everybody follows suit it's pretty -sick it often feels like a lose lose -situation being in one of these families -in a typical sibling relationship what -you put in is generally what you get out -most people can appreciate the value of -being having a close connected -relationship with one sibling with -sibling alienation this is not a typical -sibling relationship ship because there -is an interference by a third party -similar to how a targeted parent might -feel like no matter what they do what -they say any amount of effort they put -in no matter what their child keeps -rejecting them so too the targeted -sibling can never win no matter what -they're stuck in this role they cannot -escape the attitudes of their family the -perception of their siblings -no amount of explaining or trying to -show them who you actually are will -change their minds no amount of trying -to show your sibling that you love them -and want a relationship outside of your -parents -influence that will get your sibling to -understand what your parent is doing -they can't see the manipulation usually -and they fail to understand how a parent -subtly pitting their children against -one another is wrong is cruel is abusive -some siblings May grow up and mature and -develop maybe have children of their own -or have other life circumstances that -help them see the dysfunction in the -family and things might improve with -sibling relationships down the line some -families may have a single scapegoated -child for example that was me and my -family system my parent my entire life -pit my siblings against me there was -nothing I could do to stop this no -matter how many times I tried talking to -that parent and explaining them how -damaging it was for me in my -relationship -to talk badly about me to cry about how -I was hurting them and everything like -that no amount of begging mom please -stop talking bad about me and my -brothers I used to beg it's affecting my -relationships please stop crying about -how much you're hurt by me um she didn't -care or maybe she cared too much I don't -know but she couldn't stand other people -loving me more than I loved her she -couldn't stand anyone else being closer -to me than she was she couldn't stand -anyone else loving me that's why her -love always felt like possession it felt -possessive she had to own me she could -only be the person I loved but in -reality there's enough love to go around -I love a lot of people in my life and I -think a lot of people love me so and I -love watching my family members and -friends and everyone who loves my kids -love on my kids so I don't understand -this but that's how it was other -families might have more than one -scapegoat or the scapegoat might revolve -it might change according to the parents -feelings and whims and emotions -regardless life for the scape coded -sibling often is extremely isolating -painful and full of self-doubt and low -self-esteem especially with so many -different family members bullying you or -buying into this false narrative about -who you are perhaps even spreading lies -or false information or gross -exaggerations that are spun in a way to -destroy your reputation it can be -difficult to maintain any type of -self-worth whatsoever in that environment -imagine your parent siblings and -potentially most of your extended family -members think you're a problem think -you're messed up something is wrong with -you you're not good enough and they -don't accept you they don't consider you -one of them they don't treat you like -part of the family maybe deep down you -too don't feel like you're part of the -family maybe deep down you too don't -feel like part of them feeling like an -outsider in one's own family of origin -has got to be one of the most painful -experiences that I've been through I -can't think of many situations that are -more Lonesome than that out of any group -on this planet you would imagine that -you'd feel most safe and secure with the -people who have raised you what is the -upside here so what what now how do we -survive sibling alienation how do we -stop sibling alienation what lessons can -be learned I won't sugarcoat it this -really sucks it's painful heartbreaking -really a lot of people have been through -parental alienation and sibling -alienation and sometimes siblings mature -and realize what you already have -realized and there is hope but as for -the waiting process the back and the -forth in the meantime the being rejected -constantly here's my advice it probably -applies to targeted parents too first -you can only control your thoughts your -emotions -your perceptions not your sibling just -because your sibling isn't present in -your life now doesn't mean that they're -gone forever they've been under the -destructive influence of Mind Control I -think this gives me a lot more empathy -for my siblings I know it's so difficult -to see what's true what's real what's -not and the manipulation is creates this -Dark Cloud where you can't see straight -so I have lots of empathy for my -brothers even when they do reject me -remember that your sibling wants to love -you deep down they do love you and they -want to express that love but they've -been forced to choose one good thing -that I like to tell myself is what they -believe about me is not true what they -believe about me is not true I have two -other pieces of advice while going -through sibling alienation the second piece -of advice in addition to controlling -your thoughts and perceptions and -emotions is to ask yourself the -following question would you want to be -the person you H you would have to be in -order to to fit in and be accepted by -your family for me the answer was not -just no it was heck no no way no thank -you I'd rather die than be the person -I'd have to be to fit in to my family of -origin why because that person would not -be me I wouldn't be true to myself if I -were molding myself into the person I -had to be to fit into my family been -there done that tried that 20 years -never worked never will so if you were -put in a scapegoat role growing up -please know that means that there is -something fierce inside of you that sets -you apart and that fierceness inside of -you threatens the system that's why -you've been given the scapegoated role -to begin with and that's beautiful I -think my final piece of device as the -targeted sibling you might have also -been the scapegoat a child but you have -the best chance at ending the cycle do -you realize how amazing that is a lot of -your siblings your alienated siblings -they might go on to repeat the cycle -themselves or find themselves alienated -from their own kids find themselves in a -relationship with a toxic person because -that's all they know that's the road map -of love -relationships and you have the chance -now to change things for not only your -children but your children's children -your great grandchildren the rest of the -human race to come from your family of -origin not only have you EXP experience -the worst of the abuse opening your eyes -to the dysfunction but there is also -something in you that rejects the -dysfunction no more many people maybe -even most people will grow up to repeat -their parents mistakes and behaviors -some may identify that something is off -with their family they may even identify -that it's but they might also fail to -change anything moving forward or stand -up to Injustice the minority of people -both identify and reject the dysfunction -in their families of origin this is why -ending the generational patterns the -generational behaviors and Trauma is -often such a lonely process because so -few people have the courage to not only -identify it and admit to themselves that -their parent has mistreated them but -also say I'm not okay with this and set -boundaries not many people have the -strength to do that and remember if you -feel guilty about putting up no contact -with your parent who is actually and -neglectful to say that's okay to accept -their mistreatment of you is actually -enabling your parent to continue their -bad behavior in a lot of ways it's a -loving act of kindness to tell your -parent I'm not going to allow you in my -life if you treat me and the people I -love this way it's an act of kindness -because you're you're not enabling them -to be a mean and and hurtful person I -hope that makes sense if you're watching -this video and you could relate to -anything that I've said please like the -video and subscribe to my channel make -sure you have the Bell notification -turned on and leave a comment down below -what do you think of all of this have -you been an alienated sibling have you -been a triangulated sibling did you find -this insightful or helpful let me know -in the comments if you'd like to hear -more about other lesser known forms of -Parental alienation like grandparent -alienation or extended family alienation -I can personally attest to quite -honestly every form of alienation in -one's personal relationship so I'd be -happy to discuss this more I'm laughing -I think as a coping mechanism if you've -been alienated from one of your parents -it's pretty common you've been alienated -from your siblings and others as well -and I'm so sorry you've been through -this but there is life on the other side -of alienation I promise you if you -haven't reached out to that targeted -parent yet that parent you lost time -with that parent you believed negative -things about that you were scared of and -angry at please consider reaching out to -them when I reached out to my dad and -started to repair the relationship with -him that's when my healing Journey -really began and I'm so grateful to have -him in my life now thank you so much for -watching and I'll see you next time bye \ No newline at end of file