diff --git a/.github/workflows/labeler.yaml b/.github/workflows/labeler.yaml index 373948c..b86c085 100644 --- a/.github/workflows/labeler.yaml +++ b/.github/workflows/labeler.yaml @@ -7,6 +7,9 @@ jobs: permissions: contents: read pull-requests: write + issues: write runs-on: ubuntu-latest steps: - uses: actions/labeler@v5.0.0 + with: + configuration-path: .github/labeler.yml diff --git a/README.es.md b/README.es.md deleted file mode 100644 index 105417f..0000000 --- a/README.es.md +++ /dev/null @@ -1,502 +0,0 @@ -# 🚀 STRUCT: Generador Automático de Estructuras de Proyectos - -[![en](https://img.shields.io/badge/lang-en-red.svg)](https://github.com/httpdss/struct/blob/master/README.md) [![es](https://img.shields.io/badge/lang-es-yellow.svg)](https://github.com/httpdss/struct/blob/master/README.es.md) - -![Banner de Struct](extras/banner.png) - -> [!WARNING] -> Este proyecto aún está en desarrollo y puede contener errores. Úsalo bajo tu propio riesgo. - -## 📄 Tabla de Contenidos - -- [Introducción](#-introducción) -- [Características](#-características) -- [Instalación](#️-instalación) - - [Usando pip](#usando-pip) - - [Desde el código fuente](#desde-el-código-fuente) - - [Usando Docker](#usando-docker) -- [Inicio Rápido](#-inicio-rápido) -- [Uso](#-uso) -- [Configuración YAML](#-configuración-yaml) -- [Esquema YAML](#-esquema-yaml) -- [Desarrollo](#-desarrollo) -- [Licencia](#-licencia) -- [Financiamiento](#-financiamiento) -- [Contribuyendo](#-contribuyendo) -- [Agradecimientos](#-agradecimientos) -- [Problemas Conocidos](#-problemas-conocidos) - -## 📦 Introducción - -STRUCT es un script potente y flexible diseñado para automatizar la creación de estructuras de proyectos basadas en configuraciones YAML. Admite variables de plantilla, permisos de archivos personalizados, obtención de contenido remoto y múltiples estrategias de manejo de archivos para optimizar tu proceso de configuración de desarrollo. - -Está dirigido a desarrolladores, ingenieros DevOps y cualquier persona que quiera automatizar la creación de estructuras de proyectos. Puede usarse para generar código de plantilla, archivos de configuración, documentación y más. - -## ✨ Características - -- **Configuración YAML**: Define la estructura de tu proyecto en un simple archivo YAML. -- **Variables de Plantilla**: Usa marcadores de posición en tu configuración y reemplázalos con valores reales en tiempo de ejecución. También admite filtros personalizados de Jinja2 y modo interactivo para completar las variables. -- **Permisos de Archivos Personalizados**: Establece permisos personalizados para tus archivos directamente desde la configuración YAML. -- **Obtención de Contenido Remoto**: Incluye contenido de archivos remotos especificando sus URLs. -- **Estrategias de Manejo de Archivos**: Elige entre múltiples estrategias (sobrescribir, omitir, añadir, renombrar, respaldar) para gestionar archivos existentes. -- **Ejecutar en Seco**: Previsualiza las acciones sin hacer cambios en tu sistema de archivos. -- **Validación de Configuración**: Asegura que tu configuración YAML es válida antes de ejecutar el script. -- **Registro Detallado**: Obtén registros detallados de las acciones del script para una fácil depuración y monitoreo. - -## 🛠️ Instalación - -### Usando pip - -Puedes instalar STRUCT usando pip: - -```sh -pip install git+https://github.com/httpdss/struct.git -``` - -### Desde el código fuente - -Alternativamente, puedes clonar el repositorio e instalarlo localmente. Consulta la sección [Desarrollo](#-desarrollo) para más detalles. - -### Usando Docker - -Puedes usar la imagen de Docker para ejecutar el script sin instalarlo en tu sistema. Consulta la sección [Inicio Rápido](#-inicio-rápido) para más detalles. - -## 🐳 Inicio Rápido - -### Inicio Rápido Usando Docker - -1. Crea un archivo de configuración YAML para la estructura de tu proyecto. Consulta una configuración de ejemplo [aquí](./example/structure.yaml). -2. Ejecuta el siguiente comando para generar la estructura del proyecto: - -```sh -docker run \ - -v $(pwd):/workdir \ - -u $(id -u):$(id -g) \ - ghcr.io/httpdss/struct:main generate \ - /workdir/example/structure.yaml \ - /workdir/example_output -``` - -### Inicio Rápido Usando Docker Alpine - -Para pruebas, puedes ejecutar un contenedor Docker de Alpine e instalar el script dentro de él: - -```sh -docker run -it --entrypoint="" python:3.10-alpine sh -l -``` - -Dentro del contenedor: - -```sh -apk add python-pip git vim -pip install git+https://github.com/httpdss/struct.git -mkdir example -cd example/ -touch structure.yaml -vim structure.yaml # copia el contenido de la carpeta de ejemplo -struct structure.yaml . -``` - -## 📝 Uso - -Ejecuta el script con el siguiente comando usando uno de los siguientes subcomandos: - -- `generate`: Genera la estructura del proyecto basada en la configuración YAML. -- `generate-schema`: Genera esquema JSON para las plantillas de estructura disponibles. -- `validate`: Valida la configuración YAML para asegurarte de que sea válida. -- `info`: Muestra información sobre el script y sus dependencias. -- `list`: Lista las estructuras disponibles. - -Para más información, ejecuta el script con la opción `-h` o `--help` (esto también está disponible para cada subcomando): - -```sh -struct -h -``` - -### Ejemplo Simple - -```sh -struct generate terraform-module ./mi-modulo-terraform -``` - -### Ejemplo Más Completo - -```sh -struct generate \ - --log=DEBUG \ - --dry-run \ - --vars="project_name=MiProyecto,author_name=JuanPerez" \ - --backup=/ruta/al/respaldo \ - --file-strategy=rename \ - --log-file=/ruta/al/archivo_de_registro.log \ - terraform-module \ - ./mi-modulo-terraform - -``` - -### Comando Generate Schema - -El comando `generate-schema` crea definiciones de esquema JSON para las plantillas de estructura disponibles, facilitando que las herramientas e IDEs proporcionen autocompletado y validación. - -#### Uso Básico - -```sh -# Generar esquema a stdout -struct generate-schema - -# Generar esquema con ruta de estructuras personalizada -struct generate-schema -s /ruta/a/estructuras/personalizadas - -# Guardar esquema en archivo -struct generate-schema -o schema.json - -# Combinar ruta personalizada y archivo de salida -struct generate-schema -s /ruta/a/estructuras/personalizadas -o schema.json -``` - -#### Opciones del Comando - -- `-s, --structures-path`: Ruta a definiciones de estructura adicionales (opcional) -- `-o, --output`: Ruta del archivo de salida para el esquema (predeterminado: stdout) - -El esquema generado incluye todas las estructuras disponibles tanto del directorio contribs integrado como de cualquier ruta de estructuras personalizada que especifiques. Esto es útil para: - -- Autocompletado IDE al escribir archivos `.struct.yaml` -- Validación de referencias de estructura en tus configuraciones -- Descubrimiento programático de plantillas disponibles - -## 📄 Configuración YAML - -Aquí tienes un ejemplo de un archivo de configuración YAML: - -```yaml -files: - - README.md: - content: | - # {{@ project_name @}} - This is a template repository. - - script.sh: - permissions: '0777' - content: | - #!/bin/bash - echo "Hello, {{@ author_name @}}!" - - LICENSE: - file: https://raw.githubusercontent.com/nishanths/license/master/LICENSE - - archivo_remoto.txt: - file: file:///ruta/al/archivo/local.txt - - archivo_github.py: - file: github://owner/repo/branch/path/to/file.py - - archivo_github_https.py: - file: githubhttps://owner/repo/branch/path/to/file.py - - archivo_github_ssh.py: - file: githubssh://owner/repo/branch/path/to/file.py - - archivo_s3.txt: - file: s3://bucket_name/key - - archivo_gcs.txt: - file: gs://bucket_name/key - - src/main.py: - content: | - print("Hello, World!") -folders: - - .devops/modules/mod1: - struct: terraform/module - - .devops/modules/mod2: - struct: terraform/module - with: - module_name: mymod2 - - ./: - struct: - - docker-files - - project/go -variables: - - project_name: - description: "The name of the project" - default: "MyProject" - type: string - - author_name: - description: "The name of the author" - type: string - default: "John Doe" -``` - -### Variables de plantilla - -Puedes usar variables de plantilla en tu archivo de configuración encerrándolas entre `{{@` y `@}}`. Por ejemplo, `{{@ project_name @}}` será reemplazado con el valor de la variable `project_name` en tiempo de ejecución. Si las variables no se proporcionan en la línea de comandos, se solicitarán interactivamente. - -Si necesitas definir bloques, puedes usar la notación de inicio de bloque `{%@` y la notación de final de bloque `%@}`. - -Para definir comentarios, puedes usar la notación de inicio de comentario `{#@` y la notación de fin de comentario `@#}`. - -#### Variables de plantilla predeterminadas - -- `file_name`: El nombre del archivo que se está procesando. -- `file_directory`: El nombre del directorio del archivo que se está procesando. - -#### Variables de plantilla interactivo - -Si no proporcionas todas las variables en la línea de comandos, se solicitarán interactivamente. - -La struct definida debe incluir las variables en una seccion de `variables` con la siguiente estructura: - -```yaml -variables: - - variable_name: - description: "Descripción de la variable" - type: string - default: "Valor predeterminado" -``` - -como puedes ver, cada variable debe tener una descripción, un tipo y un valor predeterminado (opcional). Este valor predeterminado se usará si no se proporciona la variable en la línea de comandos. - -#### Filtros personalizados de Jinja2 - -##### `latest_release` - -Este filtro obtiene la versión más reciente de una release en un repositorio de GitHub. Toma el nombre del repositorio como argumento. - -```yaml -files: - - README.md: - content: | - # MyProject - Latest release: {{@ "httpdss/struct" | latest_release @}} -``` - -Esto utiliza PyGithub para obtener la última release del repositorio, por lo que configurar la variable de entorno `GITHUB_TOKEN` te dará acceso a repositorios privados. - -Si ocurre un error en el proceso, el filtro devolverá `LATEST_RELEASE_ERROR`. - -NOTA: puedes usar este filtro para obtener la última versión de un proveedor de Terraform. Por ejemplo, para obtener la última versión del proveedor `aws`, puedes usar `{{@ "hashicorp/terraform-provider-aws" | latest_release @}}` o el proveedor de datadog `{{@ "DataDog/terraform-provider-datadog" | latest_release @}}`. - -##### `slugify` - -Este filtro convierte una cadena en un slug. Toma un argumento opcional para especificar el carácter separador (el valor predeterminado es `-`). - -```yaml -files: - - README.md: - content: | - # {{@ project_name @}} - This is a template repository. - slugify project_name: {{@ project_name | slugify @}} -``` - -##### `default_branch` - -Este filtro obtiene el nombre de la rama predeterminada de un repositorio de GitHub. Toma el nombre del repositorio como argumento. - -```yaml -files: - - README.md: - content: | - # MyProject - Default branch: {{@ "httpdss/struct" | default_branch @}} -``` - -### Cláusula `with` - -La cláusula `with` te permite pasar variables adicionales a estructuras anidadas. Estas variables se fusionarán con las variables globales y se pueden usar dentro de la estructura anidada. - -Ejemplo: - -```yaml -folders: - - .devops/modules/mod1: - struct: terraform/module - - .devops/modules/mod2: - struct: terraform/module - with: - module_name: mymod2 -``` - -## 📝 Esquema YAML - -Para asegurar que tus archivos de configuración YAML cumplan con la estructura esperada, puedes usar el esquema JSON proporcionado. Esto ayuda a validar tus archivos YAML y proporciona autocompletado en editores compatibles como VSCode. - -### Configuración en VSCode - -1. Instala la [extensión YAML](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml) para VSCode. -2. Añade la siguiente configuración a los ajustes de tu espacio de trabajo (`.vscode/settings.json`): - -```json -{ - "yaml.schemas": { - "https://raw.githubusercontent.com/httpdss/struct/refs/heads/main/struct-schema.json": ".struct.yaml" - } -} -``` - -Esta configuración asociará el esquema JSON con todos los archivos .struct.yaml en tu espacio de trabajo, proporcionando validación y autocompletado. - -## 🔄 Script de Disparador de GitHub - -El script `github-trigger.py` es una utilidad diseñada para activar el flujo de trabajo `run-struct` en todos los repositorios privados de una organización de GitHub que cumplan con ciertos criterios. Este script es especialmente útil para automatizar tareas en múltiples repositorios. - -### 📋 Características - -- Filtra repositorios por un tema específico (por ejemplo, `struct-enabled`). -- Verifica la existencia de un archivo `.struct.yaml` en la rama predeterminada del repositorio. -- Comprueba la presencia del archivo de flujo de trabajo `run-struct` en `.github/workflows/`. -- Activa el evento de despacho del flujo de trabajo en los repositorios elegibles. - -### 🚀 Uso - -Para usar el script, asegúrate de cumplir con los siguientes requisitos: - -1. Un token de acceso personal de GitHub válido con los permisos necesarios (configurado como la variable de entorno `GITHUB_TOKEN`). -2. La biblioteca `PyGithub` instalada (`pip install PyGithub`). - -Ejecuta el script con el siguiente comando: - -```sh -python3 scripts/github-trigger.py -``` - -#### Argumentos - -- ``: El nombre de la organización de GitHub. -- ``: El tema para filtrar los repositorios (por ejemplo, `struct-enabled`). - -#### Ejemplo - -```sh -export GITHUB_TOKEN=tu_token_de_acceso_personal -python3 scripts/github-trigger.py mi-org struct-enabled -``` - -### 🛠️ Cómo Funciona - -1. El script se conecta a la API de GitHub utilizando el token proporcionado. -2. Itera a través de todos los repositorios privados de la organización especificada. -3. Para cada repositorio: - - Verifica si el repositorio tiene el tema especificado. - - Comprueba la existencia de un archivo `.struct.yaml` en la rama predeterminada. - - Confirma la presencia del archivo de flujo de trabajo `run-struct`. - - Activa el evento de despacho del flujo de trabajo si se cumplen todas las condiciones. - -### ⚠️ Notas - -- Asegúrate de configurar la variable de entorno `GITHUB_TOKEN` antes de ejecutar el script. -- El token debe tener permisos suficientes para acceder a repositorios privados y activar flujos de trabajo. -- Los errores durante la ejecución (por ejemplo, archivos faltantes o permisos insuficientes) se registrarán en la consola. - -## 🪝 Ganchos de Pre-generación y Post-generación - -Puedes definir comandos de shell para ejecutar antes y después de la generación de la estructura usando las claves `pre_hooks` y `post_hooks` en tu configuración YAML. Son opcionales y te permiten automatizar pasos de preparación o limpieza. - -- **pre_hooks**: Lista de comandos de shell a ejecutar antes de la generación. Si algún comando falla (código distinto de cero), la generación se aborta. -- **post_hooks**: Lista de comandos de shell a ejecutar después de completar la generación. Si algún comando falla, se muestra un error. - -Ejemplo: - -```yaml -pre_hooks: - - echo "Preparando el entorno..." - - ./scripts/prep.sh - -post_hooks: - - echo "¡Generación completa!" - - ./scripts/cleanup.sh -files: - - README.md: - content: | - # Mi Proyecto -``` - -**Notas:** - -- La salida de los ganchos (stdout y stderr) se muestra en la terminal. -- Si un pre-hook falla, la generación se detiene. -- Si no se definen hooks, no ocurre nada extra. - -## 🗺️ Soporte de Mappings - -Puedes proporcionar un archivo YAML de mappings para inyectar mapas clave-valor en tus plantillas. Esto es útil para referenciar valores específicos de entorno, IDs o cualquier otro mapeo que quieras usar en tus archivos generados. - -### Ejemplo de archivo de mappings - -```yaml -mappings: - teams: - devops: devops-team - aws_account_ids: - myenv-non-prod: 123456789 - myenv-prod: 987654321 -``` - -### Uso en plantillas - -Puedes referenciar valores del mapping en tus plantillas usando la variable `mappings`: - -```jinja -{{@ mappings.aws_account_ids['myenv-prod'] @}} -``` - -Esto se renderizará como: - -```text -987654321 -``` - -### Usar mappings en la cláusula `with` - -También puedes asignar un valor desde un mapping directamente en la cláusula `with` para llamadas a struct de carpetas. Por ejemplo: - -```yaml -folders: - - ./: - struct: - - configs/codeowners - with: - team: {{@ mappings.teams.devops @}} - account_id: {{@ mappings.aws_account_ids['myenv-prod'] @}} -``` - -Esto asignará el valor `devops-team` a la variable `team` y `987654321` a `account_id` en el struct, usando los valores de tu archivo de mappings. - -### Pasar el archivo de mappings - -Usa el argumento `--mappings-file` con el comando `generate`: - -```sh -struct generate --mappings-file ./mimapa.yaml mi-estructura.yaml . -``` - -## 👩‍💻 Desarrollo - -Para comenzar con el desarrollo, sigue estos pasos: - -- Clona el repositorio -- Crea un entorno virtual - -```sh -python3 -m venv .venv -source .venv/bin/activate -``` - -- Instala las dependencias - -```sh -pip install -r requirements.txt -pip install -r requirements.dev.txt -``` - -## 📜 Licencia - -Este proyecto está licenciado bajo la Licencia MIT - consulta el archivo [LICENSE](LICENSE) para más detalles. - -## 💰 Financiamiento - -Si encuentras este proyecto útil, considera apoyarlo a través de donaciones: [patreon/structproject](https://patreon.com/structproject) - -## 🤝 Contribuyendo - -¡Las contribuciones son bienvenidas! Por favor, abre un issue o envía un pull request. - -## 🙏 Agradecimientos - -Un agradecimiento especial a todos los contribuyentes que hicieron posible este proyecto. - -## 🐞 Problemas Conocidos - -- [ ] TBD diff --git a/README.md b/README.md index be1984e..4eacd22 100644 --- a/README.md +++ b/README.md @@ -1,552 +1,125 @@ # 🚀 STRUCT: Automated Project Structure Generator -[![en](https://img.shields.io/badge/lang-en-red.svg)](https://github.com/httpdss/struct/blob/master/README.md) [![es](https://img.shields.io/badge/lang-es-yellow.svg)](https://github.com/httpdss/struct/blob/master/README.es.md) - ![Struct Banner](extras/banner.png) -## 📄 Table of Contents - -- [Introduction](#-introduction) -- [Features](#-features) -- [Installation](#️-installation) -- [Quick Start](#-quick-start) -- [Usage](#-usage) -- [YAML Configuration](#-yaml-configuration) -- [YAML Schema](#-yaml-schema) -- [GitHub Trigger Script](#-github-trigger-script) -- [Development](#-development) -- [License](#-license) -- [Funding](#-funding) -- [Contributing](#-contributing) -- [Acknowledgments](#-acknowledgments) -- [Known Issues](#-known-issues) - -## 📦 Introduction +**STRUCT** is a powerful, flexible tool for automating project structure creation through YAML configurations. Generate consistent project layouts, boilerplate code, and configurations with template variables, remote content fetching, and intelligent file handling. -STRUCT is a powerful and flexible script designed to automate the creation of project structures based on YAML configurations. It supports template variables, custom file permissions, remote content fetching, and multiple file handling strategies to streamline your development setup process. +> 📚 **[View Complete Documentation](docs/index.md)** | 🚀 **[Quick Start Guide](docs/quickstart.md)** | 🔧 **[Installation](docs/installation.md)** -This is targeted towards developers, DevOps engineers, and anyone who wants to automate the creation of project structures. It can be used to generate boilerplate code, configuration files, documentation, and more. +## ✨ Key Features -## ✨ Features +- **📝 YAML-Based Configuration** - Define project structures in simple, readable YAML +- **🔧 Template Variables** - Dynamic content with Jinja2 templating and interactive prompts +- **🌐 Remote Content** - Fetch files from GitHub, HTTP/HTTPS, S3, and Google Cloud Storage +- **🛡️ Smart File Handling** - Multiple strategies for managing existing files (overwrite, skip, backup, etc.) +- **🪝 Automation Hooks** - Pre and post-generation shell commands +- **🎯 Dry Run Mode** - Preview changes before applying them +- **✅ Validation & Schema** - Built-in YAML validation and IDE support -- **YAML Configuration**: Define your project structure in a simple YAML file. -- **Template Variables**: Use placeholders in your configuration and replace them with actual values at runtime. Also supports custom Jinja2 filters and interactive mode to fill in the variables. -- **Custom File Permissions**: Set custom permissions for your files directly from the YAML configuration. -- **Remote Content Fetching**: Include content from remote files by specifying their URLs. -- **File Handling Strategies**: Choose from multiple strategies (overwrite, skip, append, rename, backup) to manage existing files. -- **Dry Run**: Preview the actions without making any changes to your file system. -- **Configuration Validation**: Ensure your YAML configuration is valid before executing the script. -- **Verbose Logging**: Get detailed logs of the script's actions for easy debugging and monitoring. +## 🚀 Quick Start -## 🛠️ Installation +### Installation -### Using pip - -You can install STRUCT using pip: - -```sh +```bash +# Install via pip pip install git+https://github.com/httpdss/struct.git -``` - -### From Source - -Alternatively, you can clone the repository and install it locally. See the [Development](#-development) section for more details. - -### Using Docker - -You can use the Docker image to run the script without installing it on your system. See the [Quick Start](#-quick-start) section for more details. - -## 🐳 Quick Start - -### Quick Start Using Docker -1. Create a YAML configuration file for your project structure. See sample configuration [here](./example/structure.yaml). -2. Run the following command to generate the project structure: - -```sh -docker run \ - -v $(pwd):/workdir \ - -u $(id -u):$(id -g) \ - ghcr.io/httpdss/struct:main generate \ - /workdir/example/structure.yaml \ - /workdir/example_output -``` - -### Quick Start Using Docker Alpine - -For testing, you can run an alpine Docker container and install the script inside it: - -```sh -docker run -it --entrypoint="" python:3.10-alpine sh -l +# Or run with Docker +docker run -v $(pwd):/workdir ghcr.io/httpdss/struct:main generate my-config.yaml ./output ``` -Inside the container: - -```sh -apk add python-pip git vim -pip install git+https://github.com/httpdss/struct.git -mkdir example -cd example/ -touch structure.yaml -vim structure.yaml # copy the content from the example folder -struct generate structure.yaml . -``` - -## 📝 Usage - -Run the script with the following command using one of the following subcommands: - -- `generate`: Generate the project structure based on the YAML configuration. -- `generate-schema`: Generate JSON schema for available structure templates. -- `validate`: Validate the YAML configuration file. -- `info`: Display information about the script and its dependencies. -- `list`: List the available structs +### Basic Usage -For more information, run the script with the `-h` or `--help` option (this is also available for each subcommand): - -![Struct List](./docs/vhs/usage.gif) - -```sh -struct -h -``` - -### Simple Example - -```sh +```bash +# Generate a Terraform module structure struct generate terraform-module ./my-terraform-module -``` - -### More Complete Example - -```sh -struct generate \ - --log=DEBUG \ - --dry-run \ - --backup=/path/to/backup \ - --file-strategy=rename \ - --log-file=/path/to/logfile.log \ - terraform-module \ - ./my-terraform-module -``` - -### Generate Schema Command - -The `generate-schema` command creates JSON schema definitions for available structure templates, making it easier for tools and IDEs to provide autocompletion and validation. - -#### Basic Usage -```sh -# Generate schema to stdout -struct generate-schema +# List available structures +struct list -# Generate schema with custom structures path -struct generate-schema -s /path/to/custom/structures - -# Save schema to file -struct generate-schema -o schema.json - -# Combine custom path and output file -struct generate-schema -s /path/to/custom/structures -o schema.json +# Validate a configuration +struct validate my-config.yaml ``` -#### Command Options - -- `-s, --structures-path`: Path to additional structure definitions (optional) -- `-o, --output`: Output file path for the schema (default: stdout) - -The generated schema includes all available structures from both the built-in contribs directory and any custom structures path you specify. This is useful for: - -- IDE autocompletion when writing `.struct.yaml` files -- Validation of structure references in your configurations -- Programmatic discovery of available templates - -## 📝 YAML Configuration - -### Configuration Properties - -When defining your project structure in the YAML configuration file, you can use various properties to control the behavior of the script. Here are the available properties: - -- **skip**: If set to `true`, the file or folder will be skipped and not created. -- **skip_if_exists**: If set to `true`, the file or folder will be skipped if it already exists. -- **permissions**: Set custom file permissions using a string representation of the octal value (e.g., `'0777'`). -- **content**: Define the content of the file directly in the YAML configuration. -- **file**: Specify a local or remote file to include. Supported protocols include `file://`, `http://`, `https://`, `github://`, `githubhttps://`, `githubssh://`, `s3://`, and `gs://`. - -Example: +### Example Configuration ```yaml files: - README.md: - skip: true content: | # {{@ project_name @}} - This is a template repository. - - script.sh: - skip_if_exists: true - permissions: '0777' - content: | - #!/bin/bash - echo "Hello, {{@ author_name @}}!" - - LICENSE: - file: https://raw.githubusercontent.com/nishanths/license/master/LICENSE - - remote_file.txt: - file: file:///path/to/local/file.txt - - github_file.py: - file: github://owner/repo/branch/path/to/file.py - - github_https_file.py: - file: githubhttps://owner/repo/branch/path/to/file.py - - github_ssh_file.py: - file: githubssh://owner/repo/branch/path/to/file.py - - s3_file.txt: - file: s3://bucket_name/key - - gcs_file.txt: - file: gs://bucket_name/key - - src/main.py: - content: | - print("Hello, World!") + Generated with STRUCT + - .gitignore: + file: github://github/gitignore/main/Python.gitignore + folders: - - .devops/modules/mod1: - struct: terraform/module - - .devops/modules/mod2: - struct: terraform/module + - src/: + struct: project/python with: - module_name: mymod2 - - ./: - struct: - - docker-files - - project/go -variables: - - project_name: - description: "The name of the project" - default: "MyProject" - type: string - - author_name: - description: "The name of the author" - type: string - default: "John Doe" -``` - -These properties allow you to customize the behavior and content of the files and folders generated by the script, providing flexibility and control over your project structure. - -### Template variables - -You can use template variables in your configuration file by enclosing them in `{{@` and `@}}`. For example, `{{@ project_name @}}` will be replaced with the value of the `project_name` variable at runtime. If this are not set when running the script, it will prompt you to enter the value interactively. + app_name: "{{@ project_name | slugify @}}" -If you need to define blocks you can use starting block notation `{%@` and end block notation `%@}`. - -To define comments you can use the comment start notation `{#@` and end comment notation `@#}`. - -#### Default template variables - -- `file_name`: The name of the file being processed. -- `file_directory`: The name of the directory of file that is being processed. - -#### Interactive template variables - -If you don't provide a default value for a variable, the script will prompt you to enter the value interactively. - -The struct defined should define the variable on a specific section of the YAML file. For example: - -```yaml variables: - - author_name: - description: "The name of the author" + - project_name: + description: "Name of your project" type: string - default: "John Doe" -``` - -as you can see, the `author_name` variable is defined on the `variables` section of the YAML file. it includes a `description`, `type` and `default` value which is used if the user doesn't provide a value interactively. - -#### Custom Jinja2 filters - -##### `latest_release` - -This filter fetches the latest release version of a GitHub repository. It takes the repository name as an argument. - -```yaml -files: - - README.md: - content: | - # MyProject - Latest release: {{@ "httpdss/struct" | latest_release @}} -``` - -This uses PyGithub to fetch the latest release of the repository so setting the `GITHUB_TOKEN` environment variable will give you access to private repositories. - -If there is an error in the process, the filter will return `LATEST_RELEASE_ERROR`. - -NOTE: you can use this filter to get the latest release for a terraform provider. For example, to get the latest release of the `aws` provider, you can use `{{@ "hashicorp/terraform-provider-aws" | latest_release @}}` or datadog provider `{{@ "DataDog/terraform-provider-datadog" | latest_release @}}`. - -##### `slugify` - -This filter converts a string into a slug. It takes an optional argument to specify the separator character (default is `-`). - -```yaml -files: - - README.md: - content: | - # {{@ project_name @}} - This is a template repository. - slugify project_name: {{@ project_name | slugify @}} -``` - -##### `default_branch` - -This filter fetches the default branch name of a GitHub repository. It takes the repository name as an argument. - -```yaml -files: - - README.md: - content: | - # MyProject - Default branch: {{@ "httpdss/struct" | default_branch @}} -``` - -### `with` Clause - -The `with` clause allows you to pass additional variables to nested structures. These variables will be merged with the global variables and can be used within the nested structure. - -Example: - -```yaml -folders: - - .devops/modules/mod1: - struct: terraform/module - - .devops/modules/mod2: - struct: terraform/module - with: - module_name: mymod2 -``` - -## 📝 YAML Schema - -To ensure your YAML configuration files adhere to the expected structure, you can use the provided JSON schema. This helps in validating your YAML files and provides autocompletion in supported editors like VSCode. - -### Configuring VSCode - -1. Install the [YAML extension](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml) for VSCode. -2. Add the following configuration to your workspace settings (`.vscode/settings.json`): - -```json -{ - "yaml.schemas": { - "https://raw.githubusercontent.com/httpdss/struct/main/extras/schema.json": ".struct.yaml" - } -} -``` - -This configuration will associate the JSON schema with all .struct.yaml files in your workspace, providing validation and autocompletion. - -## 🔄 GitHub Trigger Script - -The `github-trigger.py` script is a utility designed to trigger the `run-struct` workflow for all private repositories in a GitHub organization that meet specific criteria. This script is particularly useful for automating tasks across multiple repositories. - -### Features - -- Filters repositories by a specific topic (e.g., `struct-enabled`). -- Checks for the existence of a `.struct.yaml` file in the repository's default branch. -- Verifies the presence of the `run-struct` workflow file in `.github/workflows/`. -- Triggers the workflow dispatch event for eligible repositories. - -### Usage - -To use the script, ensure you have the following prerequisites: - -1. A valid GitHub Personal Access Token with the necessary permissions (set as the `GITHUB_TOKEN` environment variable). -2. The `PyGithub` library installed (`pip install PyGithub`). - -Run the script with the following command: - -```sh -python3 scripts/github-trigger.py -``` - -#### Arguments - -- ``: The name of the GitHub organization. -- ``: The topic to filter repositories by (e.g., `struct-enabled`). - -#### Example - -```sh -export GITHUB_TOKEN=your_personal_access_token -python3 scripts/github-trigger.py my-org struct-enabled -``` - -### How It Works - -1. The script connects to the GitHub API using the provided token. -2. It iterates through all private repositories in the specified organization. -3. For each repository: - - Checks if the repository has the specified topic. - - Verifies the existence of a `.struct.yaml` file in the default branch. - - Confirms the presence of the `run-struct` workflow file. - - Triggers the workflow dispatch event if all conditions are met. - -### Notes - -- Ensure the `GITHUB_TOKEN` environment variable is set before running the script. -- The token must have sufficient permissions to access private repositories and trigger workflows. -- Errors during execution (e.g., missing files or insufficient permissions) will be logged to the console. - -## 👩‍💻 Development - -To get started with development, follow these steps: - -- Clone the repository -- Create a virtual environment - -```sh -python3 -m venv .venv -source .venv/bin/activate -``` - -- Install the dependencies - -```sh -pip install -r requirements.txt -pip install -r requirements.dev.txt -``` - -## 🛠️ Command-Line Auto-Completion - -This project uses [argcomplete](https://kislyuk.github.io/argcomplete/) to provide command-line auto-completion for the `struct` script. Follow these steps to enable auto-completion: - -1. **Install `argcomplete`**: - - ```sh - pip install argcomplete - ``` - -2. **Enable global completion** for your shell. This step is usually done once: - - ```sh - activate-global-python-argcomplete - ``` - -3. **Register the script for auto-completion**. Add the following line to your shell's configuration file (e.g., `.bashrc`, `.zshrc`): - - ```sh - eval "$(register-python-argcomplete struct)" - ``` - -4. **Reload your shell configuration**: - - ```sh - source ~/.bashrc # or source ~/.zshrc for Zsh users - ``` - -After completing these steps, you should have auto-completion enabled for the `struct` script. You can test it by typing part of a command and pressing `Tab` to see the available options. - -```sh -struct -``` - -## Articles - -- [**Defining User Prompts on STRUCT: Harnessing GPT-4.1 for Scalable Project Scaffolding**](https://medium.com/@httpdss/defining-user-prompts-on-struct-harnessing-gpt-4-1-for-scalable-project-scaffolding-e6d3b4ec4701) -- [**Unlocking Developer Productivity with STRUCT: The Ultimate Open-Source Tool for Automated Project Structures**](https://blog.devops.dev/unlocking-developer-productivity-with-struct-the-ultimate-open-source-tool-for-automated-project-8bca9b5f40f9) -- [**Automating Project Structures with STRUCT and GitHub Actions**](https://medium.com/@httpdss/automating-project-structures-with-struct-and-github-actions-64e09c40c11e) -- [**Advanced STRUCT Tips: Working with Template Variables and Jinja2 Filters**](https://medium.com/@httpdss/advanced-struct-tips-working-with-template-variables-and-jinja2-filters-b239bf3145e4) - -## 🪝 Pre-generation and Post-generation Hooks - -You can define shell commands to run before and after structure generation using the `pre_hooks` and `post_hooks` keys in your YAML configuration. These are optional and allow you to automate setup or cleanup steps. - -- **pre_hooks**: List of shell commands to run before generation. If any command fails (non-zero exit), generation is aborted. -- **post_hooks**: List of shell commands to run after generation completes. If any command fails, an error is shown. - -Example: - -```yaml -pre_hooks: - - echo "Preparing environment..." - - ./scripts/prep.sh - -post_hooks: - - echo "Generation complete!" - - ./scripts/cleanup.sh -files: - - README.md: - content: | - # My Project + default: "MyProject" ``` -**Notes:** - -- Output from hooks (stdout and stderr) is shown in the terminal. -- If a pre-hook fails, generation is halted. -- If no hooks are defined, nothing extra happens. +## 📚 Documentation -## 🗺️ Mappings Support +Our comprehensive documentation is organized into the following sections: -You can provide a mappings YAML file to inject key-value maps into your templates. This is useful for referencing environment-specific values, IDs, or any other mapping you want to use in your generated files. +### 🏁 Getting Started -### Example mappings file - -```yaml -mappings: - teams: - devops: devops-team - aws_account_ids: - myenv-non-prod: 123456789 - myenv-prod: 987654321 -``` +- **[Installation Guide](docs/installation.md)** - Multiple installation methods +- **[Quick Start](docs/quickstart.md)** - Get up and running in minutes +- **[Basic Usage](docs/usage.md)** - Core commands and options -### Usage in templates +### ⚙️ Configuration -You can reference mapping values in your templates using the `mappings` variable: +- **[YAML Configuration](docs/configuration.md)** - Complete configuration reference +- **[Template Variables](docs/template-variables.md)** - Dynamic content and Jinja2 features +- **[File Handling](docs/file-handling.md)** - Managing files, permissions, and remote content +- **[Schema Reference](docs/schema.md)** - YAML validation and IDE support -```jinja -{{@ mappings.aws_account_ids['myenv-prod'] @}} -``` +### 🔧 Advanced Features -This will render as: +- **[Hooks](docs/hooks.md)** - Pre and post-generation automation +- **[Mappings](docs/mappings.md)** - External data integration +- **[GitHub Integration](docs/github-integration.md)** - Automation with GitHub Actions +- **[Command-Line Completion](docs/completion.md)** - Enhanced CLI experience -```text -987654321 -``` +### 👩‍💻 Development -### Using mappings in the `with` clause +- **[Development Setup](docs/development.md)** - Contributing to STRUCT +- **[Known Issues](docs/known-issues.md)** - Current limitations and workarounds -You can also assign a value from a mapping directly in the `with` clause for folder struct calls. For example: +### 📖 Resources -```yaml -folders: - - ./: - struct: - - configs/codeowners - with: - team: {{@ mappings.teams.devops @}} - account_id: {{@ mappings.aws_account_ids['myenv-prod'] @}} -``` +- **[Articles & Tutorials](docs/articles.md)** - Community content and learning resources +- **[Examples](example/)** - Practical examples and use cases -This will assign the value `devops-team` to the variable `team` and `987654321` to `account_id` in the struct, using the values from your mappings file. +## 🎯 Use Cases -### Passing the mappings file +- **Infrastructure as Code** - Generate Terraform modules, Kubernetes manifests +- **Application Scaffolding** - Bootstrap microservices, APIs, frontend projects +- **DevOps Automation** - CI/CD pipeline templates, configuration management +- **Documentation** - Consistent project documentation and compliance templates -Use the `--mappings-file` argument with the `generate` command: +## 🤝 Community -```sh -struct generate --mappings-file ./mymap.yaml my-struct.yaml . -``` +- **[Contributing Guidelines](docs/development.md#contributing-guidelines)** - How to contribute +- **[GitHub Discussions](https://github.com/httpdss/struct/discussions)** - Community support +- **[Articles & Tutorials](docs/articles.md)** - Learning resources ## 📜 License -This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. - -## 💰 Funding - -If you find this project helpful, please consider supporting it through donations: [patreon/structproject](https://patreon.com/structproject) - -## 🤝 Contributing - -Contributions are welcome! Please open an issue or submit a pull request. +MIT License - see [LICENSE](LICENSE) for details. -## 🙏 Acknowledgments +## 💰 Support -Special thanks to all the contributors who helped make this project possible. +If STRUCT helps your workflow, consider supporting the project: [patreon/structproject](https://patreon.com/structproject) -## 🐞 Known Issues +--- -- [ ] TBD +**📚 [Complete Documentation](docs/index.md)** | **🐛 [Report Issues](https://github.com/httpdss/struct/issues)** | **💬 [Discussions](https://github.com/httpdss/struct/discussions)** diff --git a/docs/articles.md b/docs/articles.md new file mode 100644 index 0000000..7030922 --- /dev/null +++ b/docs/articles.md @@ -0,0 +1,184 @@ +# Articles and Resources + +Collection of articles, tutorials, and external resources about STRUCT. + +## Official Articles + +### [Defining User Prompts on STRUCT: Harnessing GPT-4.1 for Scalable Project Scaffolding](https://medium.com/@httpdss/defining-user-prompts-on-struct-harnessing-gpt-4-1-for-scalable-project-scaffolding-e6d3b4ec4701) + +Learn how to leverage GPT-4.1 to create intelligent project scaffolding with STRUCT. This article covers advanced prompting techniques and AI-driven project generation. + +**Topics covered:** + +- AI-powered template generation +- Dynamic project structures +- GPT-4.1 integration patterns +- Scalable scaffolding strategies + +### [Unlocking Developer Productivity with STRUCT: The Ultimate Open-Source Tool for Automated Project Structures](https://blog.devops.dev/unlocking-developer-productivity-with-struct-the-ultimate-open-source-tool-for-automated-project-8bca9b5f40f9) + +A comprehensive overview of STRUCT's capabilities and how it can boost developer productivity through automation. + +**Topics covered:** + +- Project structure automation +- Developer workflow optimization +- Open-source benefits +- Real-world use cases + +### [Automating Project Structures with STRUCT and GitHub Actions](https://medium.com/@httpdss/automating-project-structures-with-struct-and-github-actions-64e09c40c11e) + +Discover how to integrate STRUCT with GitHub Actions for automated project setup and maintenance across repositories. + +**Topics covered:** + +- GitHub Actions integration +- CI/CD automation +- Repository management +- Workflow triggers + +### [Advanced STRUCT Tips: Working with Template Variables and Jinja2 Filters](https://medium.com/@httpdss/advanced-struct-tips-working-with-template-variables-and-jinja2-filters-b239bf3145e4) + +Deep dive into STRUCT's templating system, covering advanced variable usage and custom Jinja2 filters. + +**Topics covered:** + +- Template variable patterns +- Custom Jinja2 filters +- Dynamic content generation +- Advanced templating techniques + +## Community Articles + +*Community contributions welcome! Submit a PR to add your STRUCT-related articles here.* + +## Tutorials + +### Video Tutorials + +*Coming soon - video tutorials will be added as they become available.* + +### Interactive Tutorials + +- **Getting Started Guide**: Follow along with practical examples in the [Quick Start](quickstart.md) documentation +- **Template Variables**: Learn by example in the [Template Variables](template-variables.md) guide +- **Advanced Configuration**: Explore complex scenarios in [YAML Configuration](configuration.md) + +## Use Cases + +### DevOps and Infrastructure + +- **Terraform Module Generation**: Standardize infrastructure code +- **Kubernetes Manifests**: Generate consistent deployment configurations +- **CI/CD Pipeline Templates**: Automate pipeline creation +- **Configuration Management**: Maintain consistent environments + +### Application Development + +- **Microservice Scaffolding**: Generate service boilerplate +- **API Project Templates**: Standardize REST API structures +- **Frontend Framework Setup**: Bootstrap React, Vue, or Angular projects +- **Database Schema Generation**: Create consistent database structures + +### Documentation and Compliance + +- **Project Documentation**: Generate README templates and docs +- **Compliance Templates**: Ensure regulatory requirements +- **Security Configurations**: Apply security best practices +- **Audit Trail Generation**: Create trackable project structures + +## Presentations and Talks + +*Conference presentations and community talks about STRUCT will be listed here.* + +## Research and Academic Papers + +*Academic research involving STRUCT or related concepts will be referenced here.* + +## External Tools and Integrations + +### IDE Extensions + +- **VS Code YAML Extension**: Enhanced editing with schema validation +- **IntelliJ YAML Support**: Schema-aware editing and completion + +### CI/CD Integrations + +- **GitHub Actions**: Pre-built actions for STRUCT workflows +- **GitLab CI**: Integration examples and templates +- **Jenkins**: Pipeline integration patterns + +### Complementary Tools + +- **Cookiecutter**: Alternative project templating tool +- **Yeoman**: JavaScript-focused scaffolding +- **Copilot**: AI-powered code generation + +## Community Resources + +### Forums and Discussion + +- **GitHub Discussions**: Official community forum +- **Stack Overflow**: Questions tagged with `struct-tool` + +### Code Examples + +- **GitHub Examples Repository**: Community-contributed examples +- **Gist Collections**: Snippets and configurations +- **Docker Hub**: Pre-built images and examples + +## Learning Paths + +### Beginner Path + +1. Start with [Installation](installation.md) +2. Follow the [Quick Start](quickstart.md) guide +3. Explore [Basic Usage](usage.md) +4. Learn [YAML Configuration](configuration.md) basics + +### Intermediate Path + +1. Master [Template Variables](template-variables.md) +2. Understand [File Handling](file-handling.md) strategies +3. Explore [Custom Structures](custom-structures.md) +4. Learn [Mappings](mappings.md) for data separation + +### Advanced Path + +1. Implement [Hooks](hooks.md) for automation +2. Set up [GitHub Integration](github-integration.md) +3. Contribute to [Development](development.md) +4. Create complex multi-environment setups + +## Contributing Content + +We welcome contributions to this resource collection! Here's how you can help: + +### Adding Articles + +1. Fork the repository +2. Add your article to the appropriate section +3. Include a brief description and key topics +4. Submit a pull request + +### Suggesting Resources + +- Open an issue with the "resource suggestion" label +- Include the resource link and description +- Explain why it would be valuable to the community + +### Writing Tutorials + +- Create step-by-step guides with practical examples +- Include code snippets and expected outputs +- Test instructions thoroughly before submitting + +## Newsletter and Updates + +Stay updated with the latest STRUCT developments: + +- **GitHub Releases**: Watch the repository for release notifications +- **Community Updates**: Follow project maintainers on social media +- **Blog Subscriptions**: Subscribe to the official blog for updates + +*This page is regularly updated with new content. Bookmark it for future reference!* diff --git a/docs/completion.md b/docs/completion.md new file mode 100644 index 0000000..4984eda --- /dev/null +++ b/docs/completion.md @@ -0,0 +1,211 @@ +# Command-Line Auto-Completion + +This project uses [argcomplete](https://kislyuk.github.io/argcomplete/) to provide command-line auto-completion for the `struct` script. Follow these steps to enable auto-completion: + +## Installation + +### 1. Install argcomplete + +```sh +pip install argcomplete +``` + +### 2. Enable Global Completion + +This step is usually done once per system: + +```sh +activate-global-python-argcomplete +``` + +This command sets up global completion for all Python scripts that use argcomplete. + +### 3. Register the Script + +Add the following line to your shell's configuration file: + +**For Bash** (`.bashrc` or `.bash_profile`): + +```sh +eval "$(register-python-argcomplete struct)" +``` + +**For Zsh** (`.zshrc`): + +```sh +eval "$(register-python-argcomplete struct)" +``` + +**For Fish** (`.config/fish/config.fish`): + +```fish +register-python-argcomplete --shell fish struct | source +``` + +### 4. Reload Your Shell + +```sh +# For Bash +source ~/.bashrc + +# For Zsh +source ~/.zshrc + +# For Fish +source ~/.config/fish/config.fish +``` + +## Usage + +After completing the setup, you can use auto-completion by typing part of a command and pressing `Tab`: + +```sh +struct +# Shows: generate, generate-schema, validate, info, list + +struct generate +# Shows available structure names and options + +struct generate -- +# Shows: --log, --dry-run, --backup, --file-strategy, etc. +``` + +## Advanced Configuration + +### Per-Project Completion + +If you only want completion for specific projects, you can add completion to your project's virtual environment activation script: + +```sh +# In your .venv/bin/activate file, add: +eval "$(register-python-argcomplete struct)" +``` + +### Custom Completion + +You can create custom completion functions for specific use cases: + +```sh +# Custom completion for structure names +_struct_structures() { + local structures=$(struct list --names-only 2>/dev/null) + COMPREPLY=($(compgen -W "$structures" -- "${COMP_WORDS[COMP_CWORD]}")) +} + +# Register custom completion +complete -F _struct_structures struct-generate +``` + +## Troubleshooting + +### Completion Not Working + +1. **Check argcomplete installation**: + + ```sh + python -c "import argcomplete; print('OK')" + ``` + +2. **Verify global activation**: + + ```sh + activate-global-python-argcomplete --user + ``` + +3. **Check shell configuration**: + Make sure the eval statement is in the correct shell configuration file. + +4. **Restart your shell**: + Sometimes you need to completely restart your terminal. + +### Slow Completion + +If completion is slow, you can enable caching: + +```sh +export ARGCOMPLETE_USE_TEMPFILES=1 +``` + +Add this to your shell configuration file for persistent caching. + +### Debug Completion + +Enable debug mode to troubleshoot completion issues: + +```sh +export _ARGCOMPLETE_DEBUG=1 +struct +``` + +## Platform-Specific Notes + +### macOS + +On macOS, you might need to install bash-completion first: + +```sh +# Using Homebrew +brew install bash-completion + +# Then add to ~/.bash_profile: +[[ -r "/usr/local/etc/profile.d/bash_completion.sh" ]] && . "/usr/local/etc/profile.d/bash_completion.sh" +``` + +### Windows + +For Windows users using Git Bash or WSL, follow the same steps as Linux. For PowerShell, argcomplete support is limited. + +### Docker + +When running STRUCT in Docker, completion won't work directly. However, you can create a wrapper script: + +```sh +#!/bin/bash +# struct-wrapper.sh +docker run --rm -v $(pwd):/workdir ghcr.io/httpdss/struct:main "$@" +``` + +Then set up completion for the wrapper: + +```sh +eval "$(register-python-argcomplete struct-wrapper.sh)" +``` + +## Benefits of Auto-Completion + +- **Faster typing**: Quickly complete command names and options +- **Discoverability**: See available commands and options +- **Accuracy**: Reduce typos and errors +- **Productivity**: Spend less time looking up command syntax + +## Supported Completions + +STRUCT provides completion for: + +- **Commands**: `generate`, `validate`, `list`, etc. +- **Options**: `--log`, `--dry-run`, `--backup`, etc. +- **Structure names**: All available built-in and custom structures +- **File paths**: Local files and directories +- **Enum values**: Log levels, file strategies, etc. + +## Example Session + +```sh +$ struct +generate generate-schema info list validate + +$ struct generate +configs/ docker-files project/ terraform/ + +$ struct generate terraform/ +terraform/app terraform/module + +$ struct generate -- +--backup --dry-run --file-strategy --log +--log-file --mappings-file --structures-path + +$ struct generate --log +DEBUG ERROR INFO WARNING +``` + +This makes working with STRUCT much more efficient and user-friendly! diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..285bc1f --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,141 @@ +# YAML Configuration + +## Configuration Properties + +When defining your project structure in the YAML configuration file, you can use various properties to control the behavior of the script. Here are the available properties: + +- **skip**: If set to `true`, the file or folder will be skipped and not created. +- **skip_if_exists**: If set to `true`, the file or folder will be skipped if it already exists. +- **permissions**: Set custom file permissions using a string representation of the octal value (e.g., `'0777'`). +- **content**: Define the content of the file directly in the YAML configuration. +- **file**: Specify a local or remote file to include. Supported protocols include `file://`, `http://`, `https://`, `github://`, `githubhttps://`, `githubssh://`, `s3://`, and `gs://`. + +Example: + +```yaml +files: + - README.md: + skip: true + content: | + # {{@ project_name @}} + This is a template repository. + - script.sh: + skip_if_exists: true + permissions: '0777' + content: | + #!/bin/bash + echo "Hello, {{@ author_name @}}!" + - LICENSE: + file: https://raw.githubusercontent.com/nishanths/license/master/LICENSE + - remote_file.txt: + file: file:///path/to/local/file.txt + - github_file.py: + file: github://owner/repo/branch/path/to/file.py + - github_https_file.py: + file: githubhttps://owner/repo/branch/path/to/file.py + - github_ssh_file.py: + file: githubssh://owner/repo/branch/path/to/file.py + - s3_file.txt: + file: s3://bucket_name/key + - gcs_file.txt: + file: gs://bucket_name/key + - src/main.py: + content: | + print("Hello, World!") +folders: + - .devops/modules/mod1: + struct: terraform/module + - .devops/modules/mod2: + struct: terraform/module + with: + module_name: mymod2 + - ./: + struct: + - docker-files + - project/go +variables: + - project_name: + description: "The name of the project" + default: "MyProject" + type: string + - author_name: + description: "The name of the author" + type: string + default: "John Doe" +``` + +These properties allow you to customize the behavior and content of the files and folders generated by the script, providing flexibility and control over your project structure. + +### Template Variables + +You can use template variables in your configuration file by enclosing them in `{{@` and `@}}`. For example, `{{@ project_name @}}` will be replaced with the value of the `project_name` variable at runtime. If this are not set when running the script, it will prompt you to enter the value interactively. + +If you need to define blocks you can use starting block notation `{%@` and end block notation `%@}`. + +To define comments you can use the comment start notation `{#@` and end comment notation `@#}`. + +#### Default template variables + +- `file_name`: The name of the file being processed. +- `file_directory`: The name of the directory of file that is being processed. + +#### Interactive template variables + +If you don't provide a default value for a variable, the script will prompt you to enter the value interactively. + +The struct defined should define the variable on a specific section of the YAML file. For example: + +```yaml +variables: + - author_name: + description: "The name of the author" + type: string + default: "John Doe" +``` + +as you can see, the `author_name` variable is defined on the `variables` section of the YAML file. it includes a `description`, `type` and `default` value which is used if the user doesn't provide a value interactively. + +#### Custom Jinja2 filters + +##### `latest_release` + +This filter fetches the latest release version of a GitHub repository. It takes the repository name as an argument. + +```yaml +files: + - README.md: + content: | + # MyProject + Latest release: {{@ "httpdss/struct" | latest_release @}} +``` + +This uses PyGithub to fetch the latest release of the repository so setting the `GITHUB_TOKEN` environment variable will give you access to private repositories. + +If there is an error in the process, the filter will return `LATEST_RELEASE_ERROR`. + +NOTE: you can use this filter to get the latest release for a terraform provider. For example, to get the latest release of the `aws` provider, you can use `{{@ "hashicorp/terraform-provider-aws" | latest_release @}}` or datadog provider `{{@ "DataDog/terraform-provider-datadog" | latest_release @}}`. + +##### `slugify` + +This filter converts a string into a slug. It takes an optional argument to specify the separator character (default is `-`). + +```yaml +files: + - README.md: + content: | + # {{@ project_name @}} + This is a template repository. + slugify project_name: {{@ project_name | slugify @}} +``` + +##### `default_branch` + +This filter fetches the default branch name of a GitHub repository. It takes the repository name as an argument. + +```yaml +files: + - README.md: + content: | + # MyProject + Default branch: {{@ "httpdss/struct" | default_branch @}} +``` diff --git a/docs/contributing.md b/docs/contributing.md new file mode 100644 index 0000000..9755cb0 --- /dev/null +++ b/docs/contributing.md @@ -0,0 +1,213 @@ +# Contributing to STRUCT + +Thank you for your interest in contributing to STRUCT! We welcome contributions from the community and are pleased to have you join us. + +## 🚀 Quick Start + +1. **Fork the repository** on GitHub +2. **Clone your fork** locally +3. **Create a feature branch** from `main` +4. **Make your changes** and commit them +5. **Push to your fork** and submit a pull request + +## 📋 Development Setup + +### Prerequisites + +- Python 3.8 or higher +- Git +- A text editor or IDE + +### Local Development + +Clone the repository: + +```bash +git clone https://github.com/httpdss/struct.git +cd struct +``` + +Create a virtual environment: + +```bash +python -m venv venv +source venv/bin/activate # On Windows: venv\Scripts\activate +``` + +Install in development mode: + +```bash +pip install -e . +pip install -r requirements.dev.txt +``` + +Run tests to ensure everything works: + +```bash +pytest +``` + +## 🔧 Making Changes + +### Code Style + +- Follow PEP 8 guidelines +- Use 2 spaces for indentation in YAML files +- Use 4 spaces for indentation in Python files +- Write clear, descriptive commit messages +- Add docstrings to new functions and classes + +### Testing + +- Write tests for new functionality +- Ensure all existing tests pass +- Run the test suite: `pytest` +- Check test coverage: `pytest --cov=struct_module` + +### Pull Request Guidelines + +1. **Create descriptive PR titles** that summarize the change +2. **Fill out the PR template** completely +3. **Link to related issues** when applicable +4. **Keep PRs focused** - one feature/fix per PR +5. **Update documentation** if your changes affect user-facing functionality + +## 📝 Documentation + +We use Markdown for documentation. When contributing: + +- Update relevant documentation files +- Add examples for new features +- Keep language clear and concise +- Follow the existing documentation structure + +### Documentation Structure + +```text +docs/ +├── index.md # Main documentation index +├── installation.md # Installation instructions +├── quickstart.md # Quick start guide +├── configuration.md # YAML configuration reference +├── template-variables.md # Template variable documentation +├── file-handling.md # File handling features +├── custom-structures.md # Creating custom structures +├── hooks.md # Pre/post hooks +├── mappings.md # External data mappings +├── github-integration.md # GitHub Actions integration +├── development.md # Development setup +├── completion.md # CLI completion setup +├── cli-reference.md # Complete CLI reference +├── schema.md # YAML schema reference +├── examples/ # Example configurations +├── articles.md # External articles and tutorials +├── known-issues.md # Known limitations +├── contributing.md # This file +└── funding.md # Funding information +``` + +## 🏗️ Contributing New Structures + +STRUCT includes a collection of contrib structures in `struct_module/contribs/`. To add a new structure: + +1. **Create a new YAML file** in the appropriate subdirectory +2. **Follow naming conventions**: use lowercase with hyphens +3. **Test your structure** with various scenarios +4. **Add documentation** explaining what the structure does +5. **Include examples** in your PR description + +### Structure Guidelines + +- Use descriptive file and folder names +- Include appropriate comments in YAML +- Follow the established patterns in existing structures +- Test with different variable combinations +- Ensure compatibility across platforms + +## 🐛 Reporting Issues + +### Bug Reports + +When reporting bugs, please include: + +- **OS and Python version** +- **STRUCT version** (`struct --version`) +- **Complete error message** or traceback +- **Steps to reproduce** the issue +- **Expected vs actual behavior** +- **Sample configuration** that reproduces the issue + +### Feature Requests + +For feature requests: + +- **Describe the use case** clearly +- **Explain the benefit** to other users +- **Provide examples** of how it would work +- **Consider implementation complexity** + +## 📊 Project Structure + +```text +struct/ +├── struct_module/ # Main Python package +│ ├── commands/ # CLI command implementations +│ ├── contribs/ # Contributed structure templates +│ ├── __init__.py +│ ├── main.py # CLI entry point +│ ├── file_item.py # File handling logic +│ ├── template_renderer.py # Jinja2 template rendering +│ ├── content_fetcher.py # Remote content fetching +│ ├── model_wrapper.py # AI model integration +│ └── utils.py # Utility functions +├── tests/ # Test suite +├── docs/ # Documentation +├── examples/ # Example configurations +├── requirements.txt # Production dependencies +├── requirements.dev.txt # Development dependencies +├── setup.py # Package configuration +├── struct-schema.json # JSON schema for validation +└── README.md # Project overview +``` + +## 🏷️ Issue Labels + +We use labels to categorize issues: + +- **`bug`** - Something isn't working +- **`enhancement`** - New feature or request +- **`documentation`** - Improvements to documentation +- **`good first issue`** - Good for newcomers +- **`help wanted`** - Extra attention is needed +- **`question`** - Further information is requested + +## 💬 Communication + +- **GitHub Issues** - Bug reports and feature requests +- **GitHub Discussions** - General questions and community support +- **Pull Requests** - Code contributions and reviews + +## 🙏 Recognition + +All contributors will be recognized in our documentation and releases. We appreciate every contribution, whether it's: + +- Code improvements +- Bug reports +- Documentation updates +- Feature suggestions +- Community support + +## 📜 Code of Conduct + +This project follows the [Contributor Covenant Code of Conduct](https://www.contributor-covenant.org/version/2/1/code_of_conduct/). By participating, you are expected to uphold this code. + +## 🤝 Getting Help + +If you need help contributing: + +1. Check existing [documentation](index.md) +2. Search [existing issues](https://github.com/httpdss/struct/issues) +3. Create a new issue with the `question` label +4. Join our [GitHub Discussions](https://github.com/httpdss/struct/discussions) + +Thank you for contributing to STRUCT! 🎉 diff --git a/docs/css/style.css b/docs/css/style.css deleted file mode 100644 index 5514cd8..0000000 --- a/docs/css/style.css +++ /dev/null @@ -1,73 +0,0 @@ -body { - font-family: Arial, sans-serif; - margin: 0; - padding: 0; - display: flex; - flex-direction: column; - min-height: 100vh; - background-color: #1e1e1e; - color: #c5c6c7; -} - -.container { - width: 80%; - margin: 0 auto; -} - -header, -footer { - background-color: #0b0c10; - color: #66fcf1; - text-align: center; - padding: 1em 0; -} - -header h1 { - margin: 0; - font-size: 2.5em; -} - -header p { - margin: 0.5em 0 0; -} - -main { - flex: 1; - padding: 2em 0; -} - -section { - margin-bottom: 2em; -} - -h2 { - color: #66fcf1; -} - -a { - color: #45a29e; - text-decoration: none; -} - -a:hover { - text-decoration: underline; -} - -pre { - background-color: #0b0c10; - padding: 1em; - border-radius: 5px; - overflow-x: auto; -} - -footer p { - margin: 0.5em 0; -} - -footer a { - color: #66fcf1; -} - -footer a:hover { - color: #45a29e; -} diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..0768af8 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,264 @@ +# Development Setup + +This guide will help you set up a development environment for contributing to STRUCT. + +## Prerequisites + +- Python 3.8 or higher +- Git +- Virtual environment tool (venv, virtualenv, or conda) + +## Getting Started + +### 1. Clone the Repository + +```sh +git clone https://github.com/httpdss/struct.git +cd struct +``` + +### 2. Create a Virtual Environment + +```sh +python3 -m venv .venv +source .venv/bin/activate # On Windows: .venv\Scripts\activate +``` + +### 3. Install Dependencies + +```sh +# Install runtime dependencies +pip install -r requirements.txt + +# Install development dependencies +pip install -r requirements.dev.txt + +# Install the package in development mode +pip install -e . +``` + +### 4. Verify Installation + +```sh +struct --help +``` + +## Development Workflow + +### Running Tests + +```sh +# Run all tests +pytest + +# Run tests with coverage +pytest --cov=struct_module + +# Run specific test file +pytest tests/test_specific.py +``` + +### Code Quality + +This project uses several tools to maintain code quality: + +```sh +# Format code with black +black . + +# Sort imports +isort . + +# Lint with flake8 +flake8 . + +# Type checking with mypy +mypy struct_module/ +``` + +### Pre-commit Hooks + +Install pre-commit hooks to automatically run quality checks: + +```sh +pre-commit install +``` + +This will run formatting, linting, and tests before each commit. + +## Project Structure + +```text +struct/ +├── struct_module/ # Main Python package +│ ├── commands/ # CLI command implementations +│ ├── contribs/ # Built-in structure templates +│ └── ... +├── tests/ # Test files +├── docs/ # Documentation +├── example/ # Example configurations +├── scripts/ # Utility scripts +└── requirements*.txt # Dependencies +``` + +## Adding New Features + +### 1. Create a New Command + +Commands are defined in `struct_module/commands/`. Each command should: + +- Inherit from a base command class +- Include proper argument parsing +- Have comprehensive tests +- Include documentation + +Example: + +```python +# struct_module/commands/my_command.py +from .base import BaseCommand + +class MyCommand(BaseCommand): + def add_arguments(self, parser): + parser.add_argument('--option', help='My option') + + def handle(self, args): + # Implementation here + pass +``` + +### 2. Add Structure Templates + +New structure templates go in `struct_module/contribs/`. Each template should: + +- Have a clear directory structure +- Provide good documentation +- Include example usage + +### 3. Write Tests + +All new functionality must include tests: + +```python +# tests/test_my_feature.py +import pytest +from struct_module.my_feature import MyFeature + +def test_my_feature(): + feature = MyFeature() + result = feature.do_something() + assert result == expected_value +``` + +### 4. Update Documentation + +- Add or update relevant documentation in `docs/` +- Add examples if applicable + +## Testing + +### Unit Tests + +Run unit tests to verify individual components: + +```sh +pytest tests/unit/ +``` + +### Integration Tests + +Run integration tests to verify end-to-end functionality: + +```sh +pytest tests/integration/ +``` + +## Debugging + +### Enable Debug Logging + +```sh +struct --log=DEBUG generate file://my-config.yaml ./output +``` + +### Use Python Debugger + +Add breakpoints in your code: + +```python +import pdb; pdb.set_trace() +``` + +```sh +struct --log=DEBUG generate my-config.yaml ./output +``` + +## Contributing Guidelines + +### Code Style + +- Follow PEP 8 +- Use type hints where appropriate +- Write docstrings for public functions +- Keep functions small and focused + +### Commit Messages + +Use conventional commit format: + +```text +feat: add new template variable filter +fix: resolve issue with file permissions +docs: update installation guide +test: add tests for mappings functionality +``` + +### Pull Requests + +1. Fork the repository +2. Create a feature branch: `git checkout -b feature/my-feature` +3. Make your changes +4. Write tests +5. Update documentation +6. Submit a pull request + +### Review Process + +All pull requests must: + +- Pass all tests +- Include appropriate documentation +- Follow code style guidelines +- Have a clear description of changes + +## Troubleshooting + +### Common Issues + +**Import errors**: Make sure you've installed the package in development mode: + +```sh +pip install -e . +``` + +**Test failures**: Ensure all dependencies are installed: + +```sh +pip install -r requirements.dev.txt +``` + +**Permission errors**: Check file permissions and ensure you're in the right directory. + +### Getting Help + +- Check existing issues on GitHub +- Join our Discord community +- Read the documentation thoroughly +- Ask questions in discussions + +## Resources + +- [Python Development Guide](https://docs.python.org/3/tutorial/) +- [pytest Documentation](https://docs.pytest.org/) +- [GitHub Flow](https://guides.github.com/introduction/flow/) +- [Conventional Commits](https://www.conventionalcommits.org/) diff --git a/docs/examples/index.md b/docs/examples/index.md new file mode 100644 index 0000000..d85a266 --- /dev/null +++ b/docs/examples/index.md @@ -0,0 +1,77 @@ +# Examples + +This directory contains practical examples of STRUCT configurations for various use cases. + +## Basic Examples + +- [Simple Project](simple-project.yaml) - Basic project structure with files and folders +- [Template Variables](template-variables.yaml) - Using dynamic content with variables +- [Remote Files](remote-files.yaml) - Fetching content from external sources + +## Application Development + +- [Python Project](python-project.yaml) - Complete Python application structure +- [Node.js API](nodejs-api.yaml) - REST API with Express.js +- [React Frontend](react-frontend.yaml) - Modern React application setup +- [Microservice](microservice.yaml) - Containerized microservice template + +## Infrastructure + +- [Terraform Module](terraform-module.yaml) - AWS infrastructure module +- [Kubernetes Application](k8s-application.yaml) - Complete K8s deployment +- [Docker Multi-Stage](docker-multistage.yaml) - Multi-stage Docker setup +- [CI/CD Pipeline](cicd-pipeline.yaml) - GitHub Actions workflow + +## DevOps + +- [Monitoring Setup](monitoring.yaml) - Prometheus and Grafana configuration +- [GitOps Repository](gitops-repo.yaml) - ArgoCD application structure +- [Helm Chart](helm-chart.yaml) - Kubernetes Helm chart template + +## Advanced + +- [Multi-Environment](multi-environment.yaml) - Environment-specific configurations with mappings +- [Custom Hooks](custom-hooks.yaml) - Complex automation with pre/post hooks +- [Modular Structure](modular-structure.yaml) - Composable, reusable components + +## Usage + +Each example can be used directly: + +```bash +# Use an example from this directory +struct generate ./docs/examples/python-project.yaml ./my-project + +# Or reference the raw URL +struct generate https://raw.githubusercontent.com/httpdss/struct/main/docs/examples/python-project.yaml ./my-project +``` + +## Contributing Examples + +We welcome community examples! To contribute: + +1. Create a new `.yaml` file in this directory +2. Follow the naming convention: `descriptive-name.yaml` +3. Include comments explaining key concepts +4. Add the example to this index +5. Submit a pull request + +### Example Template + +```yaml +# Example: [Brief Description] +# Use case: [What this example demonstrates] +# Requirements: [Any prerequisites or dependencies] + +files: + - README.md: + content: | + # Example Project + This demonstrates [key concept] + +variables: + - example_var: + description: "Example variable" + type: string + default: "example_value" +``` diff --git a/docs/file-handling.md b/docs/file-handling.md new file mode 100644 index 0000000..6f7ab3b --- /dev/null +++ b/docs/file-handling.md @@ -0,0 +1,209 @@ +# File Handling + +STRUCT provides flexible options for handling files and managing permissions. + +## File Properties + +### Basic Properties + +- **skip**: Skip file/folder creation entirely +- **skip_if_exists**: Skip only if the file already exists +- **permissions**: Set custom file permissions +- **content**: Define file content inline +- **file**: Reference external file content + +### Skip Behavior + +```yaml +files: + - README.md: + skip: true # Never create this file + content: "This won't be created" + + - config.yml: + skip_if_exists: true # Only create if it doesn't exist + content: "default: value" +``` + +### File Permissions + +Set custom permissions using octal notation: + +```yaml +files: + - scripts/deploy.sh: + permissions: '0755' # Executable script + content: | + #!/bin/bash + echo "Deploying..." + + - secrets/api.key: + permissions: '0600' # Read-only for owner + content: "{{@ api_key @}}" +``` + +## Content Sources + +### Inline Content + +Define content directly in the YAML: + +```yaml +files: + - app.py: + content: | + #!/usr/bin/env python3 + print("Hello, {{@ project_name @}}!") +``` + +### External Files + +Reference local or remote files: + +```yaml +files: + - LICENSE: + file: https://raw.githubusercontent.com/nishanths/license/master/LICENSE +``` + +## Remote File Protocols + +STRUCT supports multiple protocols for fetching remote content: + +### HTTP/HTTPS + +```yaml +files: + - requirements.txt: + file: https://raw.githubusercontent.com/example/repo/main/requirements.txt +``` + +### GitHub Protocols + +#### Standard GitHub + +```yaml +files: + - .gitignore: + file: github://github/gitignore/main/Python.gitignore +``` + +#### GitHub HTTPS + +```yaml +files: + - workflow.yml: + file: githubhttps://actions/starter-workflows/main/ci/python-app.yml +``` + +#### GitHub SSH + +```yaml +files: + - private-config.yml: + file: githubssh://company/private-repo/main/config/template.yml +``` + +### Cloud Storage + +#### Amazon S3 + +```yaml +files: + - data.csv: + file: s3://my-bucket/data/template.csv +``` + +#### Google Cloud Storage + +```yaml +files: + - config.json: + file: gs://my-bucket/configs/default.json +``` + +## File Handling Strategies + +Control how STRUCT handles existing files with the `--file-strategy` option: + +### Available Strategies + +- **overwrite**: Replace existing files (default) +- **skip**: Skip files that already exist +- **append**: Add content to existing files +- **rename**: Rename existing files with a suffix +- **backup**: Move existing files to backup directory + +### Usage Examples + +```sh +# Skip existing files +struct generate --file-strategy=skip file://my-config.yaml ./output + +# Backup existing files +struct generate --file-strategy=backup --backup=/tmp/backup file://my-config.yaml ./output + +# Rename existing files +struct generate --file-strategy=rename file://my-config.yaml ./output +``` + +## Advanced Examples + +### Conditional File Creation + +```yaml +files: + - docker-compose.yml: + skip: "{{@ not use_docker @}}" + content: | + version: '3.8' + services: + app: + build: . + + - Dockerfile: + skip_if_exists: true + permissions: '0644' + file: https://raw.githubusercontent.com/example/dockerfiles/main/python.Dockerfile +``` + +### Dynamic Permissions + +```yaml +files: + - "scripts/{{@ script_name @}}.sh": + permissions: '0755' + content: | + #!/bin/bash + # {{@ script_description @}} + echo "Running {{@ script_name @}}" +``` + +### Multiple Content Sources + +```yaml +files: + - README.md: + content: | + # {{@ project_name @}} + + ## Installation + + ```bash + pip install -r requirements.txt + ``` + + - requirements.txt: + file: https://raw.githubusercontent.com/example/templates/main/python-requirements.txt + + - .env.example: + file: file://./templates/env-template +``` + +## Best Practices + +1. **Use `skip_if_exists`** for configuration files that shouldn't be overwritten +2. **Set appropriate permissions** for scripts and sensitive files +3. **Use remote files** for standardized content like licenses and gitignores +4. **Combine strategies** with command-line options for different deployment scenarios +5. **Test with `--dry-run`** before applying changes to important directories diff --git a/docs/funding.md b/docs/funding.md new file mode 100644 index 0000000..55e086f --- /dev/null +++ b/docs/funding.md @@ -0,0 +1,67 @@ +# Funding STRUCT + +STRUCT is an open-source project that helps developers create consistent project structures and automate repetitive setup tasks. Your support helps us maintain and improve this tool for the entire community. + +## 💰 Support Options + +### GitHub Sponsors + +The easiest way to support STRUCT development is through GitHub Sponsors: + +- **[Sponsor @httpdss on GitHub](https://github.com/sponsors/httpdss)** + +GitHub Sponsors allows for recurring monthly contributions or one-time donations. All funds go directly toward: + +- Development time for new features +- Bug fixes and maintenance +- Documentation improvements +- Community support +- Infrastructure costs + +### Patreon + +For ongoing support with additional perks: + +- **[Support STRUCT on Patreon](https://patreon.com/structproject)** + +Patreon supporters get access to: + +- Early previews of new features +- Direct input on roadmap priorities +- Exclusive development updates +- Recognition in project documentation + +### One-Time Donations + +For one-time contributions: + +- **PayPal**: [Donate via PayPal](https://paypal.me/httpdss) + +## 🤝 Alternative Ways to Support + +If financial contribution isn't possible, you can still support STRUCT: + +### Code Contributions + +- Submit bug fixes and new features +- Improve documentation +- Add new contrib structures +- Help with testing and quality assurance + +### Community Support + +- Answer questions in GitHub Discussions +- Help other users in issues +- Share STRUCT on social media +- Write blog posts or tutorials + +### Feedback and Testing + +- Report bugs and issues +- Suggest new features +- Test beta releases +- Provide user experience feedback + +--- + +Every contribution, no matter the size, helps make STRUCT better for everyone. Thank you for your support! 🎉 diff --git a/docs/github-integration.md b/docs/github-integration.md new file mode 100644 index 0000000..fd84322 --- /dev/null +++ b/docs/github-integration.md @@ -0,0 +1,110 @@ +# GitHub Integration + +STRUCT can seamlessly integrate with GitHub to automate the generation of project structures across repositories. This documentation explains how to set up and use the GitHub trigger script. + +## GitHub Trigger Script + +The `github-trigger.py` script is a utility designed to trigger the `run-struct` workflow for all private repositories in a GitHub organization that meet specific criteria. This script is particularly useful for automating tasks across multiple repositories. + +### Features + +- Filters repositories by a specific topic (e.g., `struct-enabled`). +- Checks for the existence of a `.struct.yaml` file in the repository's default branch. +- Verifies the presence of the `run-struct` workflow file in `.github/workflows/`. +- Triggers the workflow dispatch event for eligible repositories. + +### Usage + +To use the script, ensure you have the following prerequisites: + +1. A valid GitHub Personal Access Token with the necessary permissions (set as the `GITHUB_TOKEN` environment variable). +2. The `PyGithub` library installed (`pip install PyGithub`). + +Run the script with the following command: + +```sh +python3 scripts/github-trigger.py +``` + +#### Arguments + +- ``: The name of the GitHub organization. +- ``: The topic to filter repositories by (e.g., `struct-enabled`). + +#### Example + +```sh +export GITHUB_TOKEN=your_personal_access_token +python3 scripts/github-trigger.py my-org struct-enabled +``` + +### How It Works + +1. The script connects to the GitHub API using the provided token. +2. It iterates through all private repositories in the specified organization. +3. For each repository: + - Checks if the repository has the specified topic. + - Verifies the existence of a `.struct.yaml` file in the default branch. + - Confirms the presence of the `run-struct` workflow file. + - Triggers the workflow dispatch event if all conditions are met. + +### Notes + +- Ensure the `GITHUB_TOKEN` environment variable is set before running the script. +- The token must have sufficient permissions to access private repositories and trigger workflows. +- Errors during execution (e.g., missing files or insufficient permissions) will be logged to the console. + +### Advanced Usage + +Add additional parameters to customize the script's behavior: + +- `--dry-run`: Simulate the script actions without executing any API calls. +- `--verbose`: Enable verbose output for debugging purposes. + +Example: + +```sh +python3 scripts/github-trigger.py my-org struct-enabled --dry-run --verbose +``` + +### Troubleshooting + +- **Unauthorized Error**: Make sure your token has the appropriate permissions. +- **Repository Not Found**: Ensure the organization name is correct and the token has access. +- **Missing Files**: Verify that the `.struct.yaml` and workflow files exist in each repository. + +## Automating STRUCT + +Combine GitHub Actions with STRUCT to automate project structure generation in CI/CD pipelines. Trigger the process manually or automatically based on events like pull requests or pushes. + +Example Workflow: + +```yaml +name: run-struct + +on: + workflow_dispatch: + +jobs: + generate: + uses: httpdss/struct/.github/workflows/struct-generate.yaml@main + secrets: + token: ${{ secrets.STRUCT_RUN_TOKEN }} +``` + +## Best Practices + +1. **Secure Your Token**: Store GitHub tokens in secrets management tools. +2. **Use Topics for Filtering**: Organize repositories with topics to efficiently manage workflows. +3. **Test Locally First**: Simulate script actions with `--dry-run` before executing in a CI/CD environment. +4. **Combine with Other Tools**: Use STRUCT with Terraform, Ansible, or Docker for comprehensive project management. + +## FAQs + +### Why use a GitHub Integration? + +Using GitHub integration allows you to leverage STRUCT's automation capabilities in a version-controlled environment, enabling consistent and repeatable project structures. + +### Can I customize the GitHub Action? + +Yes, you can tailor the GitHub Action to your specific needs, including custom triggers, different environments, and additional dependencies. diff --git a/docs/hooks.md b/docs/hooks.md new file mode 100644 index 0000000..c5389fa --- /dev/null +++ b/docs/hooks.md @@ -0,0 +1,220 @@ +# Pre-generation and Post-generation Hooks + +You can define shell commands to run before and after structure generation using the `pre_hooks` and `post_hooks` keys in your YAML configuration. These are optional and allow you to automate setup or cleanup steps. + +## Hook Types + +- **pre_hooks**: List of shell commands to run before generation. If any command fails (non-zero exit), generation is aborted. +- **post_hooks**: List of shell commands to run after generation completes. If any command fails, an error is shown. + +## Basic Example + +```yaml +pre_hooks: + - echo "Preparing environment..." + - ./scripts/prep.sh + +post_hooks: + - echo "Generation complete!" + - ./scripts/cleanup.sh + +files: + - README.md: + content: | + # My Project +``` + +## How Hooks Work + +### Pre-hooks + +1. Execute in the order defined +2. Run before any files or folders are created +3. If any command returns non-zero exit code, generation stops +4. Useful for environment preparation, validation, or dependency checks + +### Post-hooks + +1. Execute after all files and folders are created +2. Run even if some file operations fail +3. Errors are reported but don't stop execution +4. Useful for cleanup, initialization, or notification tasks + +## Output Handling + +- Output from hooks (stdout and stderr) is shown in the terminal +- Hook execution is logged with appropriate log levels +- Failed hooks display error messages with exit codes + +## Practical Examples + +### Environment Setup + +```yaml +pre_hooks: + - python -m venv .venv + - source .venv/bin/activate + - pip install --upgrade pip + +post_hooks: + - source .venv/bin/activate + - pip install -r requirements.txt + - echo "Virtual environment ready!" + +files: + - requirements.txt: + content: | + flask==2.3.0 + requests==2.31.0 +``` + +### Git Repository Initialization + +```yaml +pre_hooks: + - git --version # Verify git is available + +post_hooks: + - git init + - git add . + - git commit -m "Initial commit from STRUCT" + - echo "Git repository initialized" + +files: + - .gitignore: + file: github://github/gitignore/main/Python.gitignore + - README.md: + content: | + # {{@ project_name @}} + Generated with STRUCT +``` + +### Docker Setup + +```yaml +pre_hooks: + - docker --version + - echo "Setting up Docker environment..." + +post_hooks: + - docker build -t {{@ project_name | slugify @}} . + - echo "Docker image built successfully" + +files: + - Dockerfile: + content: | + FROM python:3.11-slim + WORKDIR /app + COPY . . + RUN pip install -r requirements.txt + CMD ["python", "app.py"] + - docker-compose.yml: + content: | + version: '3.8' + services: + app: + build: . + ports: + - "8000:8000" +``` + +### Database Migration + +```yaml +pre_hooks: + - echo "Checking database connection..." + - pg_isready -h localhost -p 5432 + +post_hooks: + - python manage.py migrate + - python manage.py collectstatic --noinput + - echo "Database migrations complete" + +files: + - manage.py: + content: | + #!/usr/bin/env python + import os + import sys + + if __name__ == "__main__": + os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings") + from django.core.management import execute_from_command_line + execute_from_command_line(sys.argv) +``` + +### Testing and Validation + +```yaml +pre_hooks: + - echo "Validating prerequisites..." + - node --version + - npm --version + +post_hooks: + - npm install + - npm run lint + - npm test + - echo "All tests passed!" + +files: + - package.json: + content: | + { + "name": "{{@ project_name | slugify @}}", + "version": "1.0.0", + "scripts": { + "test": "jest", + "lint": "eslint src/" + } + } +``` + +## Best Practices + +1. **Keep hooks simple**: Use external scripts for complex operations +2. **Handle errors gracefully**: Check for tool availability in pre-hooks +3. **Use absolute paths**: Hooks run in the target directory context +4. **Log important actions**: Use echo statements for user feedback +5. **Test independently**: Ensure hook commands work outside STRUCT +6. **Consider dependencies**: Order hooks based on their requirements + +## Error Handling + +### Pre-hook Failures + +```yaml +pre_hooks: + - echo "Checking Python version..." + - python --version || (echo "Python not found!" && exit 1) + - echo "Python check passed" +``` + +### Post-hook Error Tolerance + +```yaml +post_hooks: + - npm install || echo "Warning: npm install failed" + - git add . || echo "Warning: git add failed" + - echo "Setup complete (some warnings may have occurred)" +``` + +## Variables in Hooks + +You can use template variables in hook commands: + +```yaml +pre_hooks: + - echo "Setting up {{@ project_name @}}" + - mkdir -p "{{@ project_name | slugify @}}" + +post_hooks: + - echo "{{@ project_name @}} setup complete!" + - echo "Project created in: $(pwd)" + +variables: + - project_name: + description: "Name of the project" + type: string + default: "MyProject" +``` diff --git a/docs/index.html b/docs/index.html deleted file mode 100644 index 89274a9..0000000 --- a/docs/index.html +++ /dev/null @@ -1,59 +0,0 @@ - - - - - - - Struct - Project Structure Generator - - - - - -
-
-

Struct

-

Your versatile project structure generator

-
-
-
-
-

Introduction

-

Struct is a powerful tool to automate the creation of project structures based on YAML configurations. It - supports template variables, OpenAI integration for generating content, and various file handling - strategies to customize your project setup.

-
-
-

Features

-
    -
  • Define project structure using YAML
    • -
  • Support for template variables
    • -
  • Integration with OpenAI for content generation
    • -
  • Flexible file handling strategies (overwrite, skip, append, rename, backup)
    • -
  • Modular YAML inclusion for nested structures
    • -
  • Dry-run mode to preview changes without making them
    • -
-
-
-

Documentation

-

For detailed documentation, please visit our README.md file.

-

If you prefer documentation in spanish language, you can head to README.es.md

-
-
-

Contribute

-

We welcome contributions from the community! Please read our contributing - guidelines to get started.

-
-
-
-
-

© 2024 Struct Project

-

GitHub -

-
-
- - - - diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..9e24cda --- /dev/null +++ b/docs/index.md @@ -0,0 +1,47 @@ +# STRUCT Documentation + +Welcome to the comprehensive documentation for STRUCT - the Automated Project Structure Generator. + +## 📚 Documentation Sections + +### Getting Started + +- [Installation Guide](installation.md) - How to install STRUCT +- [Quick Start](quickstart.md) - Get up and running quickly +- [Basic Usage](usage.md) - Core functionality and commands + +### Configuration + +- [YAML Configuration](configuration.md) - Complete configuration reference +- [Template Variables](template-variables.md) - Working with dynamic content +- [File Handling](file-handling.md) - Managing files and permissions + +### Advanced Features + +- [Custom Structures](custom-structures.md) - Creating your own templates +- [Hooks](hooks.md) - Pre and post-generation automation +- [Mappings](mappings.md) - Using external data mappings +- [GitHub Integration](github-integration.md) - Automation with GitHub Actions + +### Development + +- [Development Setup](development.md) - Contributing to STRUCT +- [Command-Line Completion](completion.md) - Auto-completion setup + +### Reference + +- [CLI Reference](cli-reference.md) - Complete command documentation +- [Schema Reference](schema.md) - YAML validation and IDE support +- [Examples](examples/) - Practical examples and use cases + +## 🔗 External Resources + +- [GitHub Repository](https://github.com/httpdss/struct) +- [Articles and Tutorials](articles.md) +- [Known Issues](known-issues.md) + +## 🤝 Community + +- [Contributing Guidelines](contributing.md) +- [License](../LICENSE) +- [Funding](funding.md) diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 0000000..413c6b5 --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,23 @@ +# Installation Guide + +## Using pip + +Install STRUCT with pip: + +```sh +pip install git+https://github.com/httpdss/struct.git +``` + +## From Source + +Clone the repository and install locally. See the [Development](development.md) page for details. + +## Using Docker + +Run STRUCT without installing, using Docker: + +```sh +docker run -v $(pwd):/workdir -u $(id -u):$(id -g) ghcr.io/httpdss/struct:main generate file:///workdir/example/structure.yaml /workdir/example_output +``` + +Refer to the [Quick Start](quickstart.md) guide for more options. diff --git a/docs/js/script.js b/docs/js/script.js deleted file mode 100644 index 053f3d7..0000000 --- a/docs/js/script.js +++ /dev/null @@ -1,3 +0,0 @@ -document.addEventListener('DOMContentLoaded', function() { - console.log('JavaScript is running!'); -}); diff --git a/docs/known-issues.md b/docs/known-issues.md new file mode 100644 index 0000000..7e74afd --- /dev/null +++ b/docs/known-issues.md @@ -0,0 +1,295 @@ +# Known Issues + +This page tracks known issues, limitations, and workarounds for STRUCT. If you encounter any of these issues, check here for potential solutions. + +## Current Known Issues + +### Performance Issues + +#### Large Directory Structures + +**Issue**: Generation becomes slow with very large directory structures (1000+ files). + +**Workaround**: + +- Break large structures into smaller, modular components +- Use the `--dry-run` option to preview before full generation +- Consider using parallel processing for independent structures + +**Status**: Under investigation + +#### Remote File Fetching + +**Issue**: Fetching many remote files can be slow due to sequential processing. + +**Workaround**: + +- Cache frequently used remote files locally +- Use local mirrors when possible +- Batch remote operations when feasible + +**Status**: Planned for future release + +### Template Engine + +#### Complex Jinja2 Expressions + +**Issue**: Very complex Jinja2 expressions may not render correctly in all contexts. + +**Example**: + +```yaml +# May not work as expected +content: "{{@ complex | filter | another_filter if condition else fallback @}}" +``` + +**Workaround**: Break complex expressions into simpler parts or use variables: + +```yaml +variables: + - intermediate_value: + default: "{{@ complex | filter @}}" + +files: + - test.txt: + content: "{{@ intermediate_value | another_filter @}}" +``` + +**Status**: Investigating improvements + +#### Unicode in File Names + +**Issue**: File names with unicode characters may cause issues on some filesystems. + +**Workaround**: Use ASCII-safe file names or test thoroughly on target systems. + +**Status**: Platform-specific limitation + +### Platform-Specific Issues + +#### Windows Path Handling + +**Issue**: Long path names on Windows may exceed the 260-character limit. + +**Workaround**: + +- Use shorter path names +- Enable long path support in Windows 10/11 +- Use mapped drives for shorter paths + +**Status**: Windows limitation + +#### macOS Case Sensitivity + +**Issue**: macOS filesystem case-insensitivity can cause conflicts with similarly named files. + +**Workaround**: Ensure file names differ by more than just case. + +**Status**: Platform limitation + +### GitHub Integration + +#### Rate Limiting + +**Issue**: GitHub API rate limits may affect custom filters like `latest_release`. + +**Workaround**: + +- Set `GITHUB_TOKEN` environment variable for higher limits +- Cache results when possible +- Implement retry logic in scripts + +**Status**: API limitation + +#### Private Repository Access + +**Issue**: Accessing private repositories requires proper authentication. + +**Workaround**: Ensure `GITHUB_TOKEN` has appropriate permissions. + +**Status**: Expected behavior + +### Docker Issues + +#### File Permissions + +**Issue**: Files created in Docker containers may have incorrect ownership. + +**Workaround**: Use the `-u $(id -u):$(id -g)` option when running Docker: + +```sh +docker run -v $(pwd):/workdir -u $(id -u):$(id -g) ghcr.io/httpdss/struct:main generate file://config.yaml . +``` + +**Status**: Docker behavior + +#### Volume Mounting + +**Issue**: Complex volume mounting scenarios may not work as expected. + +**Workaround**: Use simpler mounting patterns and absolute paths. + +**Status**: Docker limitation + +## Limitations + +### Template Variables + +#### Variable Scope + +**Limitation**: Variables defined in nested structures don't propagate back to parent structures. + +**Design Decision**: This is intentional to maintain structure isolation. + +#### Circular References + +**Limitation**: Circular references in variable definitions are not detected. + +**Example**: + +```yaml +variables: + - var_a: + default: "{{@ var_b @}}" + - var_b: + default: "{{@ var_a @}}" +``` + +**Impact**: May cause infinite loops or unexpected behavior. + +### File Operations + +#### Atomic Operations + +**Limitation**: File operations are not atomic - partial failures may leave the system in an inconsistent state. + +**Mitigation**: Use the `--backup` option for important operations. + +#### Symlink Handling + +**Limitation**: Symbolic links are not explicitly handled and may be followed or replaced. + +**Impact**: May affect complex directory structures with symlinks. + +### Configuration + +#### YAML Complexity + +**Limitation**: Very large YAML files (>10MB) may cause performance issues. + +**Recommendation**: Split large configurations into multiple files. + +#### Schema Validation + +**Limitation**: Schema validation doesn't catch all runtime errors. + +**Note**: Use `struct validate` and testing for comprehensive validation. + +## Workarounds and Best Practices + +### Performance Optimization + +1. **Use `--dry-run`** for testing large configurations +2. **Modularize structures** instead of creating monolithic ones +3. **Cache remote content** locally when possible +4. **Use specific structure paths** instead of recursive searches + +### Error Prevention + +1. **Validate configurations** before deployment +2. **Test with small datasets** first +3. **Use version control** for all configuration files +4. **Implement backup strategies** for important operations + +### Cross-Platform Compatibility + +1. **Test on target platforms** before deployment +2. **Use relative paths** when possible +3. **Avoid platform-specific commands** in hooks +4. **Document platform requirements** clearly + +## Reporting Issues + +### Before Reporting + +1. Check this known issues page +2. Search existing GitHub issues +3. Test with the latest version +4. Prepare a minimal reproduction case + +### Issue Report Template + +```markdown +**STRUCT Version**: [e.g., 1.2.3] +**Platform**: [e.g., Ubuntu 20.04, macOS 12.0, Windows 11] +**Python Version**: [e.g., 3.9.7] + +**Expected Behavior**: +[Describe what you expected to happen] + +**Actual Behavior**: +[Describe what actually happened] + +**Reproduction Steps**: +1. [Step 1] +2. [Step 2] +3. [Step 3] + +**Configuration File**: + +```yaml +[Paste minimal configuration that reproduces the issue] +``` + +**Error Output**: + +```text +[Paste any error messages or logs] +``` + +**Additional Context**: + +[Any other relevant information] + +```text + +### Priority Guidelines + +- **Critical**: Security issues, data loss, system crashes +- **High**: Major functionality broken, widespread impact +- **Medium**: Minor functionality issues, workarounds available +- **Low**: Enhancement requests, documentation issues + +## Contributing Fixes + +We welcome contributions to fix known issues: + +1. **Fork the repository** +2. **Create a feature branch**: `git checkout -b fix/issue-name` +3. **Write tests** for your fix +4. **Update documentation** if necessary +5. **Submit a pull request** + +### Testing Requirements + +- All existing tests must pass +- New tests should cover the fixed functionality +- Manual testing on multiple platforms when applicable + +## Version Compatibility + +### Supported Versions + +- **Latest release**: Full support and active development +- **Previous minor version**: Security fixes and critical bugs +- **Older versions**: Community support only + +### Upgrade Recommendations + +- Test upgrades in non-production environments first +- Review changelog for breaking changes +- Update configurations for deprecated features + +--- + +*This page is updated regularly. Subscribe to repository notifications for updates on issue status.* diff --git a/docs/mappings.md b/docs/mappings.md new file mode 100644 index 0000000..20545c4 --- /dev/null +++ b/docs/mappings.md @@ -0,0 +1,295 @@ +# Mappings Support + +You can provide a mappings YAML file to inject key-value maps into your templates. This is useful for referencing environment-specific values, IDs, or any other mapping you want to use in your generated files. + +## What are Mappings? + +Mappings are external data files that provide structured key-value pairs for use in your STRUCT templates. They allow you to: + +- Separate data from templates +- Share common values across multiple structures +- Manage environment-specific configurations +- Centralize reference data like account IDs, team names, or service endpoints + +## Mappings File Format + +### Basic Structure + +```yaml +mappings: + teams: + devops: devops-team + frontend: frontend-team + backend: backend-team + + aws_account_ids: + development: 123456789012 + staging: 234567890123 + production: 345678901234 + + service_endpoints: + api_gateway: https://api.example.com + database: postgres://db.example.com:5432 + redis: redis://cache.example.com:6379 +``` + +### Nested Mappings + +```yaml +mappings: + environments: + development: + database_url: postgres://dev-db:5432/myapp + redis_url: redis://dev-cache:6379 + debug: true + + production: + database_url: postgres://prod-db:5432/myapp + redis_url: redis://prod-cache:6379 + debug: false + + regions: + us_east_1: + vpc_id: vpc-12345 + subnet_ids: + - subnet-abc123 + - subnet-def456 + + eu_west_1: + vpc_id: vpc-67890 + subnet_ids: + - subnet-ghi789 + - subnet-jkl012 +``` + +## Using Mappings in Templates + +### Basic Usage + +Reference mapping values using the `mappings` variable: + +```yaml +files: + - config.yml: + content: | + database_url: {{@ mappings.service_endpoints.database @}} + api_endpoint: {{@ mappings.service_endpoints.api_gateway @}} + + # Team information + owner: {{@ mappings.teams.backend @}} +``` + +### Array Access + +For nested structures and arrays: + +```yaml +files: + - terraform/main.tf: + content: | + resource "aws_instance" "web" { + ami = "ami-12345" + instance_type = "t3.micro" + + vpc_security_group_ids = [ + "{{@ mappings.regions.us_east_1.vpc_id @}}" + ] + + subnet_id = "{{@ mappings.regions.us_east_1.subnet_ids[0] @}}" + } +``` + +### Dynamic Key Access + +Use bracket notation for dynamic keys: + +```yaml +files: + - app.py: + content: | + import os + + ENVIRONMENT = "{{@ environment @}}" + DATABASE_URL = "{{@ mappings.environments[environment].database_url @}}" + DEBUG = {{@ mappings.environments[environment].debug @}} + +variables: + - environment: + description: "Target environment" + type: string + default: "development" +``` + +## Using Mappings in the `with` Clause + +You can assign values from mappings directly in the `with` clause for folder struct calls: + +```yaml +folders: + - ./infrastructure: + struct: terraform/aws-vpc + with: + vpc_id: {{@ mappings.regions.us_east_1.vpc_id @}} + subnet_ids: {{@ mappings.regions.us_east_1.subnet_ids @}} + + - ./backend: + struct: project/django + with: + team: {{@ mappings.teams.backend @}} + database_url: {{@ mappings.environments.production.database_url @}} +``` + +This approach allows you to pass specific mapping values as variables to nested structures. + +## Command Line Usage + +Use the `--mappings-file` argument with the `generate` command: + +```sh +struct generate --mappings-file ./mymap.yaml file://my-struct.yaml . +``` + +### Multiple Mappings Files + +You can specify multiple mappings files that will be merged: + +```sh +struct generate \ + --mappings-file ./common-mappings.yaml \ + --mappings-file ./env-specific-mappings.yaml \ + file://my-struct.yaml . +``` + +Later files override earlier ones for conflicting keys. + +## Practical Examples + +### Multi-Environment Deployment + +**mappings.yaml:** + +```yaml +mappings: + environments: + dev: + namespace: myapp-dev + replicas: 1 + image_tag: latest + resources: + cpu: 100m + memory: 128Mi + + prod: + namespace: myapp-prod + replicas: 3 + image_tag: v1.2.3 + resources: + cpu: 500m + memory: 512Mi +``` + +**k8s-deployment.yaml template:** + +```yaml +files: + - k8s/deployment.yaml: + content: | + apiVersion: apps/v1 + kind: Deployment + metadata: + name: {{@ app_name @}} + namespace: {{@ mappings.environments[target_env].namespace @}} + spec: + replicas: {{@ mappings.environments[target_env].replicas @}} + template: + spec: + containers: + - name: app + image: myapp:{{@ mappings.environments[target_env].image_tag @}} + resources: + requests: + cpu: {{@ mappings.environments[target_env].resources.cpu @}} + memory: {{@ mappings.environments[target_env].resources.memory @}} + +variables: + - target_env: + description: "Target environment (dev/prod)" + type: string + default: "dev" + - app_name: + description: "Application name" + type: string +``` + +### Team-Specific Configurations + +**teams.yaml:** + +```yaml +mappings: + teams: + platform: + email: platform@company.com + slack: "#platform-team" + oncall: platform-oncall@company.com + + data: + email: data@company.com + slack: "#data-team" + oncall: data-oncall@company.com +``` + +**Usage:** + +```yaml +folders: + - ./services/api: + struct: service/rest-api + with: + team_email: {{@ mappings.teams.platform.email @}} + team_slack: {{@ mappings.teams.platform.slack @}} + + - ./services/etl: + struct: service/data-pipeline + with: + team_email: {{@ mappings.teams.data.email @}} + team_slack: {{@ mappings.teams.data.slack @}} +``` + +## Best Practices + +1. **Organize by Context**: Group related mappings together (e.g., environments, teams, regions) + +2. **Use Consistent Naming**: Follow naming conventions across all mapping files + +3. **Version Control**: Keep mappings files in version control alongside your structures + +4. **Validate Data**: Ensure mapping values are correct before generating structures + +5. **Document Mappings**: Include comments or separate documentation for complex mappings + +6. **Environment Separation**: Use separate mapping files for different environments + +7. **Default Values**: Provide sensible defaults in your templates for missing mappings + +## Error Handling + +If a mapping key doesn't exist, STRUCT will show an error: + +```yaml +files: + - config.yml: + content: | + # This will cause an error if 'nonexistent' key doesn't exist + value: {{@ mappings.nonexistent.key @}} +``` + +Use the `default` filter to provide fallbacks: + +```yaml +files: + - config.yml: + content: | + # This provides a fallback value + value: {{@ mappings.team.devops | default "no_team" @}} +``` diff --git a/docs/quickstart.md b/docs/quickstart.md new file mode 100644 index 0000000..5919af3 --- /dev/null +++ b/docs/quickstart.md @@ -0,0 +1,60 @@ +# Quick Start + +## Quick Start Using Docker + +1. Create a YAML configuration file for your project structure. [See sample configuration here](../example/structure.yaml). +2. Run the following command to generate the project structure: + +```sh +docker run \ + -v $(pwd):/workdir \ + -u $(id -u):$(id -g) \ + ghcr.io/httpdss/struct:main generate \ + file:///workdir/example/structure.yaml \ + /workdir/example_output +``` + +## Quick Start Using Docker Alpine + +```sh +docker run \ + -v $(pwd):/workdir \ + -u $(id -u):$(id -g) \ + ghcr.io/httpdss/struct:alpine generate \ + file:///workdir/example/structure.yaml \ + /workdir/example_output +``` + +For testing, you can run an alpine Docker container and install the script inside it: + +```sh +docker run -it --entrypoint="" python:3.10-alpine sh -l +``` + +Inside the container: + +```sh +apk add python-pip git vim +pip install git+https://github.com/httpdss/struct.git +mkdir example +cd example/ +touch structure.yaml +vim structure.yaml # copy the content from the example folder +struct generate file://structure.yaml . +``` + +## First Example + +After installing STRUCT, try this simple example: + +```sh +struct generate terraform-module ./my-terraform-module +``` + +This will create a new terraform module structure in the `./my-terraform-module` directory. + +## Next Steps + +- Learn about [YAML Configuration](configuration.md) +- Explore [Template Variables](template-variables.md) +- Check out [Usage Examples](usage.md) diff --git a/docs/schema.md b/docs/schema.md new file mode 100644 index 0000000..5cbabe7 --- /dev/null +++ b/docs/schema.md @@ -0,0 +1,265 @@ +# YAML Schema Reference + +STRUCT provides JSON schema validation to ensure your YAML configuration files are correctly structured. This helps catch errors early and provides IDE support with autocompletion. + +## Schema Location + +The official schema is available at: + +```text +https://raw.githubusercontent.com/httpdss/struct/main/struct-schema.json +``` + +## IDE Configuration + +### VS Code + +1. Install the [YAML extension](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml) +2. Add this to your workspace settings (`.vscode/settings.json`): + +```json +{ + "yaml.schemas": { + "https://raw.githubusercontent.com/httpdss/struct/main/struct-schema.json": ".struct.yaml" + } +} +``` + +This provides validation and autocompletion for all `.struct.yaml` files. + +### JetBrains IDEs (IntelliJ, PyCharm, etc.) + +1. Go to **Settings** → **Languages & Frameworks** → **Schemas and DTDs** → **JSON Schema Mappings** +2. Click **+** to add a new mapping +3. Set **Schema file or URL** to: `https://raw.githubusercontent.com/httpdss/struct/main/struct-schema.json` +4. Set **File path pattern** to: `*.struct.yaml` + +## Generating Custom Schema + +If you have custom structures, generate a schema that includes them: + +```sh +# Generate schema with custom structures +struct generate-schema -s /path/to/custom/structures -o my-schema.json + +# Use in VS Code settings +{ + "yaml.schemas": { + "./my-schema.json": ".struct.yaml" + } +} +``` + +## Schema Structure + +The schema validates the following top-level properties: + +### `files` (array) + +Defines files to be created: + +```yaml +files: + - filename.txt: + content: "File contents" + permissions: "0644" + skip: false + skip_if_exists: false + file: "https://example.com/template.txt" +``` + +**Properties:** + +- `content` (string): Inline file content +- `permissions` (string): Octal permissions (e.g., "0755") +- `skip` (boolean): Skip file creation +- `skip_if_exists` (boolean): Skip if file exists +- `file` (string): External file URL or path + +### `folders` (array) + +Defines folders and nested structures: + +```yaml +folders: + - path/to/folder: + struct: "structure-name" + with: + variable: "value" +``` + +**Properties:** + +- `struct` (string|array): Structure name(s) to apply +- `with` (object): Variables to pass to the structure + +### `variables` (array) + +Defines template variables: + +```yaml +variables: + - variable_name: + description: "Variable description" + type: "string" + default: "default_value" +``` + +**Properties:** + +- `description` (string): Human-readable description +- `type` (string): Variable type (string, integer, boolean) +- `default` (any): Default value + +### `pre_hooks` (array) + +Shell commands to run before generation: + +```yaml +pre_hooks: + - "echo 'Starting generation...'" + - "./scripts/prepare.sh" +``` + +### `post_hooks` (array) + +Shell commands to run after generation: + +```yaml +post_hooks: + - "npm install" + - "git init" +``` + +## Validation + +### Command Line Validation + +```sh +# Validate a configuration file +struct validate my-config.yaml +``` + +### Programmatic Validation + +```python +import json +import yaml +from jsonschema import validate + +# Load schema +with open('struct-schema.json') as f: + schema = json.load(f) + +# Load and validate config +with open('my-config.yaml') as f: + config = yaml.safe_load(f) + +validate(config, schema) # Raises exception if invalid +``` + +## Common Validation Errors + +### Invalid File Structure + +```yaml +# ❌ Wrong - files should be array of objects +files: + README.md: "content" + +# ✅ Correct +files: + - README.md: + content: "content" +``` + +### Missing Required Properties + +```yaml +# ❌ Wrong - struct property missing +folders: + - src/: + with: + name: "myapp" + +# ✅ Correct +folders: + - src/: + struct: "project/node" + with: + name: "myapp" +``` + +### Invalid Variable Types + +```yaml +# ❌ Wrong - type should be string +variables: + - port: + type: number + default: 8080 + +# ✅ Correct +variables: + - port: + type: integer + default: 8080 +``` + +## Schema Extensions + +You can extend the base schema for custom validation: + +```json +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "allOf": [ + { + "$ref": "https://raw.githubusercontent.com/httpdss/struct/main/struct-schema.json" + }, + { + "properties": { + "custom_section": { + "type": "object", + "properties": { + "custom_property": {"type": "string"} + } + } + } + } + ] +} +``` + +## IDE Benefits + +With schema validation enabled, you get: + +- **Real-time validation**: Errors highlighted as you type +- **Autocompletion**: Suggested properties and values +- **Documentation**: Hover tooltips with property descriptions +- **Structure guidance**: Valid structure names and paths + +## Troubleshooting + +### Schema Not Loading + +1. Check internet connection (for remote schema) +2. Verify file path (for local schema) +3. Restart IDE after configuration changes +4. Check IDE logs for error messages + +### Validation Errors + +1. Use `struct validate` command for detailed error messages +2. Check schema documentation for required properties +3. Verify YAML syntax is correct +4. Ensure structure names exist in your installation + +### Performance Issues + +1. Use local schema files for better performance +2. Consider schema caching in your IDE +3. Limit schema complexity for large configurations + +This schema system ensures your STRUCT configurations are valid and provides a better development experience through IDE integration. diff --git a/docs/template-variables.md b/docs/template-variables.md new file mode 100644 index 0000000..a87f125 --- /dev/null +++ b/docs/template-variables.md @@ -0,0 +1,182 @@ +# Template Variables + +Template variables allow you to create dynamic content in your STRUCT configurations. This page covers all aspects of working with variables. + +## Basic Syntax + +Use template variables by enclosing them in `{{@` and `@}}`: + +```yaml +files: + - README.md: + content: | + # {{@ project_name @}} + Welcome to {{@ project_name @}}! +``` + +## Block Syntax + +For control structures, use block notation: + +- Start block: `{%@` +- End block: `%@}` + +```yaml +files: + - config.yaml: + content: | + {%@ if environment == "production" @%} + debug: false + {%@ else @%} + debug: true + {%@ endif @%} +``` + +## Comments + +Use comment notation to document your templates: + +- Start comment: `{#@` +- End comment: `@#}` + +```yaml +files: + - app.py: + content: | + {#@ This is a template comment @#} + app_name = "{{@ project_name @}}" +``` + +## Default Variables + +STRUCT provides these built-in variables: + +- `file_name`: The name of the file being processed +- `file_directory`: The directory containing the file being processed + +## Interactive Variables + +Define variables that prompt users for input: + +```yaml +variables: + - project_name: + description: "The name of your project" + type: string + default: "MyProject" + - author_name: + description: "Your name" + type: string + # No default = interactive prompt + - port: + description: "Application port" + type: integer + default: 8080 +``` + +### Variable Types + +- `string`: Text values +- `integer`: Numeric values +- `boolean`: True/false values + +## Custom Jinja2 Filters + +STRUCT includes custom filters for common tasks: + +### `latest_release` + +Fetch the latest release version from GitHub: + +```yaml +files: + - Dockerfile: + content: | + FROM node:{{@ "nodejs/node" | latest_release @}} +``` + +**Requirements**: Set `GITHUB_TOKEN` environment variable for private repos. + +### `slugify` + +Convert strings to URL-friendly slugs: + +```yaml +files: + - "{{@ project_name | slugify @}}.conf": + content: | + server_name {{@ project_name | slugify @}}; +``` + +**Options**: Optional separator character (default: `-`) + +### `default_branch` + +Get the default branch name of a GitHub repository: + +```yaml +files: + - .github/workflows/ci.yml: + content: | + on: + push: + branches: [ {{@ "httpdss/struct" | default_branch @}} ] +``` + +## The `with` Clause + +Pass additional variables to nested structures: + +```yaml +folders: + - frontend/: + struct: project/react + with: + app_name: "{{@ project_name @}}-frontend" + port: 3000 + - backend/: + struct: project/node + with: + app_name: "{{@ project_name @}}-backend" + port: 8000 +``` + +Variables defined in `with` are merged with global variables and take precedence. + +## Advanced Examples + +### Conditional File Creation + +```yaml +files: + - docker-compose.yml: + skip: "{{@ not use_docker @}}" + content: | + version: '3.8' + services: + app: + image: {{@ project_name | slugify @}}:latest +``` + +### Dynamic File Names + +```yaml +files: + - "src/{{@ module_name @}}/index.js": + content: | + // {{@ module_name @}} module + export default {}; +``` + +### Environment-Specific Content + +```yaml +files: + - config/{{@ environment @}}.yml: + content: | + {%@ if environment == "production" @%} + database_url: {{@ production_db_url @}} + {%@ else @%} + database_url: sqlite:///dev.db + {%@ endif @%} +``` diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..8856ebb --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,98 @@ +# Basic Usage + +Run the script with the following command using one of the following subcommands: + +- `generate`: Generate the project structure based on the YAML configuration. +- `generate-schema`: Generate JSON schema for available structure templates. +- `validate`: Validate the YAML configuration file. +- `info`: Display information about the script and its dependencies. +- `list`: List the available structs + +For more information, run the script with the `-h` or `--help` option (this is also available for each subcommand): + +![Struct List](./vhs/usage.gif) + +```sh +struct -h +``` + +## Generate Command + +### Simple Example + +```sh +struct generate terraform-module ./my-terraform-module +``` + +### Complete Example + +```sh +struct generate \ + --log=DEBUG \ + --dry-run \ + --backup=/path/to/backup \ + --file-strategy=rename \ + --log-file=/path/to/logfile.log \ + terraform-module \ + ./my-terraform-module +``` + +### Command Options + +- `--log`: Set logging level (DEBUG, INFO, WARNING, ERROR) +- `--dry-run`: Preview actions without making changes +- `--backup`: Specify backup directory for existing files +- `--file-strategy`: Choose how to handle existing files (overwrite, skip, append, rename, backup) +- `--log-file`: Write logs to specified file +- `--mappings-file`: Provide external mappings file + +## Generate Schema Command + +The `generate-schema` command creates JSON schema definitions for available structure templates, making it easier for tools and IDEs to provide autocompletion and validation. + +### Basic Usage of `generate-schema` + +```sh +# Generate schema to stdout +struct generate-schema + +# Generate schema with custom structures path +struct generate-schema -s /path/to/custom/structures + +# Save schema to file +struct generate-schema -o schema.json + +# Combine custom path and output file +struct generate-schema -s /path/to/custom/structures -o schema.json +``` + +### Command Options for `generate-schema` + +- `-s, --structures-path`: Path to additional structure definitions (optional) +- `-o, --output`: Output file path for the schema (default: stdout) + +The generated schema includes all available structures from both the built-in contribs directory and any custom structures path you specify. This is useful for: + +- IDE autocompletion when writing `.struct.yaml` files +- Validation of structure references in your configurations +- Programmatic discovery of available templates + +## Other Commands + +### Validate Configuration + +```sh +struct validate my-structure.yaml +``` + +### List Available Structures + +```sh +struct list +``` + +### Show Information + +```sh +struct info +```