Skip to content

Commit

Permalink
merge from develop (#1)
Browse files Browse the repository at this point in the history
* ✨ add simulated build

* add logging to simulated build

* ✨ add environment REST API

* 📝 updating inline docs

* 🔧 update spack runtime config

* 🔧 chore(pyproject.toml): fix formatting and add missing dependencies
✨ feat(pyproject.toml): add fastapi dependency

🐛 fix(pyproject.toml): add disallow_untyped_calls and disallow_untyped_defs to mypy configuration

🐛 fix(cli.py): add return type hints to service() and main() functions
✨ feat(config.yml): change server port from 8000 to 7080
✨ feat(config.py): add manifest field to SpackConfig model and change config file path to ~/.softpack/builder/

🔧 chore(environment.py): add type hints to methods and arguments
✨ feat(environment.py): add support for Spack environment commands
🚀 perf(environment.py): improve build method to fail fast
🐛 fix(environment.py): fix environment_instantiate argument type

🔧 chore(environment.py): add type hints to function arguments and return types

🐛 fix(utils.py): add type hints to async_exec function arguments and return value
🗑️ chore(test_softpack_builder.py): delete unused test file and fixture

* 🐛 fix(app.py): add missing import for __version__ and add ServiceStatus class and root endpoint
✨ feat(environment.py): add type ignore comments to env variable and async_exec function call

✅ test(test_app.py): add test for HTTP GET request to root route

* 🔧 chore(config): update ORAS model to allow optional username, token, and uri fields
🔧 chore(config): update SpackConfig model to allow optional manifest field

* 🚀 feat(pyproject.toml): add essential-generators package
🔥 chore(utils.py): remove async_exec function
♻️ refactor(environment.py): use a wrapper function for environment_create_flow in background task
🔧 chore(tox.ini): add tests directory to autoflake and isort commands

* 🐛 fix(config.py): assign path variable before returning in classmethod
✨ feat(environment.py): make flow_wrapper async function

* ✅ test(test_environment.py): add test for environment flow and env model creation

* 🚀 chore(dev.yml): rename workflow from 'dev workflow' to 'tests'
📝 docs(README.md): add badges for issue resolution time and percentage of open issues

🔧 chore(pyproject.toml): reformat dependencies section for better readability

📦 chore(pyproject.toml): add sqlalchemy and singleton-decorator dependencies
🔧 chore(pyproject.toml): update softpack-builder script path
🔧 chore(pyproject.toml): add pytest-asyncio, pytest-cases, and pytest-httpx to test extras
🔧 chore(pyproject.toml): add filterwarnings option to pytest.ini_options section

🚀 feat(app.py): add Application class as a singleton to manage FastAPI application and Typer commands
🔧 chore(app.py): refactor ServiceStatus to a dictionary
🔧 chore(app.py): refactor root() to return a dictionary instead of ServiceStatus object
🚀 feat(app.py): add register_module() method to register a module with the application
🚀 feat(app.py): add url() method to get absolute URL path
🚀 feat(app.py): add main() method as the main command line entrypoint

🔥 chore(cli.py): remove unused code
🔧 chore(config.py): remove unused settings variable and method

🎉 feat(deployments.py): add Deployments class for building and running deployments

🚀 feat(environment.py): add delay_task function for testing purposes
🔧 chore(environment.py): refactor Environment.Model to include from_yaml and asdict methods

🚀 feat(environment.py): add stage, create_manifest, and build methods to Environment class

🔧 chore(environment.py): refactor create_environment_flow to use Prefect tasks and add delay_task to each task

🎉 feat(main.py): add SoftPack Builder main entrypoint
🚀 feat(service.py): add Service module with run command to start the SoftPack Builder REST API service

🆕 feat(conftest.py): add fixtures for TestClient, CLI, and service factory and service runners.

🚨 test(test_app.py): fix test_root, test_openapi_docs, and test_openapi_redoc tests to use httpx instead of requests
✨ feat(test_app.py): add test_register_module test to check if module is registered correctly in the application

🆕 test(test_deployments.py): add tests for Deployments class and its methods

🔧 chore(test_environment.py): refactor test_environment_create_api to use a parameterized test
🚀 feat(test_environment.py): add test_environment_create_command to test environment creation via CLI

🚨 fix(tests): fix flake8 errors in test_main.py
✅ test(service.py): add test for service run
🚨 fix(tox.ini): ignore D101, D102, D103, D107 in flake8 tests for tests directory

* ✨ implement flow deployments

* 📦 update poetry lock file

* 🐛 fix(test_main.py): use pytest.raises context manager instead of try-except block in test_main function

* 🚀 chore(dev.yml): add Spack installation and create config file step to dev workflow
✅ test(conftest.py): add fixture for running Prefect agent and add specs for Argon and Zirconium projects

🐛 fix(test_environment.py): add create_environment import
✨ feat(test_environment.py): add test_environment_create_flow test
🔧 chore(tox.ini): add concurrency option to coverage run section

* 🐛 fix(dev.yml): change 'repo' to 'repository' in 'Install Spack' job (#6)

* 🐛 fix(dev.yml): change 'repo' to 'repository' in 'Install Spack' job configuration

* 🔧 chore(dev.yml): update branch names in workflow trigger to only use 'main' and 'develop' branches

* 🔧 chore(dev.yml): refactor Create config file step to use environment variables for better readability and maintainability

* 🐛 fix(dev.yml): use env variable in CONFIG_FILE instead of hardcoding it

* 🐛 fix(dev.yml): change env variable from env.CONFIG_PATH to vars.CONFIG_PATH

* 🐛 fix(dev.yml): fix typo in CONFIG_FILE variable assignment
🔧 chore(dev.yml): change CONFIG_FILE path to be relative to CONFIG_PATH

* 🐛 fix(dev.yml): update SPACK_COMMAND to use $GITHUB_WORKSPACE to locate spack binary
🔍 chore(dev.yml): add debugging information to the job

* 🐛 fix(dev.yml): remove SPACK_COMMAND environment variable and use direct path to spack command instead

* 🐛 fix(dev.yml): change CONFIG_PATH to relative path
✨ feat(dev.yml): add debug information to script

* 🐛 fix(dev.yml): change path to config file to use HOME environment variable

* 🐛 fix(dev.yml): change CONFIG_PATH to $HOME/.softpack/builder
🔧 chore(dev.yml): remove unnecessary echo, pwd, ls, and cat commands

* merge feature/singularity (#7)

* 🔧 chore(config): add SingularityConfig model to Settings
🔧 chore(deployments): change default deployment name to include flow name

🚀 feat(environment.py): add Manifest class to handle Spack manifest files
🔧 chore(environment.py): add owner field to Environment data model
🔧 chore(environment.py): change typing import to include Callable and cast
🔧 chore(environment.py): change task import to Task and add FlowRunContext import
🔧 chore(environment.py): change DaskTaskRunner import to use prefect_dask.task_runners
🔧 chore(environment.py): change ShellOperation import to use prefect_shell
🔧 chore(environment.py): change BaseModel import to use pydantic
🔧 chore(environment.py): change list[str] to List[str] in Environment data model
🔧 chore(environment.py): change dataclasses import to use dataclass decorator
🔧 chore(environment.py): change Path import to use pathlib.Path
🐛 fix(environment.py): fix from_yaml method to use self.filename instead of filename argument
🐛

🔨 refactor(environment.py): refactor Environment class to use Box for filenames and add task method
🚀 chore(environment.py): add task_run_name to Prefect task for better logging
🐛 fix(environment.py): pass kwargs to ShellOperation and spack_command methods

🔨 refactor(environment.py): extract singularity_command method from Environment class
✨ feat(environment.py): add create_container_definition method to create Spack manifest and Singularity definition

🚀 feat(environment.py): add create_container_definition task to create container definition
👷 chore(environment.py): use Environment.task decorator instead of @task() decorator
👷 chore(environment.py): use cast() to cast the return value of the task functions
👷 chore(environment.py): return FlowRunContext dictionary from create_environment function

🔀 chore(tests, tox): merge changes from branch 'guest' to 'main'
👥 chore(specs): add owner field to argon and zirconium projects
🔧 chore(tox): update envlist to include python 3.9 and remove python 3.8 from gh-actions configuration

* 🚀 chore(config.yml): update spack version to v0.19.1
🐛 fix(environment.py): add --fakeroot flag to build command and convert filenames to strings

* 🔧 chore(config.yml): add os_packages to container config
✨ feat(config.yml): add strip option to container config
✨ feat(config.yml): add concretizer.unify option to config view

🐛 fix(environment.py): replace dictionary with Box object to allow attribute access
🔥 chore(environment.py): remove unused from_yaml method
✨ feat(environment.py): use Box object to patch manifest and dump to file

🔨 refactor(environment.py): change return type of create_manifest, concretize and build methods to Environment
✨ feat(environment.py): add concretize method to Environment class
🐳 chore(environment.py): rename create_container_definition method to containerize in Environment class

🚀 chore(environment.py): refactor create_environment function to use a list of tasks instead of individual tasks
🐛 fix(environment.py): rename create_container_definition task to containerize_environment
✨ feat(environment.py): add concretize_environment task to concretize an environment before containerizing it
✨ feat(environment.py): add containerize_environment task to containerize an environment
✨ feat(environment.py): add build_environment task to build an environment

* 🔧 chore(dev.yml): add steps to install Singularity and run code coverage with codecov-action

* 🔙 revert(dev.yml): downgrade Singularity version to 3.8.7

* 🔧 updated spack config

* 🚀 chore(dev.yml): update spack reference to commit 6aa4c29119
🐛 fix(config.yml): add missing dependencies for spack build
🐛 fix(environment.py): fix logger initialization and spack command path

* 🔀 chore(dev.yml): update spack repository reference to commit 0707ffd4e466402bf19dff1add59eaf2b6d9154e
✅ test(dev.yml): run tests using poetry run tox instead of tox directly

* ✨ feat(config.yml): add support for environment path and spack config
🐛 fix(environment.py): fix ImageSpec constructor arguments
✨ feat(environment.py): add ImageSpec abstract class and BuildSpec and FinalSpec subclasses
✨ feat(environment.py): add patch_manifest method to Environment class
✨ feat(environment.py): add containerize method to Environment class
✨ feat(environment.py): add build method to Environment class
✨ feat(environment.py): add path attribute to Environment class
✨ feat(environment.py): add EnvironmentConfig class to SpackConfig
✨ feat(environment.py): add Environment classmethod to create Environment from model
✨ feat(environment.py): add environment path to Environment constructor
✨ feat(environment.py): add path attribute to Environment class
✨ feat(environment.py): add patch method to BuildSpec
✨ feat(environment.py): add patch method to FinalSpec
✨ feat(environment.py): add args method to Build

* 🔧 chore(makefile): remove tests directory from mypy command
🔧 chore(config.yml): rename spack.environment to spack.environments
🔧 chore(environment.py): change --directory to --bind and remove --allow-root and --force options
🔧 chore(environment.py): change logger type to LoggerAdapter
🔧 chore(test_environment.py): add settings fixture and change create_environment call to use Box

* 🐛 fix(conftest.py): use Path instead of str for environments path
🐛 fix(conftest.py): use tempfile.mkdtemp instead of TemporaryDirectory
🐛 fix(test_environment.py): reduce PREFECT_AGENT_TIMEOUT to 5 seconds
✨ feat(test_environment.py): add prefect_agent fixture to join threads
✨ feat(test_environment.py): add capsys fixture to capture stdout and stderr

* 🚨 fix(test_environment.py): remove unused fixture 'prefect_agent' and commented out code
✨ feat(test_environment.py): remove unused fixture 'prefect_agent' from test signature

* 🚀 feat(dev.yml): add step to install Spack templates
🐛 fix(conftest.py): use tmpdir fixture instead of tempfile to create temporary directory
🐛 fix(conftest.py): set path attribute of Environment.settings.spack.environments to Path object
🚀 feat(conftest.py): add Spack templates directory to Environment.settings.spack.manifest.spack.config.template_dirs
🐛 fix(test_environment.py): remove unused settings fixture from test_environment_create_flow() function signature

* 🔍 chore(dev.yml): add temporary files debug step to show workspace and spack-templates directory contents

* 🔧 chore(dev.yml): create Spack config file and show its content
🔧 chore(environment.py): remove settings attribute from Environment class and pass it as an argument to Manifest.patch() method
🔧 chore(conftest.py): refactor fixture to patch app settings instead of Environment settings

* 🚀 chore(dev.yml): update spack-templates repository to wtsi-hgi/spack-templates
🔒 chore(dev.yml): create GPG key for softpack
🔧 chore(config.yml): remove unused spack config options
🔧 chore(config.yml): change spack cache and environments paths to /tmp
🔧 chore(config.yml): add bind mount for spack cache and GPG key in Singularity build
🔧 chore(environment.py): change spack environments path to /tmp
🔧 chore(tests/conftest.py): change spack environments and cache paths to /tmp
🔧 chore(tests/data/specs): remove argon.yml and zirconium.yml, add test-env.yml
🔧 chore(tests/test_environment.py): increase Prefect agent timeout to 300 seconds

* 🔍 chore(dev.yml): add debug logs to check .gpg directory
✨ feat(tox.ini): add verbose flag to pytest command

* 🚀 chore(dev.yml): add test for GPG directory configuration

* 🔥 chore(dev.yml): remove TEST GPG DIR step
🔒 chore(dev.yml): set SPACK_GNUPGHOME environment variable in Create GPG key step

* 🚀 chore(dev.yml): add optional input to enable tmate debugging in manually-triggered workflows

* 🔥 chore(dev.yml): remove tmate debugging step from dev workflow
🐛 fix(dev.yml): add which command to show singularity location in Spack config step
✨ feat(dev.yml): add tmate debugging step to dev workflow when manually triggered with debug_enabled input

* 🐛 fix(config.yml): use $HOME instead of tilde (~) to reference user's home directory in bind option

* 🔥 chore(dev.yml): remove Spack config display step
✨ feat(dev.yml): add code coverage step using codecov-action

* 🔥 chore(tox.ini): remove verbose flag from pytest command

* merge from feature/singularity (#8)

* 🔧 chore(config): add SingularityConfig model to Settings
🔧 chore(deployments): change default deployment name to include flow name

🚀 feat(environment.py): add Manifest class to handle Spack manifest files
🔧 chore(environment.py): add owner field to Environment data model
🔧 chore(environment.py): change typing import to include Callable and cast
🔧 chore(environment.py): change task import to Task and add FlowRunContext import
🔧 chore(environment.py): change DaskTaskRunner import to use prefect_dask.task_runners
🔧 chore(environment.py): change ShellOperation import to use prefect_shell
🔧 chore(environment.py): change BaseModel import to use pydantic
🔧 chore(environment.py): change list[str] to List[str] in Environment data model
🔧 chore(environment.py): change dataclasses import to use dataclass decorator
🔧 chore(environment.py): change Path import to use pathlib.Path
🐛 fix(environment.py): fix from_yaml method to use self.filename instead of filename argument
🐛

🔨 refactor(environment.py): refactor Environment class to use Box for filenames and add task method
🚀 chore(environment.py): add task_run_name to Prefect task for better logging
🐛 fix(environment.py): pass kwargs to ShellOperation and spack_command methods

🔨 refactor(environment.py): extract singularity_command method from Environment class
✨ feat(environment.py): add create_container_definition method to create Spack manifest and Singularity definition

🚀 feat(environment.py): add create_container_definition task to create container definition
👷 chore(environment.py): use Environment.task decorator instead of @task() decorator
👷 chore(environment.py): use cast() to cast the return value of the task functions
👷 chore(environment.py): return FlowRunContext dictionary from create_environment function

🔀 chore(tests, tox): merge changes from branch 'guest' to 'main'
👥 chore(specs): add owner field to argon and zirconium projects
🔧 chore(tox): update envlist to include python 3.9 and remove python 3.8 from gh-actions configuration

* 🚀 chore(config.yml): update spack version to v0.19.1
🐛 fix(environment.py): add --fakeroot flag to build command and convert filenames to strings

* 🔧 chore(config.yml): add os_packages to container config
✨ feat(config.yml): add strip option to container config
✨ feat(config.yml): add concretizer.unify option to config view

🐛 fix(environment.py): replace dictionary with Box object to allow attribute access
🔥 chore(environment.py): remove unused from_yaml method
✨ feat(environment.py): use Box object to patch manifest and dump to file

🔨 refactor(environment.py): change return type of create_manifest, concretize and build methods to Environment
✨ feat(environment.py): add concretize method to Environment class
🐳 chore(environment.py): rename create_container_definition method to containerize in Environment class

🚀 chore(environment.py): refactor create_environment function to use a list of tasks instead of individual tasks
🐛 fix(environment.py): rename create_container_definition task to containerize_environment
✨ feat(environment.py): add concretize_environment task to concretize an environment before containerizing it
✨ feat(environment.py): add containerize_environment task to containerize an environment
✨ feat(environment.py): add build_environment task to build an environment

* 🔧 chore(dev.yml): add steps to install Singularity and run code coverage with codecov-action

* 🔙 revert(dev.yml): downgrade Singularity version to 3.8.7

* 🔧 updated spack config

* 🚀 chore(dev.yml): update spack reference to commit 6aa4c29119
🐛 fix(config.yml): add missing dependencies for spack build
🐛 fix(environment.py): fix logger initialization and spack command path

* 🔀 chore(dev.yml): update spack repository reference to commit 0707ffd4e466402bf19dff1add59eaf2b6d9154e
✅ test(dev.yml): run tests using poetry run tox instead of tox directly

* ✨ feat(config.yml): add support for environment path and spack config
🐛 fix(environment.py): fix ImageSpec constructor arguments
✨ feat(environment.py): add ImageSpec abstract class and BuildSpec and FinalSpec subclasses
✨ feat(environment.py): add patch_manifest method to Environment class
✨ feat(environment.py): add containerize method to Environment class
✨ feat(environment.py): add build method to Environment class
✨ feat(environment.py): add path attribute to Environment class
✨ feat(environment.py): add EnvironmentConfig class to SpackConfig
✨ feat(environment.py): add Environment classmethod to create Environment from model
✨ feat(environment.py): add environment path to Environment constructor
✨ feat(environment.py): add path attribute to Environment class
✨ feat(environment.py): add patch method to BuildSpec
✨ feat(environment.py): add patch method to FinalSpec
✨ feat(environment.py): add args method to Build

* 🔧 chore(makefile): remove tests directory from mypy command
🔧 chore(config.yml): rename spack.environment to spack.environments
🔧 chore(environment.py): change --directory to --bind and remove --allow-root and --force options
🔧 chore(environment.py): change logger type to LoggerAdapter
🔧 chore(test_environment.py): add settings fixture and change create_environment call to use Box

* 🐛 fix(conftest.py): use Path instead of str for environments path
🐛 fix(conftest.py): use tempfile.mkdtemp instead of TemporaryDirectory
🐛 fix(test_environment.py): reduce PREFECT_AGENT_TIMEOUT to 5 seconds
✨ feat(test_environment.py): add prefect_agent fixture to join threads
✨ feat(test_environment.py): add capsys fixture to capture stdout and stderr

* 🚨 fix(test_environment.py): remove unused fixture 'prefect_agent' and commented out code
✨ feat(test_environment.py): remove unused fixture 'prefect_agent' from test signature

* 🚀 feat(dev.yml): add step to install Spack templates
🐛 fix(conftest.py): use tmpdir fixture instead of tempfile to create temporary directory
🐛 fix(conftest.py): set path attribute of Environment.settings.spack.environments to Path object
🚀 feat(conftest.py): add Spack templates directory to Environment.settings.spack.manifest.spack.config.template_dirs
🐛 fix(test_environment.py): remove unused settings fixture from test_environment_create_flow() function signature

* 🔍 chore(dev.yml): add temporary files debug step to show workspace and spack-templates directory contents

* 🔧 chore(dev.yml): create Spack config file and show its content
🔧 chore(environment.py): remove settings attribute from Environment class and pass it as an argument to Manifest.patch() method
🔧 chore(conftest.py): refactor fixture to patch app settings instead of Environment settings

* 🚀 chore(dev.yml): update spack-templates repository to wtsi-hgi/spack-templates
🔒 chore(dev.yml): create GPG key for softpack
🔧 chore(config.yml): remove unused spack config options
🔧 chore(config.yml): change spack cache and environments paths to /tmp
🔧 chore(config.yml): add bind mount for spack cache and GPG key in Singularity build
🔧 chore(environment.py): change spack environments path to /tmp
🔧 chore(tests/conftest.py): change spack environments and cache paths to /tmp
🔧 chore(tests/data/specs): remove argon.yml and zirconium.yml, add test-env.yml
🔧 chore(tests/test_environment.py): increase Prefect agent timeout to 300 seconds

* 🔍 chore(dev.yml): add debug logs to check .gpg directory
✨ feat(tox.ini): add verbose flag to pytest command

* 🚀 chore(dev.yml): add test for GPG directory configuration

* 🔥 chore(dev.yml): remove TEST GPG DIR step
🔒 chore(dev.yml): set SPACK_GNUPGHOME environment variable in Create GPG key step

* 🚀 chore(dev.yml): add optional input to enable tmate debugging in manually-triggered workflows

* 🔥 chore(dev.yml): remove tmate debugging step from dev workflow
🐛 fix(dev.yml): add which command to show singularity location in Spack config step
✨ feat(dev.yml): add tmate debugging step to dev workflow when manually triggered with debug_enabled input

* 🐛 fix(config.yml): use $HOME instead of tilde (~) to reference user's home directory in bind option

* 🔥 chore(dev.yml): remove Spack config display step
✨ feat(dev.yml): add code coverage step using codecov-action

* 🔥 chore(tox.ini): remove verbose flag from pytest command

* 🔥 refactor(environment.py): move settings attribute to the top of the class
🐛 fix(environment.py): remove redundant settings attribute assignment in __init__ method

* 🐛 fix(environment.py): fix logger initialization in init_logger method
🐛 fix(test_environment.py): fix test_environment_logger test case to properly test logger initialization

* 🐛 fix(environment.py): add exist_ok=True to mkdir method
🧪 test(environment.py): add cleanup of created directory after test

* 🔧 chore(pyproject.toml): update python version to >=3.9,<4.0
🔧 chore(pyproject.toml): remove python version 3.8 from classifiers
🔧 chore(pyproject.toml): remove target-version py38 from black configuration
🔧 chore(environment.py): update docstring for from_model method
🔧 chore(test_environment.py): update create_environment call to use Box
🔧 chore(test_environment.py): remove unused PREFECT_AGENT_TIMEOUT variable
🔧 chore(test_environment.py): update logger test to trigger exception

* ✨ add artifacts support (#9)

* ✨ add artifacts support

initial support for adding adding artifacts and pushing images to container registries

* 📦 chore(deps): add mergedeep to pyproject.toml
📦 chore(deps): add semver to pyproject.toml
📦 chore(tox): use poetry to manage dependencies
📦 chore(tox): add commands_pre to install dependencies before running commands in tox.ini

* 🐛 fix(models.py): make VaultConfig fields optional
🐛 fix(settings.py): catch KeyError when trying to read vault config from file and print to stderr

* 🔧 chore(README.md): use --with instead of -E to install poetry environments
🔧 chore(softpack_builder): rename register_module to register_serializer in all files

* merge from feature/artifacts (#10)

* ✨ add artifacts support

initial support for adding adding artifacts and pushing images to container registries

* 📦 chore(deps): add mergedeep to pyproject.toml
📦 chore(deps): add semver to pyproject.toml
📦 chore(tox): use poetry to manage dependencies
📦 chore(tox): add commands_pre to install dependencies before running commands in tox.ini

* 🐛 fix(models.py): make VaultConfig fields optional
🐛 fix(settings.py): catch KeyError when trying to read vault config from file and print to stderr

* 🔧 chore(README.md): use --with instead of -E to install poetry environments
🔧 chore(softpack_builder): rename register_module to register_serializer in all files

* 🐛 fix(app.py): change datetime format to string to avoid serialization issues
🐛 fix(artifacts.py): fix typo in variable name from registry to registry
🔨 refactor(models.py): capitalize first letter of class docstring
🔨 refactor(settings.py): rename config_dir to default_config_dir and add user_config_dir to improve semantics. Also remove unused imports.

* 🔥 refactor(spack.py): remove unused buildcache create command and directory argument
🚀 feat(spack.py): add push command to buildcache to upload package to remote cache server using the hostname of the current machine

* merge from feature/artifacts (#11)

* ✨ add artifacts support

initial support for adding adding artifacts and pushing images to container registries

* 📦 chore(deps): add mergedeep to pyproject.toml
📦 chore(deps): add semver to pyproject.toml
📦 chore(tox): use poetry to manage dependencies
📦 chore(tox): add commands_pre to install dependencies before running commands in tox.ini

* 🐛 fix(models.py): make VaultConfig fields optional
🐛 fix(settings.py): catch KeyError when trying to read vault config from file and print to stderr

* 🔧 chore(README.md): use --with instead of -E to install poetry environments
🔧 chore(softpack_builder): rename register_module to register_serializer in all files

* 🐛 fix(app.py): change datetime format to string to avoid serialization issues
🐛 fix(artifacts.py): fix typo in variable name from registry to registry
🔨 refactor(models.py): capitalize first letter of class docstring
🔨 refactor(settings.py): rename config_dir to default_config_dir and add user_config_dir to improve semantics. Also remove unused imports.

* 🔥 refactor(spack.py): remove unused buildcache create command and directory argument
🚀 feat(spack.py): add push command to buildcache to upload package to remote cache server using the hostname of the current machine

* 📝 docs(spack.py): update docstring of env_buildcache method to reflect its functionality
🚀 feat(spack.py): add support for pushing build cache for a package using the command method

* 🆕 chore(requirements.txt): add new dependencies to the project to improve functionality and security
  • Loading branch information
altaf-ali committed Jun 20, 2023
1 parent f26b2a6 commit ad77a47
Show file tree
Hide file tree
Showing 42 changed files with 3,972 additions and 1,297 deletions.
67 changes: 56 additions & 11 deletions .github/workflows/dev.yml
Original file line number Diff line number Diff line change
@@ -1,17 +1,23 @@
# This is a basic workflow to help you get started with Actions

name: dev workflow
name: tests

# Controls when the action will run.
on:
# Triggers the workflow on push or pull request events but only for the master branch
push:
branches: [ master, main ]
branches: [ main ]
pull_request:
branches: [ master, main ]
branches: [ main, develop ]

# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
inputs:
debug_enabled:
type: boolean
description: Run the build with tmate debugging enabled
required: false
default: false

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
Expand All @@ -27,8 +33,11 @@ jobs:
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
- name: Checkout the repository
uses: actions/checkout@v3

- name: Setup python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-versions }}

Expand All @@ -37,14 +46,50 @@ jobs:
python -m pip install --upgrade pip
pip install poetry tox tox-gh-actions
- name: test with tox
run:
tox
- name: Install Singularity
uses: eWaterCycle/setup-singularity@v7
with:
singularity-version: 3.8.7

- name: Install Spack
uses: actions/checkout@v3
with:
repository: spack/spack
ref: 0707ffd4e466402bf19dff1add59eaf2b6d9154e
path: spack

- name: Install Spack templates
uses: actions/checkout@v3
with:
repository: wtsi-hgi/softpack-templates
ref: main
path: templates

- name: Create Spack config
env:
CONFIG_PATH: $HOME/.spack
CONFIG_FILE: config.yaml
run: |
mkdir -p ${{ env.CONFIG_PATH }}
cat <<EOF > ${{ env.CONFIG_PATH }}/$CONFIG_FILE
config:
template_dirs:
- $GITHUB_WORKSPACE/templates
EOF
- name: Create GPG key
run: |
SPACK_GNUPGHOME=$HOME/.gpg $GITHUB_WORKSPACE/spack/bin/spack gpg create softpack softpack@example.com
- name: Update PATH
run: |
echo "$GITHUB_WORKSPACE/spack/bin" >> $GITHUB_PATH
- name: list files
run: ls -l .
- name: Run tests
run: poetry run tox

- uses: codecov/codecov-action@v3
- name: Run code coverage
uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }} # not required for public repos
fail_ci_if_error: true
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -112,3 +112,6 @@ ENV/

# mkdocs build dir
site/

# config file
/config.yml
3 changes: 2 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ repos:
rev: 23.1.0
hooks:
- id: black
language_version: python3.10
language_version: python3.11
- repo: https://github.com/pycqa/flake8
rev: 5.0.4
hooks:
Expand All @@ -35,3 +35,4 @@ repos:
additional_dependencies:
- types-click
- types-PyYAML
- types-requests
42 changes: 40 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
[![python](https://img.shields.io/pypi/pyversions/softpack-builder.svg)](https://pypi.org/project/softpack-builder/)
[![Build Status](https://github.com/wtsi-hgi/softpack-builder/actions/workflows/dev.yml/badge.svg)](https://github.com/wtsi-hgi/softpack-builder/actions/workflows/dev.yml)
[![codecov](https://codecov.io/gh/wtsi-hgi/softpack-builder/branch/main/graphs/badge.svg)](https://codecov.io/github/wtsi-hgi/softpack-builder)
[![Average time to resolve an issue](http://isitmaintained.com/badge/resolution/wtsi-hgi/softpack-builder.svg)](http://isitmaintained.com/project/wtsi-hgi/softpack-builder "Average time to resolve an issue")
[![Percentage of issues still open](http://isitmaintained.com/badge/open/wtsi-hgi/softpack-builder.svg)](http://isitmaintained.com/project/wtsi-hgi/softpack-builder "Percentage of issues still open")
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg)](https://www.contributor-covenant.org/version/2/1/code_of_conduct)

Expand Down Expand Up @@ -49,7 +51,7 @@ the [Github repo][].
You can either clone the public repository:

``` console
$ git clone git://github.com/wtsi-hgi/softpack-builder
$ git clone https://github.com/wtsi-hgi/softpack-builder.git
```

Or download the [tarball][]:
Expand All @@ -64,13 +66,49 @@ Once you have a copy of the source, you can install it with:
$ pip install .
```

### Development

For development mode, clone the repository and use [Poetry][] to install the
package.

``` console
$ git clone https://github.com/wtsi-hgi/softpack-builder.git
```

Install [Poetry][]:

``` console
$ pip install poetry
```

Install [Poetry][] environments for development:

``` console
poetry install --with dev,doc,test
```

Run tests with [Tox][]

``` console
poetry run tox
```

Run [MkDocs] server to view documentation:

``` console
poetry run mkdocs serve
```


[pip]: https://pip.pypa.io
[Python installation guide]: http://docs.python-guide.org/en/latest/starting/installation/
[Github repo]: https://github.com/wtsi-hgi/softpack-builder
[tarball]: https://github.com/wtsi-hgi/softpack-builder/tarball/master
[Dask]: https://www.dask.org
[Prefect]: https://www.prefect.io

[Poetry]: https://python-poetry.org
[Tox]: https://tox.wiki
[MkDocs]: https://www.mkdocs.org

## Credits

Expand Down
6 changes: 0 additions & 6 deletions docs/guides/how-to-manually-start-a-build.md

This file was deleted.

12 changes: 12 additions & 0 deletions docs/guides/how-to-start-a-build.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
title: How to start a build
---

!!! info
This documentation is under development and may be incomplete.

SoftPack Builder starts a build automatically as it receives a webhook from
SoftPack Artifacts repo when a new input spec is pushed.

For development or debugging purposes, a build can be started manually using
the command line as well.
58 changes: 46 additions & 12 deletions docs/topics/design.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,9 @@
!!! info
This documentation is under development and may be incomplete.

SoftPack Builder is a lightweight service that listens to WebHooks from
[SoftPack Artifacts][] repository and proceeds to build a container image from a
given environment spec.

## Glossary

!!! note "TODO"
Add glossary
SoftPack Builder is a lightweight service that builds container images and
other related artifacts for a software environment based on a list of end-user
requirements.

```mermaid
flowchart TB
Expand All @@ -30,7 +25,7 @@ flowchart TB
softpack_builder -->|"REST API (GitLab, Docker Registry) / HTTP"| softpack_artifacts
```

## Execution environment
## Build execution

SoftPack Builder uses [Prefect][] to execute the build flows. The build flow is
responsible for taking the input spec (received through a webhook or the
Expand All @@ -39,15 +34,48 @@ the target system. The build flow carries out the following tasks:

- Create a spack environment file from input spec
- Start spack build
- Create contanier image
- Create module file
- Push to artifacts repo
- spack environment file
- spack lock file
- container image
- module file

!!! note "TODO"
Add flow diagram
## Input spec

The input spec provides a list of platform-agnostic requirements for building
a software environment.

| Attribute | Type |
|-------------|------------|
| name | string |
| description | string |
| packages | string[] |


## Sequence diagram

```mermaid
sequenceDiagram
autonumber
participant softpack_artifacts as SoftPack Artifacts
participant softpack_builder as SoftPack Builder
participant prefect_flow as Prefect Flow
participant task_runner as Task Runner
softpack_artifacts ->> softpack_builder: Input spec
softpack_builder ->> softpack_artifacts: Environment spec
softpack_builder ->> prefect_flow: Build environment
prefect_flow ->> task_runner: Start spack build
activate task_runner
prefect_flow ->> task_runner: Create container image
prefect_flow ->> task_runner: Create module file
prefect_flow ->> task_runner: Save artifacts
task_runner ->> softpack_artifacts: Push artifacts
deactivate task_runner
```

All tasks within the build flow are defined as asynchronous tasks to allow
scalability by running the tasks in a distributed environment. Within a flow
Expand All @@ -57,7 +85,13 @@ itself, sequential dependencies between tasks are handled by passing a
### Clutser setup

!!! note "TODO"
TODO: Add cluster setup
Add cluster setup

## Glossary

!!! note "TODO"
Add glossary


[Prefect]: https://docs.prefect.io
[PrefectFuture]: https://docs.prefect.io/api-ref/prefect/futures/#prefect.futures.PrefectFuture
Expand Down
2 changes: 1 addition & 1 deletion makefile
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ format:

lint:
flake8 $(sources) tests
mypy $(sources) tests
mypy $(sources)

unittest:
pytest
Expand Down
6 changes: 3 additions & 3 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,11 @@ nav:
- Tutorials:
- tutorials/getting_started.md
- Guides:
- guides/how-to-manually-start-a-build.md
- guides/how-to-start-a-build.md
- Reference:
- reference/cli.md
- reference/rest.md
- reference/api.md
- reference/rest.md
- Contributing: contributing.md
- Changelog: changelog.md
theme:
Expand Down Expand Up @@ -54,7 +54,6 @@ markdown_extensions:
- toc:
baselevel: 3
permalink: true
# slugify: !!python/name:pymdownx.slugs.uslugify
- meta
plugins:
- include-markdown
Expand All @@ -70,6 +69,7 @@ extra:
name: Github
- icon: material/email
link: "mailto:altaf.ali@sanger.ac.uk"

# to enable disqus, uncomment the following and put your disqus id below
# disqus: disqus_id
# uncomment the following and put your google tracking id below to enable GA
Expand Down
Loading

0 comments on commit ad77a47

Please sign in to comment.