feat: Central Hierarchy + Environment design populated by Actions

[fblanchetNaN]

The design is based on two central classes:

• Hierarchywith two attributes _configurations and _templates

  incipyt/incipyt/system.py

  Lines 127 to 143 in dc3fdde

  	class Hierarchy:
  	"""Represents all configuration and template files to commit for the new project.
  	An instance internally stores mappables between path objects and template
  	files or dictionaries modeling configuration files.
  	Functions :meth:`get_configuration` and :meth:`register_template` add
  	respectively configuration dictionary and template to the instance.
  	When the hierarchy is finally ready, functions :meth:`mkdir` :meth:`commit`
  	can be used to write folder and files in a new folder after substituting
  	variables in path and files using an :class:`incipyt.system.Environment`.
  	"""
  	def __init__(self):
  	self._configurations = {}
  	self._templates = {}

  They are dictionaries with instances of classes in incipyt/_internal/dumpers -- path-like classes with a specific method to write them, e.g. for Toml

  incipyt/incipyt/_internal/dumpers.py

  Lines 71 to 74 in dc3fdde

  	class Toml(BaseDumper):
  	def dump_in(self, config):
  	with self.substitute_path().open("w+") as file:
  	toml.dump(config, file)

  -- as keys. Each association key-value corresponds to a file path and its content to be written. For _templates, values are just Jinja template. _configurations is dedicated to all configuration files based on key-value syntax (toml, cfg, ini, yaml, json, etc), values of this dictionary are then nested dictionary describing this configuration file.
• Envrionment is basically a wrapper around a dictionary initiated with environment variables,

  incipyt/incipyt/system.py

  Lines 13 to 65 in dc3fdde

  	class Environment:
  	"""Manage environment variables using for substitutions in patterns.
  	Functions :meth:`pull`and :meth:`push` can be used to add or request a
  	specific envirnoment variable.
  	Function :meth:`requests` ask the user if a value is missing.
  	An instance is also a visitor for :class:`incipyt.system.Hierarchy`
  	elements involving pattern with environment variables.
  	"""
  	def __init__(self):
  	self._variables = os.environ.copy()
  	if "PYTHON_CMD" not in self._variables:
  	self._variables["PYTHON_CMD"] = "python"
  	def pull(self, key):
  	"""Try to pull the actual value for `key`.
  	If `key` already exists, just return the associated value, if not,
  	first asks for it -- see :func:`incipyt.system.Environment.requests`
  	-- then returns it.
  	:param key: Environment key asked.
  	:type key: str
  	:return: The actual value for `key`.
  	:rtype: str
  	"""
  	if key not in self._variables:
  	logger.info(f"Missing environement variable {key}, request it.")
  	self._variables[key] = self.requests(key)
  	return self._variables[key]
  	def push(self, key, value, update=False):
  	"""Try to push a `key` = `value` associaton.
  	:param key: Key of the association to push.
  	:type key: str
  	:param value: Value of the association to push.
  	:type value: str
  	:param update: Allow existing keys.
  	:type update: bool
  	:raises RuntimeError: Raise if `key` already exists in the actual environment.
  	"""
  	if key in self._variables and not update:
  	raise RuntimeError(
  	f"Environment variable {key} already exists, use update."
  	)
  	logger.info(f"Push environement variable {key}={value}.")
  	self._variables[key] = value

  its purpose is to deal with all variables, like {NAME} or {AUTHOR} that should be asked to the user. To do so,

  incipyt/incipyt/system.py

  Lines 67 to 74 in dc3fdde

  	def requests(self, key):
  	"""Request to the user the value to associate to `key`.
  	:param key: Key to request to the user.
  	:type key: str
  	:raises NotImplementedError: TO-DO.
  	"""
  	raise NotImplementedError(f"Request [for {key}] is not implemented.")

  have to be implemented and ask the user for a specific variable. Then it behaves as a visitor for values of Hierarchy._configuration

  incipyt/incipyt/system.py

  Lines 97 to 108 in dc3fdde

  	for key, value in template.items():
  	logger.debug(f"Visit {key} to process environment variables.")
  	if callable(value):
  	template[key] = value(self)
  	elif isinstance(value, collections.abc.MutableMapping):
  	self.visit(value)
  	elif isinstance(value, collections.abc.MutableSequence):
  	for index, element in enumerate(value):
  	if callable(element):
  	value[index] = element(self)
  	if isinstance(element, collections.abc.MutableMapping):
  	self.visit(element)

  actually values can contain some callable instead of str, if so those callable are call and the value substituted. Typically, it can be utils.Requires("{NAME}") with

  incipyt/incipyt/_internal/utils.py

  Lines 12 to 28 in dc3fdde

  	class Requires:
  	def __init__(self, template, sanitizer=None):
  	self._sanitizer = sanitizer
  	self._template = template
  	def __call__(self, environment):
  	return self._template.format(
  	**{
  	key: (
  	self._sanitizer(key, environment.pull(key))
  	if self._sanitizer
  	else environment.pull(key)
  	)
  	for _, key, _, _ in Formatter().parse(self._template)
  	if key is not None
  	}
  	)

  then it asks Environment for {NAME} : environment.pull(key).

There are similar substitution mechanisms for _template and also for using subprocess.run

incipyt/incipyt/system.py

Lines 110 to 124 in dc3fdde

	def run(self, command):
	"""Run a command after substitution using the environment.
	:param command: List of the command elements.
	:type command: List
	"""
	completed_process = subprocess.run(
	[c(self) if callable(c) else c for c in command],
	capture_output=True,
	check=True,
	)
	logger.info(
	f"""{' '.join(completed_process.args)}
	{completed_process.stdout.decode()}"""
	)

The main workflow is then to initialize a Hierarchy and let tool specific classes populate _configurations and _template

incipyt/incipyt/system.py

Lines 242 to 245 in dc3fdde

	hierarchy = Hierarchy()
	for action in actions:
	logger.info(f"Add {action} to hierarchy.")
	action.add_to(hierarchy)

---

An instance of Hierarchy stores all information about configurations files and templated files to write for the new project.

Then each tool is modeled by Action instances, they populate a hierarchy instance and define pre and post methods to run before and after the creation of the project files

incipyt/incipyt/system.py

Lines 250 to 259 in dc3fdde

	for action in actions:
	logger.info(f"Running pre-action for {action}.")
	action.pre(workon_path, environment)
	logger.info(f"Commit hierarchy on {workon_path}.")
	hierarchy.commit(workon_path, environment)
	for action in actions:
	logger.info(f"Running post-action for {action}.")
	action.post(workon_path, environment)

e.g. Git Action pre method runs git init, while post method runs git add --all and git commit.

Definitions of configurations and templates can be done with environment variables, like {NAME}. All those variables are stored in an Environment instance, it is initialized with environment variables system and can be modified by any Action using pre or post methods. Then values are used when needed and asked to user if missing.

When an Action is created, it can also register itself a hook, e.g. Git Action register a VCSIgnore hook that defines how a pattern is ignored by git. Then any Action can used those hooks, e.g. Venv Action use VCSIgnore to make VCS system aware of ignoring .env folder

incipyt/incipyt/actions/git.py

Lines 6 to 24 in dc3fdde

	class Git:
	"""Action to add Git to :class:`incipyt.system.Hierarchy`."""
	def __init__(self):
	hooks.VCSIgnore.register(self._hook)
	def add_to(self, hierarchy):
	"""Add git configuration to `hierarchy`, do nothing.
	:param hierarchy: The actual hierarchy to update with git configuration.
	:type hierarchy: :class:`incipyt.system.Hierarchy`
	"""
	def _hook(self, hierarchy, value):
	gitignore = hierarchy.get_configuration(Requirement.make(".gitignore"))
	if None not in gitignore:
	gitignore[None] = []
	gitignore[None].append(value)

[fblanchetNaN]

Note about ECS: I did some tests where _configurations and _templates are replaced by _components. i.e. each file to be written is a component of an single entity.

Finally, imo, it's not really useful as each component need its own system, so the decoupling is mainly artificial.

[fblanchetNaN]

If you want to test

import logging

logging.basicConfig(level="INFO")

import subprocess

import incipyt

env = incipyt.Environment()
env.push("NAME", "its-a_test")


incipyt.process_actions(
    "./test_project",
    env,
    [incipyt.actions.Git(), incipyt.actions.Venv(), incipyt.actions.Setuptools()],
)

[Julien00859]

I have no time to do an extensive review but from a diagonal reading, it looks great ! Thank you !

[fblanchetNaN]

I add files for CLI entrypoints

incipyt/incipyt/cli.py

Lines 10 to 18 in 6ae25a0

	@click.command()
	@click.option("--root", type=str, default="", help="Root folder.")
	def main(root):
	logging.basicConfig(level="INFO")
	process_actions(
	pathlib.Path().joinpath(root),
	Environment(),
	[actions.Git(), actions.Venv(), actions.Setuptools()],
	)

Then you can test directly with incipyt or python -m incipyt, eventually with option --root folder_name to specify a folder.

[Mesteery]

Then you can test directly with incipyt or python -m incipyt, eventually with option --root folder_name to specify a folder.

Wouldn't it be better to indicate the folder like this incipyt folder_name?
And what about the project name?

[fblanchetNaN]

Wouldn't it be better to indicate the folder like this incipyt folder_name?
And what about the project name?

Actually I just setup something basic with click (*) to have entry-points and files to edit for the CLI, not to really define the interface.

Project name (as any other missing variables used to setup tools) is ask latter (**), but of course we can add something to give it directly in the command line.

I don't think this PR should go further about the CLI (commands and options syntaxe, default values, help messages, etc).

(*) I literally just adapted the first example of clickdocumentation to have the folder as option.

import click

@click.command()
@click.option("--count", default=1, help="Number of greetings.")
@click.option("--name", prompt="Your name",
              help="The person to greet.")
def hello(count, name):
    """Simple program that greets NAME for a total of COUNT times."""
    for _ in range(count):
        click.echo("Hello, %s!" % name)

if __name__ == '__main__':
    hello()

(**) If you try to run incipyt it will display some logging information and then prompt NAME and wait for your answer.

[LeMinaw]

Suggested change

	environment.run(
	[
	utils.Requires("{PYTHON_CMD}"),
	"-m",
	"venv",
	str(workon.joinpath(".env")),
	]
	)
	environment.push(
	"PYTHON_CMD", str(workon.joinpath(".env/bin/python")), update=True
	)
	bin_dir = "Scripts" if sys.platform == "win32" else "bin"
	env_path = workon / ".env"
	python_path = env_path / bin_dir / "python"
	environment.run([utils.Requires("{PYTHON_CMD}"), "-m", "venv", str(env_path)])
	environment.push("PYTHON_CMD", str(python_path), update=True)

On Windows platform, python executable lies in ./venv/Scripts/python (and so do other scripts or binaries)

[Mesteery]

- sys.platform.startswith("win")
+ sys.platform == "win32"

[LeMinaw]

To avoid repeating those idiom in each hook register, __call__ and __init__ method, maybe hooks should inherit from a Hook superclass, e.g:

Note Hook in not an ABC because among other things it provides concrete implementation of __call__.

class Hook:
    _hooks = []

    @classmethod
    def register(cls, hook):
        cls._hooks.append(hook)

    def __init__(self, hierarchy):
        self._hierarchy = hierarchy

    def __str__(self):
        return self.__class__.__name__

    def __call__(self, *args, **kwargs):
        if self._hooks:
            logger.info(
                f"len(self._hooks)} {self} hooks called with parameters {args}, {kwargs}"
                # Maybe we can think of a nice way for child classes to redefine this logging message
            )
        
        for hook in self._hooks:
            hook(self._hierarchy, *args, **kwargs)


class ProjectURL(Hook):
    def __str__(self):
        # not needed, will default to class name according to superclass
        return "project-url"
    
    def __call__(self, url_kind, url_value):
        """Here we can provide docs and relevant hook call signature"""
        
        super().__call__(url_kind, url_value)

[fblanchetNaN]

Good idea, can you try to implement something in that sense ? You can directly commit on the branch.

[LeMinaw]

On my machine (win platform), for some reason folder is a bytes object and therefore an exception is raised here