[WIP] New version with cleaner options

[pjbull]

We've seen a lot of potential features for this where we need to handle forking paths gracefully. By default cookiecutter can't do this (see cookiecutter/cookiecutter#848). It's been years, so we can't reasonably expect this to change upstream...

This implements a monkey-patching workaround to enable this behavior. It introduces a couple of major changes, so here are my recommendations.

Here are the big differences for a user:

• we now need to run ccds <path to repo> instead of cookiecutter <path to repo>
• the options and their defaults are changing
• as a consequence of supporting more environments/dependency managers at setup time, we stop supporting them at execution time. this means that teams will have to pick one of each and stick to it across developers. I think this is pretty widespread already. (i.e. make create_environment will only support one of the options rather than multiple like it does now)

Implementation details are:

• Add setup.py to make this package installable and give it a CLI
• add monkey_patch.py to patch the cookiecutter functions that we need to handle our use case
• support a list of dictionaries in cookiecutter.json that let's a use pick an option and then sub-options
• use post_gen_project.py to create the environment file based on a standard list of libraries. (Add package list that populates requirements.txt to the workflow #5)

There is also work for a number of longstanding items in this branch as well:

• we now support multiple storage providers (Google Cloud Storage support #120)
• we allow user to choose package manager (Allow user to choose package manager #118)
• we allow user to choose a dependency format
• src is now {{ cookiecutter.module_name }} (Rename src directory to project_name #140)
• remove test_environment.py which is just cruft IMO

Done:

• tag current master as v1 so anyone relying on the current flow/structure can continue to use it easily
• implement the rails for the high priority items
• make comprehensive tests run on CI/CD to de-risk
• support multiple dependency formats (environment.yml, requirements.txt, pipenv)
• default pydata dependencies options
• add support for azure/google cloud
• add ccds command and make cookiecutter-data-science a proper package

Remaining items

Cookiecutter default structure

• user supplied config files
• revise generated python package boilerplate (make optional)
• mkdocs in place of sphinx
• add lint command to Makefile

Cookiecutter options

• add deon - WIP Add deon and nbautoexport options #244
• add nbautoexport - WIP Add deon and nbautoexport options #244
• formatting commands / configuration (e.g., black)
• test suite configuration (e.g., pytest install and make commands)
• add

Infrastructure

• tests passing on windows
• release command and make PyPI release

Docs

• add documentation for new installation (pip install cookiecutter-data-science) and new initiation (ccds <path to repo>)
• add table of options with links to the project documentation in our docs
• update screencast and add screenshots ( Video of example in documentation is displaying error. #197 ) of the new flow
• add options for

[isms]

Thanks @pjbull. Approving general direction of edits on this WIP PR, seems like this hits many of the customization requests that we have been getting and is a reasonable workaround for the cookiecutter limitations.

[ned2]

I would suggest changing cookiecutter.module_name to cookiecutter.package_name. While it's technically true that all Python packages are modules, when arranged into a collection of other modules and having a __init__.py, they then also become a package, and this is more salient from the end user perspective. For example, the status as package informs the user that it can be pip installed.

[jamesmyatt]

I would suggest changing cookiecutter.module_name to cookiecutter.package_name. While it's technically true that all Python packages are modules, when arranged into a collection of other modules and having a __init__.py, they then also become a package, and this is more salient from the end user perspective. For example, the status as package informs the user that it can be pip installed.

module_name is correct. For example, "sklearn" is the module name for the package named "scikit-learn".

[ned2]

module_name is correct. For example, "sklearn" is the module name for the package named "scikit-learn".

Ah yes. Good point. Just found it a little counter-intuitive at first.

[tim-werner]

Are there any updates regarding this pull request? The master branch is quite outdated and I think this pull request adds a lot of value and addresses many open issues.

[hoangphan1509]

You should check for this file also https://github.com/drivendata/cookiecutter-data-science/blob/new-cli/%7B%7B%20cookiecutter.repo_name%20%7D%7D/docs/commands.rst.

`
{% if 's3' in cookiecutter.dataset_storage %}

Syncing data to S3
^^^^^^^^^^^^^^^^^^

• make sync_data_to_s3 will use aws s3 sync to recursively sync files in data/ up to s3://{{ cookiecutter.s3_bucket }}/data/.
• make sync_data_from_s3 will use aws s3 sync to recursively sync files from s3://{{ cookiecutter.s3_bucket }}/data/ to data/.
  {% endif %}
  `

[hackalog]

Or perhaps you can expect this to change upstream: New maintainier. Cookiecutter 2.0 coming. cookiecutter/cookiecutter#1256

[pjbull]

I've got a decent head of steam on a comprehensive set of tests we needed anyway for this change (and for the project generally). I'll keep going to get all the environment options tested on all platforms and then we can assess what to look to cookiecutter for and what to do ourselves.

The biggest decision is whether this becomes a tool where you have to do ccds .... or cookiecutter .... I guess, once this is all working, we can always integrate upstream cookiecutter functionality and remove the monkey_patch. However, I wouldn't want to go back to cookiecutter ... being the entrypoint since that is annoying churn in the command that you run and the packages you need installed to create environments.

[pjbull]

Moving this work on to an official v2 branch!