Script to create new projects

[Ricocotam]

From issue #17 I created a minimal example of what I had in mind. This is a minimal example and some questions need to be answered.

Path of the new project

We need to create a folder for the project. I propose to use 2 different name : project_path and project_name. We then could have an architecture like :

+ project_path
    + project_name
        - __init__.py
        + models
            + networks
                - __init__.py
                - factory.py
                - mynetwork.py
    + logs

(+ denotes a folder, - a file)

This is an open suggestion. I choose this implementation so logs' folder is also created and is with the rest of the code.

Network as minimal example

I did a minimal example so we can discuss implementation details. Creating files and folders is done with pathlib.Path to use the / operation to join paths. The real question is : how to we fill created files with basic content ? I choose to have raw text written in the new.py I am Pull Requesting but I don't think that's the better option as this is not easily maintainable and it leads to a really long new.py file. My suggestion is to have a basic file we can copy to the project created. This requires to store quite a lot of files in the lib only used to be copied so it has some drawbacks too.

I am also working on a complete project creation with options to create any relevant directory but I didn't want to pollute the discussion with useless lines of code

[Cadene]

Thanks @Ricocotam, it's much needed!

However, we need a way to avoid putting executable code into a string such as this

I would suggest creating template python files in the bootstrap directory, copying them and even replacing some elements in it.

For instance, instead of this, we would have the following file in bootstrap/templates/default/project/models/networks/network.py:

import torch.nn as nn

class MyNetwork(nn.Module):
    def __init__(self, *args, **kwargs):
        super(MyNetwork, self).__init__()
        # Assign args
    def forward(self, x):
        # x is a dictionnary given by Dataset class
        pred = self.net(x)
        return pred  # This is a tensor (or several tensors)

Also, we discussed with @MicaelCarvalho.
In a first time, we would like to create a simple structure with a network, a criterion, a metric and a dataset. Also, these 4 "modules" will take the name of the project.
For instance, python -m bootstrap.new --project_name MyExample will create:

myexample.bootstrap.pytorch/myexample/models/networks/myexample.py
myexample.bootstrap.pytorch/myexample/models/criterions/myexample.py
myexample.bootstrap.pytorch/myexample/models/metrics/myexample.py
myexample.bootstrap.pytorch/myexample/datasets/myexample.py

With the following class names:

class MyExampleNetwork
class MyExampleCriterion
class MyExampleMetric
class MyExampleDataset

As well as other files such as __init__.py, factory.py and:

myexample.bootstrap.pytorch/README.md
myexample.bootstrap.pytorch/.gitignore
myexample.bootstrap.pytorch/setup.cfg
myexample.bootstrap.pytorch/setup.py
myexample.bootstrap.pytorch/LICENSE.txt
myexample.bootstrap.pytorch/myexample/__version__.py

Lastly, files from bootstrap/templates will be parsed and some strings will be replaced such as:

{PROJECT_NAME}, {PROJECT_NAME_LOWER}, {PROJECT_NAME_UPPER}

Do you agree? Is it clear? @Ricocotam

[Ricocotam]

Alright, so I did what you said on the latest commit

[Cadene]

We might want to keep project_path.
What do you think @MicaelCarvalho ?

[Cadene]

As specified here, we would like to follow this path pattern bootstrap/templates/default/project/datasets/dataset.py instead of your current bootstrap/template/datasets/template_dataset.py.

[Cadene]

'metric' can be removed here

[Ricocotam]

Not sure to understand what you mean ?

[Cadene]

def factory(engine=None, mode="train"):
    opt = Options()['model.metric']

    if opt['name'] == '{PROJECT_NAME_LOWER}':

[Cadene]

Metric

[Cadene]

Thanks! @Ricocotam
I added some comments

We should probably add a test to verify that everything can work.
It implies to have some default return values (torch.tensor(0)) instead of raise NotImplementedError or pass.
What do you think? @Ricocotam @MicaelCarvalho

[Ricocotam]

I'll fix your different comments soon. Anything to add ?

[Ricocotam]

So we need some tests now

The only thing I didn't do is to name custom dataset after the project name, I kept mydataset.py : myexample.bootstrap.pytorch/myexample/datasets/myexample.py-> myexample.bootstrap.pytorch/myexample/datasets/mydataset.py

I can write some tests (we just have to check files have the right content and that they are created) but I know how it works already. May be, to ensure code is both clear and that agree on what/how it does it, someone else should right the tests

[Ricocotam]

Solved in #35