Sample Experiment Notebook

[chadac]

This is something I've been thinking about and applied partially, but haven't been able to put into practice. It would be nice to have a sample notebook showing how to document experiments for the future.

What I've found with ML projects is that onboarding new people is often difficult because the experimentation process is often difficult because experiments on the data are often lost or obfuscated by being mixed with other experiments. Oftentimes, these issues exist because

1. People don't know that they can use %autoreload 2 to automatically reload code from modules, which makes it way easier to share code between notebooks.
2. Usually notebooks are all stored within a single folder, meaning it can be hard to find actively used notebooks in experiments.

I think showing conventions to quickly save code to notebooks and then share among them would be a nice way to expedite these processes. The current README shows how one can call code from modules inside their current notebook. Perhaps notebook helper code could be installed as a separate top-level package and installed as a dev dependency to encourage cleanliness. What do you think @Tsmith5151 ?

[Tsmith5151]

@chadac completely agree with putting together a sample notebook highlighting recommended practices when getting started on a project. Here are a couple of sections that might be worthy to include:

• Reloading modules: %autoreload
• Execute Python Module within notebook:
  • modifying notebook path: %cd ..
  • %run python -m source.foo.bar
• Unit testing code w/ pytest
• Code quality checks before submitting PR
  • Checking source code is pep8 standards: autopep8 --in-place --aggressive --recursive source/
  • Perform code linting w/ flake8: flake8 source/

Thoughts?

[chadac]

Nice thing about this module setup is that we've eliminated the need for the %cd command. As long as the notebook uses the same virtualenv, you can %run -m source.foo.bar anywhere and it will work. Definitely worth mentioning both approaches though.

Unit testing and code quality would be a great thing to add as well. Perhaps we can break that off into a separate issue?

[Tsmith5151]

Assuming the .ipynb is in notebooks/ you would need to set the working path up a level. Yep, will add an issue for testing.

[chadac]

Depends on what we're trying to access. If the notebook is within notebooks/, it should be able to access top-level packages as they are installed in editable mode. Need to create a testable example though. Let me create some notebooks and see what happens.

Ideally, I think notebooks should be organized into multiple subfolders without needing to change notebook code.