Skip to content

GullyBurns/databricks_to_nbdev_template

 
 

Repository files navigation

databricks to nbdev template

Use this template to more easily create an nbdev project that builds itself from databricks scripts.

This is a simple fork from the existing nbdev template with a couple of extra steps in the github action workflow.

1. install nbdev, jupyter, and the existing library (with dependencies)
2. run a script (`./db2nb/convert_databricks_to_jupyter.py`) that descends 
   the `databricks` directory tree and converts all the databricks workbooks
   there into *.ipynb files at the top level. There is currently 
   _no mirroring_ of the databricks file structure. 
3. Runs git add+commit+push to add the notebooks to Github
4. Uses nbdev to 
    a. run tests
    b. build library 
    c. build documentation
5. Runs git add+commit+push to add all generated material to Github  

Thus, the ideal methodology is to provide a round-trip methodology where coders can use Databricks to develop a script - and then seamlessly generate the notebooks, documentation, and libraries on github for easy reuse and potential deployment to non-databricks environments.

This should facilitate code publication and documentation within CZI.

Troubleshooting Tips

  • Start with a local copy of settings.ini and run nbdev_build_lib && nbdev_clean_nbs && nbdev_build_docs to instantiate your library.
  • Start with a local copy of the repo to set up the settings.ini file
  • Make sure you are using the latest version of nbdev with pip install -U nbdev
  • If you are using an older version of this template, see the instructions above on how to upgrade your template.
  • It is important for you to spell the name of your user and repo correctly in settings.ini or the website will not have the correct address from which to source assets like CSS for your site. When in doubt, you can open your browser's developer console and see if there are any errors related to fetching assets for your website due to an incorrect URL generated by misspelled values from settings.ini.
  • If you change the name of your repo, you have to make the appropriate changes in settings.ini
  • After you make changes to settings.ini, rerun nbdev_build_lib && nbdev_clean_nbs && nbdev_build_docs to make sure all changes are propagated appropriately.

Previewing Documents Locally

It is often desirable to preview nbdev generated documentation locally before having it built and rendered by GitHub Pages. This requires you to run Jekyll locally, which requires installing Ruby and Jekyll. Instructions on how to install Jekyll are provided on Jekyll's site. You can run the command make docs_serve from the root of your repo to serve the documentation locally after calling nbdev_build_docs to generate the docs.

In order to allow you to run Jekyll locally this project contains manifest files, called Gem files, that specify all Ruby dependencies for Jekyll & nbdev. If you do not plan to preview documentation locally, you can choose to delete docs/Gemfile and docs/Gemfile.lock from your nbdev project (for example, after creating a new repo from this template).

About

Template for nbdev projects

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 75.9%
  • Jupyter Notebook 24.1%