added create_tutorials_html.py and tutorials.html.jinja

[rum1887]

Automatically lists qutip-qip tutorials in doc #204

Description

Added a Python script (create_tutorials_html.py) to list tutorials from qutip-tutorials repository

Running make html will run the script which in turn will clone qutip-tutorials directory into a temporary directory, fetch tutorials, generate HTML and tutorials_v5.rst files and finally wipe off the temporary directory.

Files generated runtime

HTML : doc/source/tutorials-website/qutip-qip.html and doc/source/tutorials-website/qutip-qip_v5.html
RST : doc/source/tutorials_v5.rst

---

Old Description

Added a Python script create_tutorials_html.py to fetch tutorials from qutip-tutorials repository and display it here. Used GitHub APIs to fetch the files. The script automates linking and syncing the tutorial files in qutip-tutorials with the qutip-qip repo.

I have pushed the generated tutorials.rst file as well. So, by cloning and running make HTML, you should be able to view the tutorials page with tutorials in qutip-tutorials as of today. Changes in the tutorials directory can be synced by running the script.

Steps to reproduce/sync

1. In the qutip-qip/doc/source/tutorials-website directory run

python3 create_tutorials_html.py

This should create qutip.qip.html and qutip-qip-v5.html in the tutorial-websites dir and tutorials.rst and tutorials-v5.rst in the source directory. (Source dir hosts individual .rst files that render as HTML files)

2. Move to the doc directory cd ../.. and run

make html

This should create the _build file in the doc directory. Navigate to tutorials.html to view the file

To-Do

• Add the newly created script create_tutorials_html.py to the makefile so it runs when html make is called.
• Sort files based on the title (optional)
• Add a link to version 5

Caveats

Can only run the script twice per hour due to rate limits (1000 requests per hour )on the number of GitHub API requests an unauthenticated (No access rights to the repo ) user can make

[rum1887]

Hello @BoxiLi !
All files inside qutip-qip
Looking forward to your feedback!

[BoxiLi]

The plot you show looks great!

Maybe one can create a separate folder under source for this. This will make the files cleaner.

And yes it would be great to add this to the build logic. You may need to add the new packages to doc/requirement.txt.

[rum1887]

The plot you show looks great!

Maybe one can create a separate folder under source for this. This will make the files cleaner.

And yes it would be great to add this to the build logic. You may need to add the new packages to doc/requirement.txt.

I have created a folder and added the libraries to requirements.txt

[BoxiLi]

The .DS_store is probably unnecessary? And you can use make html SPHINXOPTS="-W to turn warnings into errors when test locally.

[rum1887]

The .DS_store is probably unnecessary? And you can use make html SPHINXOPTS="-W to turn warnings into errors when test locally.
yeah will remove .DS_store.

I get the following error when I run make html SPHINX OPTS= "W"
Sphinx error:
Builder name SPHINX not registered or available through entry point
make: *** [SPHINX] Error 2

[BoxiLi]

I should be SPHINXOPTS I think, no space

[rum1887]

I should be SPHINXOPTS I think, no space

Warning, treated as error:
autodoc: failed to import module 'circuit' from module 'qutip_qip'; the following exception was raised:
No module named 'qutip_qip.version'
make: *** [html] Error 2

this is the error I get now, not related to what I have modified

[BoxiLi]

This is probably because you installed the release version of qutip-qip, but try to render the doc in the master branch. Maybe try installing the master branch version first. pip install . in the src folder.

[rum1887]

This is probably because you installed the release version of qutip-qip, but try to render the doc in the master branch. Maybe try installing the master branch version first. pip install . in the src folder.

worked thanks!

Warning, treated as error:
duplicate label tutorials, other instance in /Users/ramya/qutip-qip/doc/source/tutorials_v4.rst
make: *** [html] Error 2

File "tutorials.rst" and "tutorials-v5.rst" is generated using the same .jinja template. Sphinx is treating the v5 as a duplicate although the links are different. For now, I'll just remove tutorials_v4.rst.

And also there's another caveat, GitHub API requests have a rate limit of 1000 requests per hour for an unauthenticated user which means I can only run the script twice every hour (create_tutorials_html.py).

I had looked for other ways to sync up the tutorials like Git submodules, Git subtree etc, but they required duplication of files inside this repo.

[rum1887]

This is probably because you installed the release version of qutip-qip, but try to render the doc in the master branch. Maybe try installing the master branch version first. pip install . in the src folder.

worked thanks!

Warning, treated as error:
duplicate label tutorials, other instance in /Users/ramya/qutip-qip/doc/source/tutorials_v4.rst
make: *** [html] Error 2

• File "tutorials.rst" and "tutorials-v5.rst" is generated using the same .jinja template. Sphinx is treating the v5 as a duplicate although the links are different. For now, I'll just remove tutorials_v4.rst.
• And also there's another caveat, GitHub API requests have a rate limit of 1000 requests per hour for an unauthenticated user i.e. I can only run the script twice every hour (create_tutorials_html.py). I had looked for other ways to sync up the tutorials like Git submodules, Git subtree etc, but they required duplication of files inside this repo.

removing v4 builds without any error, how to remove .DStore, tutorials.rst from the commit tho?

[rum1887]

This is probably because you installed the release version of qutip-qip, but try to render the doc in the master branch. Maybe try installing the master branch version first. pip install . in the src folder.

worked thanks!

Warning, treated as error:
duplicate label tutorials, other instance in /Users/ramya/qutip-qip/doc/source/tutorials_v4.rst
make: *** [html] Error 2

• File "tutorials.rst" and "tutorials-v5.rst" is generated using the same .jinja template. Sphinx is treating the v5 as a duplicate although the links are different. For now, I'll just remove tutorials_v4.rst.
• And also there's another caveat, GitHub API requests have a rate limit of 1000 requests per hour for an unauthenticated user i.e. I can only run the script twice every hour (create_tutorials_html.py). I had looked for other ways to sync up the tutorials like Git submodules, Git subtree etc, but they required duplication of files inside this repo.

removing v4 builds without any error, how to remove .DStore, tutorials.rst from the commit tho?

removed XP
Now this builds without any error

[BoxiLi]

And also there's another caveat, GitHub API requests have a rate limit of 1000 requests per hour for an unauthenticated user i.e. I can only run the script twice every hour (create_tutorials_html.py). I had looked for other ways to sync up the tutorials like Git submodules, Git subtree etc, but they required duplication of files inside this repo.

Ok, didn't expect that. If GitHub action belongs to the "authenticated user", you can push things to your fork (e.g. master, but remember to reset it back to upstream after this) and let GitHub action in your fork test things.

Yes rebase is the right way to remove it!

Looks like a successful build!

[BoxiLi]

One question, is create_tutorials_html.py automatically executed when make html, or manually at the moment?

[rum1887]

One question, is create_tutorials_html.py automatically executed when make html, or manually at the moment?

manually at the moment

[rum1887]

One question, is create_tutorials_html.py automatically executed when make html, or manually at the moment?

running create_tutorials_html.py more than twice will throw an error, so if this is added to the build logic make html will also be throwing an error if its run more than twice ig .
There's no other way to access files from another GitHub repository without duplication.

[rum1887]

One question, is create_tutorials_html.py automatically executed when make html, or manually at the moment?

running create_tutorials_html.py more than twice will throw an error, so if this is added to the build logic make html will also be throwing an error if its run more than twice ig . There's no other way to access files from another GitHub repository without duplication.

every function in the create_tutorials_html.py has get(url) : sends multiple get requests to the qutip-tutorials

[BoxiLi]

Ok, so the main problem is the get(url). An alternative would be just git clone the tutorial repo in a temporary folder and work locally with it.

It is fine if the files are copied, the key point is that they are only stored temporarily and not pushed to the qutip-qip repo. Sorry if it wasn't clear. In the create_tutorials_html.py, you could create a temporary directory (see https://docs.python.org/3/library/tempfile.html#module-tempfile), get the files there, and work with them. And when it is done, the files will be removed automatically.

[rum1887]

Ok, so the main problem is the get(url). An alternative would be just git clone the tutorial repo in a temporary folder and work locally with it.

It is fine if the files are copied, the key point is that they are only stored temporarily and not pushed to the qutip-qip repo. Sorry if it wasn't clear. In the create_tutorials_html.py, you could create a temporary directory (see https://docs.python.org/3/library/tempfile.html#module-tempfile), get the files there, and work with them. And when it is done, the files will be removed automatically.

This should work, but we wanted the links to automatically sync with the qutip-tutorials repo. Temp dir is created by cloning qutip-tutorial, so the links will not be in sync with new changes.

[BoxiLi]

This should work, but we wanted the links to automatically sync with the qutip-tutorials repo. Temp dir is created by cloning qutip-tutorial, so the links will not be in sync with new changes.

I think I missed something here so I don't really get it. Given that

• The create_tutorials_html.py is executed every time when the doc is built by make html.
• And it clones the up-to-date repo at runtime.

Every time the doc is built, it uses the newest links of the tutorials, right? This should be sufficient, as long as we trigger the build every week/day.

In the end, even with request, we need something to trigger create_tutorials_html.py on a weekly/daily basis, so that it refreshes the link. Cloning the tutorial every time should be the same.

[rum1887]

yeah thanks for the clarification

I have a pending question.

Where are the files that generate the other .rst files in the source folder? I don't completely get how they are generated by Sphinx.

[BoxiLi]

None of the srt files in the source folders are generated by Sphinx at runtime. (Also, if they are generated at runtime, we don't need to save them in the repo.)
The only exception is apidoc, which was generated automatically by Sphinx and then optimized by hand (https://qutip-qip.readthedocs.io/en/stable/contribution-docs.html#api-documentation). This kind of thing is usually done by changing the configurations in conf.py.

[rum1887]

Great, the test passed. It should be safe now to remove the manually generated tutorial_v5.rst I assume?

yes

[rum1887]

This is really close now! The official Hackathon ends today but if I understand the rules correctly, it is ok to add a few fine-tuning commits afterwards.

yup

[rum1887]

@BoxiLi this works as expected. This can be merged now ig

[BoxiLi]

Thanks! I'll add a few fine-tuning comments in a day or so.

Right now, however, if I click on the generated links to the notebook, I get 404 error. Did something go wrong with the links?

If you want to check, you can download the generated HTML in the action artefacts https://github.com/qutip/qutip-qip/actions/runs/5259223981?pr=207.

[BoxiLi]

It is ok if the list is only generated for qutip v5, but could you add the link of the main v4 tutorial page (https://qutip.org/qutip-tutorials/) to the sentence

You can find the tutorials for QuTiP Version 4 here.

[rum1887]

Thanks! I'll add a few fine-tuning comments in a day or so.

Right now, however, if I click on the generated links to the notebook, I get 404 error. Did something go wrong with the links?

If you want to check, you can download the generated HTML in the action artefacts https://github.com/qutip/qutip-qip/actions/runs/5259223981?pr=207.

@BoxiLi The links should be working now

[BoxiLi]

Nope, something is still missing. I guess a / after tmp?

[rum1887]

now?

[rum1887]

Thanks! I'll add a few fine-tuning comments in a day or so.

Right now, however, if I click on the generated links to the notebook, I get 404 error. Did something go wrong with the links?

If you want to check, you can download the generated HTML in the action artefacts https://github.com/qutip/qutip-qip/actions/runs/5259223981?pr=207.

@BoxiLi Do you have any feedback?

[BoxiLi]

Here you go. Looks pretty good! A few minor changes. Thank you very much.

Could you also add pandoc to contribution-docs.rst

[rum1887]

Here you go. Looks pretty good! A few minor changes. Thank you very much.

Could you also add pandoc to contribution-docs.rst

@BoxiLi You want me to add something like this ?

A few .rst files are generated runtime from html files using the pandoc https://pandoc.org_ tool.

[BoxiLi]

A few .rst files are generated runtime from html files using the pandoc https://pandoc.org/_ tool.

Yes, something like The generation of the documentation also requires pandoc (with a link to their official web).

So that people know that they need to install it and how.

[rum1887]

A few .rst files are generated runtime from html files using the pandoc https://pandoc.org/_ tool.

Yes, something like The generation of the documentation also requires pandoc (with a link to their official web).

So that people know that they need to install it and how.

alright, done

[rum1887]

@BoxiLi so, this is not pushed to the stable version of the doc? need more changes or additional issues to be resolved?

[rum1887]

I'm open to contribute more

[BoxiLi]

Well spotted! It is missing from the rendered version on readthedoc indeed. Didn't notice it... I opened an issue #219. Will have a closer look in the next few days but of course you are also welcome to dig a bit into it.