Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: auto-mount edx-platform python requirements
These changes make to possible to run: tutor mounts add /path/to/my-xblock The xblock directory with then be auto-magically bind-mounted in the "openedx" image at build time, and the lms*/cms* containers at run time. This makes it effectively possible to work as a developer on edx-platform requirements. We take the opportunity to move some openedx-specific code to a dedicated module. Close openedx-unsupported/wg-developer-experience#177
- Loading branch information
Showing
12 changed files
with
310 additions
and
46 deletions.
There are no files selected for viewing
1 change: 1 addition & 0 deletions
1
changelog.d/20231009_164751_regis_mount_edx_platform_packages.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
- [Feature] Make it easy to work on edx-platform Python requirements with `tutor mounts add /path/to/my/package`. (by @regisb) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,130 @@ | ||
.. _edx_platform: | ||
|
||
Working on edx-platform as a developer | ||
====================================== | ||
|
||
Tutor supports running in development with ``tutor dev`` commands. Developers frequently need to work on a fork of some repository. The question then becomes: how to make their changes available within the "openedx" Docker container? | ||
|
||
For instance, when troubleshooting an issue in `edx-platform <https://github.com/openedx/edx-platform>`__, we would like to make some changes to a local fork of that repository, and then apply these changes immediately in the "lms" and the "cms" containers (but also "lms-worker", "cms-worker", etc.) | ||
|
||
Similarly, when developing a custom XBlock, we would like to hot-reload any change we make to the XBlock source code within the containers. | ||
|
||
Tutor provides a simple solution to these questions. In both cases, the solution takes the form of a ``tutor mounts add ...`` command. | ||
|
||
Working on the "edx-platform" repository | ||
---------------------------------------- | ||
|
||
Download the code from the upstream repository:: | ||
|
||
cd /my/workspace/edx-plaform | ||
git clone https://github.com/openedx/edx-platform . | ||
|
||
Check out the right version of the upstream repository. If you are working on the `current "zebulon" release <https://docs.openedx.org/en/latest/community/release_notes/index.html>`__ of Open edX, then you should checkout the corresponding branch:: | ||
|
||
# "zebulon" is an example. You should put the actual release name here. | ||
# I.e: aspen, birch, cypress, etc. | ||
git checkout open-release/zebulon.master | ||
|
||
On the other hand, if you are working on the Tutor :ref:`"nightly" <nightly>` branch then you should checkout the master branch:: | ||
|
||
git checkout master | ||
|
||
Then, mount the edx-platform repository with Tutor:: | ||
|
||
tutor mounts add /my/workspace/edx-plaform | ||
|
||
This command does a few "magical" things 🧙 behind the scenes: | ||
|
||
1. Mount the edx-platform repository in the image at build-time. This means that when you run ``tutor images build openedx``, your custom repository will be used instead of the upstream. In particular, any change you've made to the installed requirements, static assets, etc. will be taken into account. | ||
2. Mount the edx-platform repository at run time. Thus, when you run ``tutor dev start``, any change you make to the edx-platform repository will be hot-reloaded. | ||
|
||
You can get a glimpse of how these auto-mounts work by running ``tutor mounts list``. It should output something similar to the following:: | ||
|
||
$ tutor mounts list | ||
- name: /home/data/regis/projets/overhang/repos/edx/edx-platform | ||
build_mounts: | ||
- image: openedx | ||
context: edx-platform | ||
- image: openedx-dev | ||
context: edx-platform | ||
compose_mounts: | ||
- service: lms | ||
container_path: /openedx/edx-platform | ||
- service: cms | ||
container_path: /openedx/edx-platform | ||
- service: lms-worker | ||
container_path: /openedx/edx-platform | ||
- service: cms-worker | ||
container_path: /openedx/edx-platform | ||
- service: lms-job | ||
container_path: /openedx/edx-platform | ||
- service: cms-job | ||
container_path: /openedx/edx-platform | ||
|
||
Working on edx-platform Python dependencies | ||
------------------------------------------- | ||
|
||
Quite often, developers don't want to work on edx-platform directly, but on a dependency of edx-platform. For instance: an XBlock. This works the same way as above. Let's take the example of the `"edx-ora2" <https://github.com/openedx/edx-ora2>`__ package, for open response assessments. First, clone the Python package:: | ||
|
||
cd /my/workspace/edx-ora2 | ||
git clone https://github.com/openedx/edx-ora2 . | ||
|
||
Then, check out the right version of the package. This is the version that is indicated in the ``edx-platform/requirements/edx/base.txt``. Be careful that the version that is currently in use in your version of edx-platform is **not necessarily the latest version**:: | ||
|
||
git checkout <my-version-tag-or-branch> | ||
|
||
Then, mount this repository:: | ||
|
||
tutor mounts add /my/workspace/edx-ora2 | ||
|
||
Verify that your repository is properly bind-mounted by running ``tutor mounts list``:: | ||
|
||
$ tutor mounts list | ||
- name: /my/workspace/edx-ora2 | ||
build_mounts: | ||
- image: openedx | ||
context: req-edx-ora2 | ||
- image: openedx-dev | ||
context: req-edx-ora2 | ||
compose_mounts: | ||
- service: lms | ||
container_path: /openedx/requirements/edx-ora2 | ||
- service: cms | ||
container_path: /openedx/requirements/edx-ora2 | ||
- service: lms-worker | ||
container_path: /openedx/requirements/edx-ora2 | ||
- service: cms-worker | ||
container_path: /openedx/requirements/edx-ora2 | ||
- service: lms-job | ||
container_path: /openedx/requirements/edx-ora2 | ||
- service: cms-job | ||
container_path: /openedx/requirements/edx-ora2 | ||
|
||
It is quite possible that your package is not automatically recognized and bind-mounted by Tutor. In such a case, you will need to create a :ref:`Tutor plugin <plugin_development_tutorial>` that implements the :py:data:`tutor.plugins.openedx.hooks.EDX_PLATFORM_PYTHON_PACKAGES` patch:: | ||
|
||
import tutor.plugins.openedx.hooks as openedx_hooks | ||
openedx_hooks.EDX_PLATFORM_PYTHON_PACKAGES.add_item("my-package") | ||
|
||
After you implement and enable that plugin, ``tutor mounts list`` should display your directory among the bind-mounted directories. | ||
|
||
You should then re-build the "openedx" Docker image to pick up your changes:: | ||
|
||
tutor images build openedx-dev | ||
|
||
Then, whenever you run ``tutor dev start``, the "lms" and "cms" container should automatically hot-reload your changes. | ||
|
||
To push your changes in production, you should do the same with ``tutor local`` and the "openedx" image:: | ||
|
||
tutor images build openedx | ||
tutor local start -d | ||
|
||
Do I have to re-build the "openedx" Docker image after every change? | ||
-------------------------------------------------------------------- | ||
|
||
No, you don't. Re-building the "openedx" Docker image may take a while, and you don't want to run this command every time you make a change to your local repositories. Because your host directory is bind-mounted in the containers at runtime, your changes will be automatically applied to the container. If you run ``tutor dev`` commands, then your changes will be automatically picked up. | ||
|
||
If you run ``tutor local`` commands (for instance: when debugging a production instance) then your changes will *not* be automatically picked up. In such a case you should manually restart the containers:: | ||
|
||
tutor local restart lms cms lms-worker cms-worker | ||
|
||
Re-building the "openedx" image should only be necessary when you want to push your changes to a Docker registry, then pull them on a remote server. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,6 +9,7 @@ Open edX customization | |
|
||
plugin | ||
theming | ||
edx-platform | ||
edx-platform-settings | ||
google-smtp | ||
nightly | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
""" | ||
Hooks that only make sense in the context of Open edX. | ||
""" | ||
from __future__ import annotations | ||
|
||
from tutor.core.hooks import Action, Context, Filter | ||
|
||
#: List of python package names that can be potentially installed next to edx-platform. | ||
#: Whenever a user runs: ``tutor mounts add /path/to/name`` "name" will be matched to | ||
#: the regular expressions in this filter. If it matches, then the directory will be | ||
#: automatically bind-mounted in the "openedx" Docker image at build time and run | ||
#: time. They will be mounted in ``/openedx/requirements/<name>``. Then, ``pip install | ||
#: -e .`` will be run in this directory at build-time. And the same host directory | ||
#: will be bind-mounted in that location at run time. This allows users to | ||
#: transparently work on edx-platform dependencies. | ||
#: | ||
#: By default, xblocks and some common packages are already present in this | ||
#: filter. Add your own edx-platform dependencies to this filter to make it easier for | ||
#: users to work on edx-platform dependencies. | ||
#: | ||
#: See the list of all edx-platform base requirements here: | ||
#: https://github.com/openedx/edx-platform/blob/master/requirements/edx/base.txt | ||
#: | ||
#: :parameter list[str] names: Add here simple names ("my-package") or regular | ||
#: expressions. | ||
EDX_PLATFORM_PYTHON_PACKAGES: Filter[list[str], []] = Filter() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,132 @@ | ||
from __future__ import annotations | ||
|
||
import os | ||
import re | ||
import typing as t | ||
|
||
from tutor import bindmount | ||
from tutor import hooks | ||
from . import hooks as openedx_hooks | ||
|
||
|
||
@hooks.Filters.APP_PUBLIC_HOSTS.add() | ||
def _edx_platform_public_hosts( | ||
hosts: list[str], context_name: t.Literal["local", "dev"] | ||
) -> list[str]: | ||
if context_name == "dev": | ||
hosts += ["{{ LMS_HOST }}:8000", "{{ CMS_HOST }}:8001"] | ||
else: | ||
hosts += ["{{ LMS_HOST }}", "{{ CMS_HOST }}"] | ||
return hosts | ||
|
||
|
||
@hooks.Filters.IMAGES_BUILD_MOUNTS.add() | ||
def _mount_edx_platform_build( | ||
volumes: list[tuple[str, str]], path: str | ||
) -> list[tuple[str, str]]: | ||
""" | ||
Automatically add an edx-platform repo from the host to the build context whenever | ||
it is added to the `MOUNTS` setting. | ||
""" | ||
if os.path.basename(path) == "edx-platform": | ||
volumes += [ | ||
("openedx", "edx-platform"), | ||
("openedx-dev", "edx-platform"), | ||
] | ||
return volumes | ||
|
||
|
||
@hooks.Filters.COMPOSE_MOUNTS.add() | ||
def _mount_edx_platform_compose( | ||
volumes: list[tuple[str, str]], name: str | ||
) -> list[tuple[str, str]]: | ||
""" | ||
When mounting edx-platform with `tutor mounts add /path/to/edx-platform`, | ||
bind-mount the host repo in the lms/cms containers. | ||
""" | ||
if name == "edx-platform": | ||
path = "/openedx/edx-platform" | ||
volumes += [ | ||
("lms", path), | ||
("cms", path), | ||
("lms-worker", path), | ||
("cms-worker", path), | ||
("lms-job", path), | ||
("cms-job", path), | ||
] | ||
return volumes | ||
|
||
|
||
# Auto-magically bind-mount xblock directories and some common dependencies. | ||
openedx_hooks.EDX_PLATFORM_PYTHON_PACKAGES.add_items( | ||
[ | ||
r".*[xX][bB]lock.*", | ||
"edx-enterprise", | ||
"edx-ora2", | ||
"edx-search", | ||
r"platform-plugin-.*", | ||
] | ||
) | ||
|
||
|
||
def iter_mounted_edx_platform_python_requirements(mounts: list[str]) -> t.Iterator[str]: | ||
""" | ||
Parse the list of mounted directories and yield the directory names that are for | ||
edx-platform python requirements. Names are yielded in alphabetical order. | ||
""" | ||
names: set[str] = set() | ||
for mount in mounts: | ||
for _service, host_path, _container_path in bindmount.parse_mount(mount): | ||
name = os.path.basename(host_path) | ||
for regex in openedx_hooks.EDX_PLATFORM_PYTHON_PACKAGES.iterate(): | ||
if re.match(regex, name): | ||
names.add(name) | ||
break | ||
|
||
yield from sorted(names) | ||
|
||
|
||
hooks.Filters.ENV_TEMPLATE_VARIABLES.add_item( | ||
( | ||
"iter_mounted_edx_platform_python_requirements", | ||
iter_mounted_edx_platform_python_requirements, | ||
) | ||
) | ||
|
||
|
||
@hooks.Filters.IMAGES_BUILD_MOUNTS.add() | ||
def _mount_edx_platform_python_requirements_build( | ||
volumes: list[tuple[str, str]], path: str | ||
) -> list[tuple[str, str]]: | ||
""" | ||
Automatically bind-mount edx-platform Python requirements at build-time. | ||
""" | ||
name = os.path.basename(path) | ||
for regex in openedx_hooks.EDX_PLATFORM_PYTHON_PACKAGES.iterate(): | ||
if re.match(regex, name): | ||
volumes.append(("openedx", f"req-{name}")) | ||
volumes.append(("openedx-dev", f"req-{name}")) | ||
break | ||
return volumes | ||
|
||
|
||
@hooks.Filters.COMPOSE_MOUNTS.add() | ||
def _mount_edx_platform_python_requirements_compose( | ||
volumes: list[tuple[str, str]], name: str | ||
) -> list[tuple[str, str]]: | ||
""" | ||
Automatically bind-mount edx-platform Python requirements at runtime. | ||
""" | ||
for regex in openedx_hooks.EDX_PLATFORM_PYTHON_PACKAGES.iterate(): | ||
if re.match(regex, name): | ||
# Bind-mount requirement | ||
path = f"/openedx/requirements/{name}" | ||
volumes += [ | ||
("lms", path), | ||
("cms", path), | ||
("lms-worker", path), | ||
("cms-worker", path), | ||
("lms-job", path), | ||
("cms-job", path), | ||
] | ||
return volumes |
Oops, something went wrong.