OPL release script pointing to an unofficial outdated release file

[xcompass]

It seems the file is not updated since last Sept.
https://github.com/openwebwork/webwork2/blob/master/bin/OPL_releases/Makefile#L2

What's the release cycle for this? Would it be better if it can be automated? e.g. using github action/workflow?

@heiderich

[taniwallach]

The idea behind having a webwork-open-problem-library-METADATA.tar.gz file with reasonably up-to-date metadata for the OPL was save the need for OPL_update to be run often on many individual webwork servers, as it is a very slow process.

@heiderich wrote up the code to make this possible, and the initial implementation (also in docker-config/docker-entrypoint.sh) refers to his forks. As he has not been active much for a while - I guess things are stuck as they were when he last made a "release".

It would be good to set up a long-term approach for this, and for the releases to be in the main OPL repository, once the approach is stable. It would be great if this the creation of releases could be somewhat automated, or if we could work out a reasonable approach to creating updated metadata releases frequently enough for this to be useful, while not overwhelming.

@xcompass - do you have ideas of how this could be automated?

Maybe some control-file in the master branch of the OPL could be used as an indicator of when a new metadata release should be created. @jwj61 @gajennings - what do you think? Would you be willing to modify such a control file or otherwise do something to trigger a metadata release after the master OPL branch had sufficient changes to warrant it? Do you have other ideas of how best to trigger a meta-data release.

@dlglin - Should this be in the project tracker somewhere?

[gajennings]

This sounds like a good idea. I don't know how to build such a file (would it be a mysql script that would take the result of opl-update and propagate it? or what?) but I'd be willing to modify the control file as needed if I knew how it worked. George George Jennings Math Dept., Calif. State University Dominguez Hills

…

________________________________ From: Nathan Wallach <notifications@github.com> Sent: Wednesday, October 14, 2020 1:16 AM To: openwebwork/webwork2 <webwork2@noreply.github.com> Cc: George Jennings <gjennings@csudh.edu>; Mention <mention@noreply.github.com> Subject: Re: [openwebwork/webwork2] OPL release script pointing to an unofficial outdated release file (#1118) The idea behind having a webwork-open-problem-library-METADATA.tar.gz file with reasonably up-to-date metadata for the OPL was save the need for OPL_update to be run often on many individual webwork servers, as it is a very slow process. @heiderich<https://urldefense.com/v3/__https://github.com/heiderich__;!!P7nkOOY!-aIWxp_yxUMb2XJmrnzBoqdMUvHpZbcobss7HU7N0XrT5rbbuBdZlmPU7AA5tkSZkw$> wrote up the code to make this possible, and the initial implementation (also in docker-config/docker-entrypoint.sh) refers to his forks. As he has not been active much for a while - I guess things are stuck as they were when he last made a "release". It would be good to set up a long-term approach for this, and for the releases to be in the main OPL repository, once the approach is stable. It would be great if this the creation of releases could be somewhat automated, or if we could work out a reasonable approach to creating updated metadata releases frequently enough for this to be useful, while not overwhelming. @xcompass<https://urldefense.com/v3/__https://github.com/xcompass__;!!P7nkOOY!-aIWxp_yxUMb2XJmrnzBoqdMUvHpZbcobss7HU7N0XrT5rbbuBdZlmPU7ADqfRJ2sQ$> - do you have ideas of how this could be automated? Maybe some control-file in the master branch of the OPL could be used as an indicator of when a new metadata release should be created. @jwj61<https://urldefense.com/v3/__https://github.com/jwj61__;!!P7nkOOY!-aIWxp_yxUMb2XJmrnzBoqdMUvHpZbcobss7HU7N0XrT5rbbuBdZlmPU7ABXj6TBBA$> @gajennings<https://urldefense.com/v3/__https://github.com/gajennings__;!!P7nkOOY!-aIWxp_yxUMb2XJmrnzBoqdMUvHpZbcobss7HU7N0XrT5rbbuBdZlmPU7ACcOVASWA$> - what do you think? Would you be willing to modify such a control file or otherwise do something to trigger a metadata release after the master OPL branch had sufficient changes to warrant it? Do you have other ideas of how best to trigger a meta-data release. @dlglin<https://urldefense.com/v3/__https://github.com/dlglin__;!!P7nkOOY!-aIWxp_yxUMb2XJmrnzBoqdMUvHpZbcobss7HU7N0XrT5rbbuBdZlmPU7ABba61mxw$> - Should this be in the project tracker somewhere? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/openwebwork/webwork2/issues/1118*issuecomment-708240074__;Iw!!P7nkOOY!-aIWxp_yxUMb2XJmrnzBoqdMUvHpZbcobss7HU7N0XrT5rbbuBdZlmPU7AB9dw8X7g$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABSBLKLKALODXYW5HVU2OCLSKVM4JANCNFSM4OL23ODA__;!!P7nkOOY!-aIWxp_yxUMb2XJmrnzBoqdMUvHpZbcobss7HU7N0XrT5rbbuBdZlmPU7AB3tRho_A$>.

[taniwallach]

I don't know how to build such a file (would it be a mysql script that would take the result of opl-update and propagate it? or what?) but I'd be willing to modify the control file as needed if I knew how it worked.

@heiderich wrote a script which is in WW 2.15: #1002 which created a "release" of the OPL which provides meta-data files, and an archive of the OPL at that point in time. It needs to be run on a WW server with the OPL installed and at the relevant branch/location. (A Docker setup suffices.) As I recall it runs OPL_update and then dumps the SQL tables into files, before doing the "GitHub" work to make a release. Ex: https://github.com/heiderich/webwork-open-problem-library/releases

Using the scripts requires setting up a token on GitHub which gives write access to the relevant (main/foked) OPL repository and can create the releases.

There is an explanation of how to use the tool it in the thread there. See #1002 (comment) where there are the steps to be run.

If you set it up and test it using your fork of the OPL, you could quickly set up a new meta-data release which the WW 2.15 distribution could be set to use.

In the long term, it would be better to have the releases in the main OPL repository, but a suitable access token with sufficient privileges would be needed to set that up.

The idea of a special file is related to providing a mechanism by which to indicate to a "agent" of some sort to automatically run the process to create a new release. @xcompass might be able to provide help and advice on how that could be done.

The tools in #1002 builds on prior scripts to dump and load the meta-data: #985 but are focused on making the process very simple to do.

[dlglin]

It looks like you can automate tasks in github using workflows:
https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions
I have to confess that I don't fully understand how to implement this, but based on the documentation scripts can be initiated either by certain triggers (a pull request for example) or on a schedule.

[jwj61]

I have never set up a github script, but have been involved in projects which did that.  One runs tests on code for pull requests, another ran pdflatex after every merge on a document several people were editing and sharing via github.

If someone wants to do this, that's fine.  I personally don't see this as a high priority since OPL-update is usually only run once per term, and for me, it just took 15 minutes on a machine which was already running at 100% from other jobs.

I will also remind people that we are looking for someone to replace me as the main overseer of the OPL.

[taniwallach]

As a temporary measure, I created a new OPL metadata release at https://github.com/taniwallach/webwork-open-problem-library/releases/tag/2021-08-03. I'm not changing the Makefile to use it now. I hope we can get a release made in the main OPL instead.

A manual procedure to get the updated metadata into a running Docker WeBWorK container:

1. First enter bash in the container: docker container exec -it webwork2_app_1 bash
2. Then run:

cd $WEBWORK_ROOT/bin/OPL_releases/
curl -L -o webwork-open-problem-library-METADATA.tar.gz https://github.com/taniwallach/webwork-open-problem-library/releases/download/2021-08-03/webwork-open-problem-library-METADATA.tar.gz
tar xfz webwork-open-problem-library-METADATA.tar.gz
mv /opt/webwork/libraries/webwork-open-problem-library/JSON-SAVED /opt/webwork/libraries/webwork-open-problem-library/JSON-SAVED.pre-2021-08-03
mv /opt/webwork/libraries/webwork-open-problem-library/TABLE-DUMP /opt/webwork/libraries/webwork-open-problem-library/TABLE-DUMP.pre-2021-08-03
cp -r webwork-open-problem-library $APP_ROOT/libraries/
cd $WEBWORK_ROOT/bin
./restore-OPL-tables.pl 
./load-OPL-global-statistics.pl

One the two metadata directories are in place, the docker-entrypoint.sh should just use those ones if it needs to reload something. Unless htdocs/DATA is mounted from outside the Docker container, this seems to occur on each startup, but takes very little time.

---

Note: The script only runs bin/OPL-update which does update tagging-taxonomy.json (which seems not to have changed, but does not update the other JSON files which are now only created when bin/updateOPLextras.pl is run. Those additional files do not seem to be in use.