Ww216 patches stat updates

[mgage]

this makes www-data owner of htdocs/tmp
it also runs update-OPL-statistics to prevent complaints from the library about the statistics file not existing.

[mgage]

I've added startup instructions -- I think that this PR is now ready for testing.

If you are testing you will need to remove some cached files from your docker. Make sure these files are not being used
in some other project:

docker-compose down (stop running containers)
docker image rm webwork
docker image prune
docker volume rm webwork2_mysql
docker volume rm webwork2_oplVolume
cd ../ww-docker-data/courses
remove the admin course directory (rm -rf admin)

it's important that both admin course directory and the admin entry in the database are removed (in this case we removed the entire database). If the course database entry and the course directory are not in sync then you will get anomolous behavior.

return to webwork2 and follow the instructions in WeBWorKDocker startup instructions.md

[taniwallach]

I have not tested yet, but my experience is that running chown and chmod on htdocs/tmp on a production server where that directory is externally mounted (so persistent) using Docker leads to very significant startup delays.
I think that it is probably unwise to make that a part of the default startup process.

[mgage]

in this case webwork2/htdocs/tmp is internally mounted. In fact everything is internally mounted for this "out of the box experience" version of setting up webwork except for the courses (including admin course). Once this works I plan to use your notes and create instructions for docker in which things are externally mounted and ready for production. I'll keep your experiences in mind. I'll also do more tests on the current configuration and see if I can find start up delays. ty

[mgage]

update-OPL-statistics needs improvements. It balks if there is a course in the course directory but no corresponding tables in the database. This is a common reason why automatic docker builds fail. Fixing this could be part of a broader fix to reconcile or at least fail gracefully when there are discrepancies between the courses listed in the database and courses listed in the course directory

[dlglin]

I've never been happy with using the courses directory as the authoritative list of courses on the server, but short of creating a new data structure somewhere (e.g. additional DB table) I don't see a better way to generate the list.

[taniwallach]

I think we could add a phase to lib/WeBWorK/Utils/CourseManagement.pm to take the current list, and compare to what the DB returns from a call like

      show tables where Tables_in_webwork like '%_set_user';

which would return all the _set_user tables in the database (picking a sample table from the set of course tables).

Another option might be to query if one specific the table exists for a "course" candidate using the tableExists function from lib/WeBWorK/DB/Schema/NewSQL.pm or lib/WeBWorK/DB/Schema/NewSQL/Std.pm.

There are calls using this approach in 2 places:

./WeBWorK/Utils/CourseManagement.pm:1104:	    my $database_table_exists = ($db->{$table}->tableExists) ? 1:0;
./WeBWorK/Utils/CourseIntegrityCheck.pm:108:	    my $database_table_exists = ($db->{$table}->tableExists) ? 1:0;

so the potential to add such a test should be reasonable to implement.

[mgage]

Checking both the courses directory and the courses table in the database would be a good sanity check -- particularly with regard to the admin course. Several times I have removed the admin course, but not the table entry when trying to rebuild the docker box from scratch and get errors that I wasn't expecting.

[taniwallach]

I'm not sure there is a need for an "offline" file of instructions. There is already the Wiki page at https://github.com/openwebwork/webwork2/wiki/Docker-newbie-instructions.

I there is an intention to keep the file, I recommend moving it into either the docs of docker-config subdirectory and not to clutter up the root directory of the repository more than necessary.

[taniwallach]

I think that this should not be run on each startup.
I recommend touching a file in a suitable location once it was run once, and only running it if that file does not exist.

[taniwallach]

On production servers, the recursive -R on htdocs/tmp is a real slow-down on startup time.
If you really have a need for the chown and chmod on htdocs/tmp - I recommend putting it into different lines and not using -R but just processing all first and second level directories/files without going deeper. Something like

chown www-data:www-data htdocs/tmp htdocs/tmp/* htdocs/tmp/*/*
chmod ug+w htdocs/tmp htdocs/tmp/* htdocs/tmp/*/*

which I suspect should handle most issues without needing to recursively process the entire directory tree under htdocs/tmp.

[mgage]

replaced by patch5