determine log file/discovery report location for user job output

[jmartin-sul]

should be deterministic/predetermined, should not be specified by users of the web app.

work break down:

• the parent directory will be a setting (PRs use yml settings and dor_services initializer for configuration #269, Job output parent dir #311) (allows for easy dev and testing vs. prod)
• user_id/bundle_dir portion of path will be deterministically computed from bundlecontext (Job output parent dir #311)
• use project name instead of bundle_dir (see comment below);
  • ensure project name works for linux file system (validation, or change troubling chars to X or ...) bundle_context: project_name alphanum-ish only; output_dir uses project_name instead of bundle_dir #320 bundle_context: output_dir REALLY uses project_name #324
  • ensure project name is unique for new submissions (and ideally, let user know of the problem - see BC controller: when save fails, better error page to user #310)
• output files written to approp directory, instead of how it is done now. (probably with a timestamp as part of name to distinguish multiple runs)

Plan:

(writeable_parent_dir)/user_id/bundle_dir/job_output_here.whatever

requirements:

• Ben or whomever will occasionally need to delete the progress.yml file (not the progress log) from the preassembly run.
• the bundle dir must be unique to the user, but multiple users may use the same bundle names so the user_id dir is a sort of namespacing for the bundles.
• the bundle dir name will be the name of the input bundle dir from the user (to make it easy for Ben/whomever may need to manually traverse these dirs)
• the output files for preassembly will be something like
  • my_bundle_progress.yml
  • my_bundle.log
• the output file for discovery report will be something like
  • my_bundle_discovery_report.json

NOTE: this does not mean there aren't other ways to do this, e.g. use the database to store all the artifacts, or use the database to assign the unique location for the files ... just for now, the team agreed the file system was the easiest way forward, especially since Ben occasionally has had to manually delete progress files.

[jmartin-sul]

further elaboration: this won't be a single log file for everything. it'll likely be a pre-determined sub-path within the user's pre-assembly bundle (thus, different for every job, but easily computed from user input).

[blalbrit]

We discussed earlier today that many of the mounts are read-only to the preassembly server. Would consider choosing a single writable path for this.

Slightly aged list of mount paths is here: https://consul.stanford.edu/display/CollAcc/Content+Mount+Paths - I'll take a pass and make sure this is up to date.

@ndushay @jmartin-sul

[atz]

We could also use the dumbest of document stores as the permanent persistence home for "output" files.

[jmartin-sul]

• keep progress log file functionality

• we will keep the bundle mounts read-only

  • thus, progress log file and discovery report output must go to a location outside the user's bundle directory.

• we have /dor/preassembly as a writeable location for output.

  • need a naming convention for where to write output from a run.

• not this work cycle: @blalbrit says no need to keep output more than a month, could maybe keep it for less, file a (devops?) ticket for rotating out old output via cron/whenever.

  • output in the hundreds of KB per run, and we only expect 10s of runs per month, so we can punt on this for now, shouldn't use much space.
  • if we do prune output, we'll have to update the DB record to indicate that the output has been deleted

• ticket for future work cycle: investigate document stores

  • same caveat about noting culled output applies to this solution too

[ndushay]

On changing from bundle_dir to project name in output directory:

[ndushay]

On which characters to allow in project name: