[Feature:InstructorUI] Capability / image buildtime error

[micpap25]

Works on #10747
Gives an error for gradeables at buildtime if the requested capability / image does not exist in the context of that gradeable.

To test, you can modify an existing config for a development course to either have a capability that does not exist or an image that does not exist and rebuild the gradeable for that course.

[micpap25]

There is an issue with this PR, in that the erroring process calls exit(1) which cancels the entire build, which is normal and fine for single builds but bad for the batch build that is called when the site is being installed (as some gradeables are expected to fail). Barb says it's good to change the build code to not error when any individual gradeable build errors

[micpap25]

@powe97 should work, please review!

[micpap25]

e2e testing needs at least 2 unused capabilities to work properly, so I added one to do so.

[DarthNyan]

I found this broke existing autograding examples, such as c_failure_messages and c_malloc_not_allowed. This occurred because "submitty/autograding-default:latest" is set as the default container when "container_image" is not specified, so any config running in jailed_sandbox mode on a machine lacking that image is expected to fail.

Probably good to discuss with Barb what the best fix is, as it will likely involve skipping this check for jailed_sandbox gradeables or editing a lot of existing examples.

I did confirm that it correctly failed to build when missing capabilities or images and correctly succeeded with a valid docker config.

[DarthNyan]

Discussed with Barb. Best fix is to bypass the buildtime image check for non-docker gradables, but we still want to confirm capabilities exist.

[micpap25]

I thought the current goal was to dockerize all configs. Is that incorrect?

[bmcutler]

@micpap25

There are some cases of autograding that will run on the machine directly, outside of docker, in the jailed sandbox. E.g., graphics grading that uses the graphics card & display.

@DarthNyan is pointing out that we have a default value for the docker container, even though we default the use of docker to false. When use of docker is false, it doesn't make sense to check for the presence of the unused-for-this-gradeable, default docker container.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 20.83%. Comparing base (1c8b53b) to head (4578046).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #10839   +/-   ##
=========================================
  Coverage     20.83%   20.83%           
  Complexity     9108     9108           
=========================================
  Files           259      259           
  Lines         34933    34933           
  Branches        459      459           
=========================================
  Hits           7277     7277           
  Misses        27201    27201           
  Partials        455      455           

Flag	Coverage Δ
autograder	21.34% <ø> (ø)
js	2.13% <ø> (ø)
migrator	100.00% <ø> (ø)
php	19.52% <ø> (ø)
python_submitty_utils	80.08% <ø> (ø)
submitty_daemon_jobs	88.88% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Chriun]

The functionality is good but it still does not pass integration checks. In addition the list of capabilities being checked does not encompass the entire list.