drake-ci: Allow manually publishing "experimental" releases from PRs?

[EricCousineau-TRI]

For authorized users (e.g. @RussTedrake, @gizatt, @peteflorence), there may be points in which they'd like to use a non-nightly release, either for prototyping or as a hotfix for the Underactuated course.

Per f2f with @jwnimmer-tri, it'd be nice if there was a workflow where a user could request a release from a PR by mentioning drake-jenkins-bot with the desired image name (either linux-xenial-unprovisioned-gcc-bazel-experimental-snopt-packaging or mac-highsierra-unprovisioned-clang-bazel-experimental-snopt-packaging), that could be published to an unstable/ URL with the pull request number, and possibly the Git SHA (for robustness if it's revised and re-published).

Would the main challenge here be how to prune stale items?

[jamiesnape]

Several years ago now, we did used to archive build artifacts, so it is perfectly possible. We stopped because the plugin was flaky, primarily on Windows builds. I can look into it.

[EricCousineau-TRI]

Awesome, thanks!

[jwnimmer-tri]

(Aside: if the bot is the hard part, the "Build Now" interface is also reasonably easy to use.)

My thought here was that https://github.com/RobotLocomotion/drake-ci/blob/master/driver/configurations/bazel/step-build.cmake#L142 could be amended to push to S3, if that was easy?

[jamiesnape]

Pushing to S3 is trivial, so if people don't mind looking in the Jenkins console log for a URL in the short term with an automatic expiration based on N days, they could have this today.

[EricCousineau-TRI]

That'd be excellent. Sorry that I wrote that poorly - the @drake-jenkins-bot is just a nice-to-have, but ultimately being able trigger the job somehow (via Build with Parameters on a given PR, and inspect the log afterwards) would definitely work too!

[jamiesnape]

The unstable built on continuous do actually go to S3 already, they are just not timestamped or the history kept somewhere immediately accessible.

• https://drake-packages.csail.mit.edu/drake/continuous/drake-latest-mac.tar.gz
• https://drake-packages.csail.mit.edu/drake/continuous/drake-latest-xenial.tar.gz

[EricCousineau-TRI]

The caveat to the continuous packages is that these changes must be on master. We'd like people to publish packages on an experimental PR (possibly late at night or very early in the morning, when no TRI reviewers are available to permit the merge).

[jamiesnape]

The one issue that I would need to check is "for authorized users". Easy with the Jenkins interface, less so with the semi-broken stable of the plugin responsible @drake-jenkins-bot ... please.

[EricCousineau-TRI]

In that case, maybe punt on the @drake-jenkins-bot plugin, and leave that for improvement if the Build with Parameters workflow is too cumbersome?

[jamiesnape]

The caveat to the continuous packages is that these changes must be on master. We'd like people to publish packages on an experimental PR (possibly late at night or very early in the morning, when no TRI reviewers are available to permit the merge).

Yes, it was just a clarification as @jwnimmer-tri pointed to a line of code that does lead to an unloaded file.

[EricCousineau-TRI]

Per f2f with @jamiesnape, it seems like using the Build with Parameters workflow should work well for experimental packaging. It does not seem like we'd saturate our S3 account if these packaging requests are triggered manually, and uses the same authentication bits as the rest of the Jenkins infrastructure.

The experimental artifacts would last as long as the Jenkins build (28 days); if the artifact should last longer than that, then the artifact should be hosted elsewhere.

[RussTedrake]

Thanks all! I’ll ask tomorrow in the office to make sure I know the specific interface you’re recommending.

[jamiesnape]

These jobs are now setup to publish:

• https://drake-jenkins.csail.mit.edu/view/Packaging/job/linux-xenial-unprovisioned-gcc-bazel-experimental-snopt-packaging/
• https://drake-jenkins.csail.mit.edu/view/Packaging/job/mac-highsierra-unprovisioned-clang-bazel-experimental-snopt-packaging/

If you look toward the end of the console output you will see a line similar to:

-- Package URL 1 of 1: https://drake-packages.csail.mit.edu/drake/experimental/drake-<timestamp>-<commit>-<mac|xenial>.tar.gz

[RussTedrake]

Roger. Awesome. Thx.

[EricCousineau-TRI]

Tested via #7936. Will re-open if Russ encounters issues using this with underactuated CI.