Skip to content

run upstream tests #503

Open
Open
@tianon

Description

@tianon

Similar to docker-library/golang#552, there are a lot of upstream tests we could run and should run somewhere, but during build doesn't necessarily make sense.

In https://salsa.debian.org/ruby-team/ruby/-/blob/5b91c2c6b64195da7110d341daba904b9d12c9d5/debian/tests/run-all, we can see that Debian does some clever things to run a lot of the upstream tests post-build via "autopkgtests" (which is very akin to how we run tests on our images after they're built -- they typically run against the built + installed packages).

The basic gist is ruby test/runner.rb, and Debian's got a pretty hefty collection of tests they exclude which will probably inspire tests we need to exclude too (especially # FIXME for now, we are excluding the tests for C extensions; couldn't figure out how to properly build them without building everything else, since we'll have the same issue with doing this post-build).

So we'll need to recreate our "download the source code" logic to get the test files back, but that's the easy part. My own very basic testing of this idea on ruby:slim comes up with "7 failures, 6 errors, 176 skips", so there's clearly some work to do here (for each of those, fixing the image, accommodating the test somehow with runtime flags/packages/etc, or building up a list of tests to skip and writing down justifications for why we think they're valid to skip/ignore).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions