Fix CI infrastructure: update Ruby matrix, fix Python pip failure

[ianks]

Situation

All Python and Ruby 3.0 CI jobs have been broken on main for a while. Python fails because bin/before-install tries sudo pip install -U pip, which blows up on newer Ubuntu runners where pip is Debian-managed and lacks a RECORD file. Ruby 2.7/3.0 fail because minitest 5.27+ requires >= 3.1, and both versions are EOL anyway.

Execution

Removed the unnecessary pip upgrade from bin/before-install — the system pip works fine. Dropped Ruby 2.7 and 3.0 from the CI matrix, added 3.4 and 4.0. Bumped required_ruby_version to >= 3.1 to match. Added rexml as a dev dependency since it was removed from stdlib in Ruby 4.0.

[Copilot]

Pull request overview

This PR updates CI and supporting code to restore broken Ruby/Python pipelines by aligning dependency/runtime expectations with newer runtimes and runner images.

Changes:

• Adjust Ruby CI/support code for newer Ruby versions (update matrix, bump required Ruby version, normalize/ignore new warning/backtrace noise in tests).
• Fix Python distributed queue Lua script invocations to match the Redis Lua script signatures and relax pytest output assertions to tolerate warning summaries.
• Remove the problematic pip self-upgrade from CI bootstrap and simplify the Python make test target to run tox only.

Reviewed changes

Copilot reviewed 13 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
ruby/test/support/output_test_helpers.rb	Adds backtrace normalization and filters stderr lines to reduce Ruby 4.x warning noise in tests.
ruby/test/minitest/queue/test_time_reporter_test.rb	Uses stderr filtering helper to keep assertions stable under newer Ruby warnings.
ruby/test/integration/rspec_redis_test.rb	Filters deprecations in stderr assertions and normalizes backtrace output for snapshot-like comparisons.
ruby/test/integration/minitest_redis_test.rb	Filters deprecations in stderr assertions and normalizes XML/backtrace output.
ruby/test/integration/minitest_bisect_test.rb	Filters deprecations and normalizes backtraces for bisect output stability.
ruby/test/integration/grind_redis_test.rb	Filters deprecations in stderr assertions for grind integration tests.
ruby/lib/minitest/queue/test_data.rb	Improves backtrace parsing regex to handle updated backtrace formatting.
ruby/ci-queue.gemspec	Raises required Ruby version and adds additional dependencies for Ruby 4.x compatibility.
ruby/Gemfile.lock	Updates locked dependency versions to match the new Ruby baseline and added deps.
ruby/.ruby-version	Pins local dev Ruby to 4.0.
python/tests/test_pytest.py	Makes pytest output matching tolerant of a warning summary line.
python/ciqueue/distributed.py	Updates Redis Lua script KEYS/ARGV to match the Lua scripts (ack/requeue/reserve).
python/Makefile	Makes make test run tox only (no autopep8 formatting step).
bin/before-install	Removes pip self-upgrade that breaks on Debian-managed pip.
.github/workflows/tests.yml	Updates Ruby/Python matrices and tweaks Python bootstrap steps for runner changes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

sudo pip install autopep8 || true masks installation failures and can hide real environment problems. Since make test no longer runs autopep8, consider removing this step entirely; otherwise, prefer installing with the intended interpreter (python -m pip ...) and letting the step fail if the dependency is required.

Suggested change

sudo pip install autopep8 || true

[Copilot]

Now that make test only runs tox, running it without setting TOX_ENV will use the Makefile default (which still includes py37). Since CI dropped Python 3.7, consider updating the default TOX_ENV list to match the supported matrix to avoid surprising local failures.

[Copilot]

rexml is required by the library at runtime (e.g., require 'rexml/document' in the JUnit reporter and configuration helpers). Declaring it as a development dependency means gem consumers on Ruby versions where REXML is no longer bundled may hit a LoadError. Consider making rexml a runtime dependency (add_dependency) instead of (or in addition to) add_development_dependency.

[Copilot]

filter_deprecation_warnings is named as if it only removes Ruby deprecation noise, but it also filters generic warnings like "already initialized constant" / "previous definition of". That can mask real issues in tests. Consider narrowing the filter to deprecation-specific messages (or renaming the helper to reflect broader suppression).

Suggested change

	line.include?("is not part of the default gems since Ruby") ||
	line.include?("warning: already initialized constant") ||
	line.include?("warning: previous definition of")
	line.include?("is not part of the default gems since Ruby")

[Copilot]

The reserve.lua logic now depends on build:<id>:workers being accurate (it uses SCARD to decide whether a worker can immediately pick up its own requeued tests). In the Python implementation, workers are added to this set but never expired/cleaned up, so stale worker IDs from prior runs with the same build_id can change behavior and add avoidable delays. Consider expiring the workers set (and possibly requeued-by) similarly to the Ruby implementation, or deleting it when the build finishes.