[SPARK-46658][DOCS] Loosen Ruby dependency specification

[nchammas]

What changes were proposed in this pull request?

As promised here, this change loosens our Ruby dependency specification so that Bundler can update transitive dependencies more easily.

Other changes included:

• Remove the direct dependency on webrick, because Jekyll fixed the problem that caused us to add it in the first place.
• Add explanatory comments to parts of the document generation process that are not obvious.

We can still build our docs using Ruby 2.7, but we should push devs to install Ruby 3 since Ruby 2 is EOL and we are unable to upgrade some of our doc dependencies until we're running Ruby 3.

Why are the changes needed?

Make the document building process more robust to future updates coming from the Ruby ecosystem.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

I built and reviewed the docs on both Ruby 2.7.8 and Ruby 3.3.0 using the following command:

SKIP_SCALADOC=1 SKIP_PYTHONDOC=1 SKIP_RDOC=1 bundle exec jekyll build

Was this patch authored or co-authored using generative AI tooling?

No.

[nchammas]

cc @HyukjinKwon @srowen - We should update the build images to include Ruby 3.

spark/dev/create-release/spark-rm/Dockerfile

Lines 84 to 85 in 11ac856

	$APT_INSTALL ruby2.7 ruby2.7-dev && \
	gem install --no-document $GEM_PKGS

spark/dev/infra/Dockerfile

Lines 59 to 60 in 4957c1a

	ruby \
	ruby-dev \

I would do this myself, but non-committers cannot really test changes to these images fully.

[HyukjinKwon]

Merged to master.