Add experimental flag to enable YJIT compiler

[sethboyles]

enables YJIT compiler on CC jobs.

YJIT is optimized for long running processes, trading a higher memory usage for speed improvements.

Most of the jobs worked fine without alteration with default config values, but I did have to bump up the memory thresholds on the cc-worker.

for more on running YJIT:
https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md
https://news.ycombinator.com/item?id=29161810

• I have viewed signed and have submitted the Contributor License Agreement

• I have made this pull request to the develop branch

• I have run CF Acceptance Tests on bosh lite

[sethboyles]

we may want to separate the config flag into separate flags per job, so you can just enable the API server with YJIT, for example

[rkoster]

Has this functionality ever been tested? I'm wondering how this would have worked given the fact that the ruby package provided by the bosh-packages-ruby-release does not pass the --enable-yjit compile flag.

[sethboyles]

AFAIK the --enable-yjit compile-time flag was added with Ruby 3.2, because they changed some things to use Rust (so you need Rust installed to compile (and maybe use?) yjit). But in Ruby 3.1 it was just available in CRuby. Compare the docs for 3.1:

https://github.com/ruby/ruby/blob/ruby_3_1/doc/yjit/yjit.md

Installation
Current YJIT versions are installed by default with CRuby. Make sure to specify the "--yjit" command line option to enable it at runtime.

vs 3.2:

Installation
Requirements
You will need to install:

A C compiler such as GCC or Clang
GNU Make and Autoconf
The Rust compiler rustc and Cargo (if you want to build in dev/debug mode)
The Rust version must be [>= 1.58.0](https://github.com/ruby/ruby/blob/ruby_3_2/yjit/Cargo.toml).
To install the Rust build toolchain, we suggest following the [recommended installation method](https://www.rust-lang.org/tools/install). Rust also provides first class [support](https://www.rust-lang.org/tools) for many source code editors.

So, yeah, I had verified this was working with Ruby 3.1 by checking the logs (just printing Ruby version to see the +YJIT addition, but it clearly doesn't work anymore. I don't think anyone did any testing on the actual performance improvements.

[sethboyles]

you can also check this yourself by installing ruby 3.1 and 3.2 and running ruby -v --yjit.

3.1

$ ruby -v --yjit
ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) +YJIT [x86_64-linux]

3.2

$ ruby -v --yjit
ruby: warning: Ruby was built without YJIT support. You may need to install rustc to build Ruby with YJIT.
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]

[rkoster]

Thanks @sethboyles, that's great context to have, I did not know that the rust dependency was introduced later.