chore: Move codegen venv setup into build stage

[kuznetsss]

High Level Overview of Change

This PR moves python venv setup into build stage. Also move venv setup into python code so venv is created only when generation is needed.

Context of Change

Python venv setup is relatively slow so moving it into build stage helps build system to do something else in parallel while python dependencies are being downloaded.

API Impact

• Public API: New feature (new methods and/or new fields)
• Public API: Breaking change (in general, breaking changes should only impact the next api_version)
• libxrpl change (any change that may affect libxrpl or dependents of libxrpl)
• Peer protocol change (must be backward compatible or bump the peer protocol version)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.4%. Comparing base (6e24522) to head (c5aeb62).
⚠️ Report is 24 commits behind head on develop.

Additional details and impacted files

@@            Coverage Diff            @@
##           develop   #6617     +/-   ##
=========================================
- Coverage     81.4%   81.4%   -0.0%     
=========================================
  Files          999     999             
  Lines        74423   74423             
  Branches      7557    7553      -4     
=========================================
- Hits         60613   60606      -7     
- Misses       13810   13817      +7     

see 5 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mathbunnyru]

A few comments, nothing major

[a1q123456]

I think there're a couple of things we can do

1. remove the macro files from the reconfigure dependency list so that we don't reconfigure automatically after the macro files get changed
2. generate a stamp file on the first time when we configure (when the file does not exist) to say we don't need to regenerate
3. when we build the project, we check if the stamp file is up to date, call the scripts and update the stamp file if not

Because currently, cmake still regenerates even if we touch the macro files and we want to avoid regeneration as much as possible.

[a1q123456]

I'm just thinking, does it make sense to call list_output each time when we run cmake?

I think what we expect is:

1. if the macro files haven't been updated - we don't regenerate
2. if the macro files have been updated by git - we don't regenerate
3. if the macro files have been modified by the current develop - we generate

but what we're doing currently is:

1. if the macro files haven't been updated - we don't regenerate
2. if the macro files have been updated - we reconfigure, call list_output.py, and regenerate

Would it be easier to have a stamp file containing the hash of the script file, the template files, and the macro files, and then, we check the stamp file into git, and we only generate when the hash is different?

Doing so means that:

1. when git pulls a commit that modifies the macro files, the stamp file must be in the commit as well - we don't regenerate
2. when the developer modifies a macro file, cmake checks the hash and decides to regenerate

I think this is the safest way

[github-actions]

This PR has conflicts, please resolve them in order for the PR to be reviewed.

[github-actions]

All conflicts have been resolved. Assigned reviewers can now start or resume their review.

[a1q123456]

following up the discussion, we'll move the code generation process outside of cmake.

[bthomee]

I'm somewhat concerned it will be too confusing for developers to keep track of which cmake commands they need to run and when.

I also wonder if it will work fine in IDEs - after I run the conan install command, my IDE for instance will automatically pick up the CMake profile and configure everything; all I need to do then is build the code. Will my workflow have to change? Do I need to remember to run setup_code_gen and/or code_gen each time I switch branches, for instance?

[a1q123456]

I'm somewhat concerned it will be too confusing for developers to keep track of which cmake commands they need to run and when.

I also wonder if it will work fine in IDEs - after I run the conan install command, my IDE for instance will automatically pick up the CMake profile and configure everything; all I need to do then is build the code. Will my workflow have to change? Do I need to remember to run setup_code_gen and/or code_gen each time I switch branches, for instance?

I'm somewhat concerned it will be too confusing for developers to keep track of which cmake commands they need to run and when.

I also wonder if it will work fine in IDEs - after I run the conan install command, my IDE for instance will automatically pick up the CMake profile and configure everything; all I need to do then is build the code. Will my workflow have to change? Do I need to remember to run setup_code_gen and/or code_gen each time I switch branches, for instance?

No, you only have to run cmake --build . --target code_gen when you update the macro files or the template files, and the ci pipeline will ask you to do it if you update those files but forget to run the scripts