Running tests on Windows

[cderv]

WIP: Draft PR until it is working

• Concert run-test.sh to a run-test.ps1 version
• ./configure.cmd should run on CI - adapt our custom quarto-dev action
• Deal with python at as simply as possible to make this works
• Deal with Julia environment (new tests added in Add tests for Julia #4012)
• Adapt our actions to handle windows OS too
• Fix renv activation when running tests (issue)
• Improve renv caching to respect R version
• Fix failed tests (possibly in the PR or open issue for each one found)
  • Installing extension from url fails on Windows #4124
  • Do not error on file existence when filtering resources #4255
  • Wrong figure folder built using custom format due to mutation #4260
  • notebook-view appends URL to figure path somewhere and this error on Windows  #4337
  • Fix embeding notebook when external href is provided #4345
• Adapt PR to recent changes to workflows on multicore test suite #4354 and following commits 19fcc97...8c538d3
• Improve the python environment setup as this is not well designed for local and CI usage currently.
• Improve the julia environment setup to be like Python and R

Follow up work

• Adapt tests for qualified paths for Windows #4339

The following are related to ignored tests on Windows and should be fixed

• Specific handling for custom writer lua file for R Markdown processing #4347
• Removing empty dir is not working as wrong path is created (when outputing to stdout) #4194

Adaptations

• Removing ./tests/bin/* folder
  • This folder and source bin/activate were not useful (some hardcoded miniconda path in it). Removing it seems to work ok.
• Adding a .python_version file to pin a python version to use in tests/
  • This is understood by setup-python
  • It allows to use pyenv to automically use the right version of python in when in tests/
• Supporting Windows in our custom action quarto-dev which install quarto development version
  • It was not an easy piece to make things work on Unix and Windows. Especially because we run a quarto.cmd, which is not found in PATH on windows's bash. Using cmd shell is a pain as it seems you can't really set %GITHUB_PATH% on GHA (🤷‍♂️ ). So for now, it is hardcoded path but won't probably change soon. Anyway it is working now.
• Adding some caching for Julia plus adaptation so that it runs on Windows too.

Probably some more to change

• We are setting up environment in GHA in some steps, but we are also doing renv::restore() and pip install in the run-test. It seems a duplication. I am thinking of separating config-tests-env script for configure a test environment, and then having run-test to just run the test. But still need to think through CI usage vs Local usage.

• Having a quarto.ps1 instead of quarto.cmd would be better for dev windows user. I never used CMD, and other windows dev probably use either Powershell or Git Bash.

Notes on developing tests for Linux and windows

• When using regex matching, newline needs to be OS independant (3452940)
• If a tests does not need to be run on several OS (too costly to adapt, or not necessary to make OS independant), it can be easily ignored (418d077)
• Some calls to quarto needs to be explicitly made to quarto.cmd on Windows because in dev version quarto.exe is not available and quarto without extension will not be found (6be8232)
• Windows file locking system must be taken into account (example in cleaning where one need to move out a directory before removing it c29a182)
• Windows paths specificity needs to be dealt with (example with path ending / vs \ 9f37289)

More comments to come as I go through with it

[dragonstyle]

I would like to note that this is an extremely exciting PR.

[cderv]

This is now ready for integration. I just launched a last test with the latest change from main.

All tests passes except 2 or 3 I think that I ignore for now until we fixe them. There are quite a few changes for local environment so it is worth reading and checking before we merge. A README is available to explain.

This PR

• Revisit how we set up our testing environment locally (for R Python Julia) - Currently it is with locked state we'll update on the go. Challenge was to have something working for multi OS seemlessly; Using pipenv for Python helps with that
• Adapte run-tests.sh and add a run-tests.ps1
• Adapt GHA workflow to run on windows
• Fix some tests
• Fix some bugs in code (and other fix were done in external PR)
• If anyone is available tomorrow for a quick demo and presentation, and to test the new local configuration for test environment that would be cool.

Hopefully the change will work for everyone local environment.

Limitation and what to do next

Currently we rely on testing our code with locked state of packages for Julia, R and Python. This is using tools that will rely on lockfile to recreate a fix state.

This means that we'll need to update the different lockfile on regular basis - I should add a note in the README probably

Making it possible to test with latest version of all packages will be next in line among test environment improvment.

[cderv]

@dragonstyle putting you as reviewer as you seemed to be available and interested to test all this locally together.

[cscheid]

if [[ -z $FORCE_NO_PIPENV ]]
then
  OLD_VIRTUAL_ENV=$VIRTUAL_ENV
  if [[ -n $VIRTUAL_ENV ]]
  then
    deactivate
  end
  echo "> Activating virtualenv for Python tests"
  source "$(pipenv --venv)/bin/activate"
  quarto_venv_activated="true"
end
# ...
# at the end
# this is probably not correct if we were not in a venv but you get the idea.
if [[ -n $OLD_VIRTUAL_ENV ]]
then
  deactivate
  source $OLD_VIRTUAL_ENV/bin/activate
end

[cderv]

@cscheid

So I did another approach as deactivating was not robust. For example I am using pyenv on Linux to manage my python and virtualenv and deactivate was not existing.

So I leveraged some pipenv feature. Here is how it works

• pipenv will always create a virtual environment even when run in another virtuaenv. This means by default it will activate the new virtualenv. This environment will be located in tests/.venv
• If I detect that $VIRTUAL_ENV is set though before activating with pipenv, I remember this, and at the end, I run source $OLD_VIRTUAL_ENV/bin/activate to reactivate the initial virtualenvironment.
• To skip the default behavior QUARTO_TESTS_FORCE_NO_PIPENV can be set to any value. In that case, tests are running in the current python environment with no change.

I believe this work well and answer our discussion.

[cderv]

Thanks @cscheid and @dragonstyle - I'll still have some last adjustments as discussed today with you both, and this will be ready to merge tomorrow ! 🎉

So cool !

[cderv]

I did the modification for the virtual environment, and added some information to the documentation. Ready now to merge !

[cscheid]

Feel free to merge anytime!