Cache

[janiversen]

with python in place, we can make our own cache key, and thereby only have:

3 (Windows, Ubuntu, MacOS) x 5 (python 3.8 - 313) = 15 cache.

We need to avoid making new caches for each pull request, that currently is the cause of why some jobs are slow in starting and that the real clock time is too high.

Current timing is at 15 minutes 2 seconds.

1. Change work order since windows takes longer it should be started first.

[alexrudd2]

The two other big slowdowns:

(1) Switching to using pre-commit means that its environment needs to be initialized
(2) #1673 removed the parallel execution with pytest-xdist. (Hopefully the Python3.11 problem is transient and this can be reverted)

[janiversen]

Removing xdist did not make a big difference because I controlled something like 20 runs in we más had 2 cores. But of course once I get rid of the spontaneous errors I will activate it again.

I made another change that speed things up, nullmodem ! the tests (apart from a few) no longer uses the network but communicates through a couple of class variables

Pre-commit is actually expensive in CI, but your idea with the env variable could be the solution.

[alexrudd2]

I made another change that speed things up, nullmodem ! the tests (apart from a few) no longer uses the network but communicates through a couple of class variables

o_0 that's great!

Pre-commit is actually expensive in CI, but your idea with the env variable could be the solution.

What if we don't use it in CI? We can use something like renovatebot to keep the versions in pre-commit in sync with the versions in pyproject.toml.

[janiversen]

That's another possibility, however I have no experience in using that bot...it needs to read the precommit yaml and then somehow update py project.toml

I hope that soon precommit will be able to use toml so we can get rid of this double configuration.

[alexrudd2]

That's another possibility, however I have no experience in using that bot...it needs to read the precommit yaml and then somehow update py project.toml

It's a great bot; if you'd like I can set it up. Here's an example of updating both files at the same time.

I hope that soon precommit will be able to use toml so we can get rid of this double configuration.

Unfortunately, the maintainer is an arrogant jerk that will never do this: pre-commit/pre-commit#1165. :(

[alexrudd2]

Also, we should be able to drop black soon in favor of ruff
https://github.com/astral-sh/ruff/blob/main/crates/ruff_python_formatter/README.md

In fact it could be done today, if you are OK with some small formatting changes and using an alpha.

[janiversen]

Also, we should be able to drop black soon in favor of ruff https://github.com/astral-sh/ruff/blob/main/crates/ruff_python_formatter/README.md

In fact it could be done today, if you are OK with some small formatting changes and using an alpha.

I am all in favor of moving towards ruff, so please do..

If I remember right you were also closing in on pylint?

Regarding the bot, I will have to read a bit more, I am pr nature sceptical with apps that run outside github, but it does look like a good candidate. An alternative is precommit.ci that seems to do a similar thing.

I think github also have a bot that updates, I remember we use (or used) that in Homeassistant.

[alexrudd2]

I am all in favor of moving towards ruff, so please do..

OK, I will test the alpha. It has ~80 minor formatting changes compared to black right now.

If I remember right you were also closing in on pylint?

Many of pylint's rules can be done by ruff, but it does more thorough checks and cannot be fully replaced yet (astral-sh/ruff#970)

Regarding the bot, I will have to read a bit more, I am pr nature sceptical with apps that run outside github, but it does look like a good candidate. An alternative is precommit.ci that seems to do a similar thing.

precommit.ci suffers from the problem of the same arrogant author and does not play well with other tools. I would not recommend it.

I think github also have a bot that updates, I remember we use (or used) that in Homeassistant.

Yeah, it's dependabot. Not a bad bot, but renovatebot is significantly better.

[janiversen]

@alexrudd2 I think you are trying to give me a heart attack !! I was sitting here testing caching, it worked find until I suddenly had 20 caches instead of 15....guess what "someone" merged a PR.

Just joking, I think I have solved most of the caching issues, except for pre-commit.

[alexrudd2]

Just joking, I think I have solved most of the caching issues, except for pre-commit.

I propose dropping pre-commit from the CI and running the tools manually. The versions can be kept in sync between pyproject.toml and pre-commit by renovatebot. See here for a PR where it groups them.

[janiversen]

Just joking, I think I have solved most of the caching issues, except for pre-commit.

I propose dropping pre-commit from the CI and running the tools manually. The versions can be kept in sync between pyproject.toml and pre-commit by renovatebot. See here for a PR where it groups them.

I will give it try, but it probably will be end of next week....I am off in a couple of hours, and will only be online a 1-2 hours each day until next Friday.

[janiversen]

@alexrudd2 just thinking loud!

pre-commit really does not bring a lot to this project, another option is to:

• save the precommit hook in e.g githooks
• change documentation to say "cp githooks/precommit ..." instead of "pre-commit install"

And then suddenly we have all dependencies back in pyproject.toml.

What is the disadvantage I have overlooked ?

I mean for this project, with its "huge" developer population.

[alexrudd2]

What is the disadvantage I have overlooked ?

pre-commit has minor value to me because it's easy to transfer between projects and install/uninstall. I tried using bare git hooks before and they were a bit fussy, but I'm OK using them if you can get them to work.

[github-advanced-security]

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.