Fix test_naming.py so it doesn't let through directories that need be ignored

[valeriupredoi]

Description

• Closes Tests with Python=3.8 and 3.9 fail #3081

For some reason pytest is looking into the .git dir but only for Pythons 3.8 and 3.9 - this is new behaviour, see failed test which is reproducible. Am very confused as to why this is happening!

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🛠 This pull request has a descriptive title
• 🛠 Code is written according to the code quality guidelines
• 🛠 Tests run successfully
• 🛠 All checks below this pull request were successful

[remi-kazeroni]

Thanks @valeriupredoi! It would be good to have this in soon so that we have ✅ tests to merge other PRs into main 👍

[valeriupredoi]

cheers @remi-kazeroni - @schlunma or @bouweandela one of you guys could have a look and merge please 🍺

[bouweandela]

For some reason pytest is looking into the .git dir but only for Pythons 3.8 and 3.9 - this is new behaviour, see failed test which is reproducible. Am very confused as to why this is happening!

How about opening an issue with pytest and asking about it?

[schlunma]

I don't think this fixes the actual problem. If I read the doc of the ignore option correctly,

--ignore=path         ignore path during collection (multi-allowed).
--ignore-glob=path    ignore path pattern during collection (multi-allowed).

then this just ignores the .git directory when looking for tests, but does not ignore it when running the test.

I guess we have to change the actual test

ESMValTool/tests/unit/test_naming.py

Lines 49 to 65 in e22261d

	def test_avoid_casing_collisions(self):
	"""
	Check that there are no names differing only in the capitalization
	This includes folders differing from files
	"""
	for dirpath, dirnames, filenames in os.walk(self.esmvaltool_folder):
	for dirname in dirnames:
	if dirname in IGNORE:
	dirnames.remove(dirname)
	self.assertEqual(
	len(filenames) + len(dirnames),
	len({name.lower()
	for name in filenames + dirnames}),
	'Colliding names found at {0}. Please do not '
	'use names that only differ in '
	'capitalization'.format(dirpath))

to ignore the entire .git directory and all it's subdirs, and not only the .git itself (which is currently the case).

[valeriupredoi]

@bouweandela that is indeed a very good point! I'll do that now - but in the meantime, let's get this in so we have our tests pass, we can always revert once pytest find a solution/release a new version 🍺

[valeriupredoi]

@schlunma I don't think there is a need to change the test - this fix does the trick since I ran the GA tests

[schlunma]

I don't think this an issue of pytest...this is our test failing because there are two objects in the .git dir with the same name but different capitalization. The fact that this is happening only for Python 3.8 and 3.9 in that particular case is probably random. I am not sure how git creates subdirs and files in .git, but there is probably some random element.

[schlunma]

As you can see, now the 3.10 test is also failing: https://github.com/ESMValGroup/ESMValTool/actions/runs/4382991244/jobs/7672693535

[valeriupredoi]

it isn't random, but it is reproducible (ran a few instances since I too thought that was a fluke) - why? I don't know TBF

[valeriupredoi]

As you can see, now the 3.10 test is also failing: https://github.com/ESMValGroup/ESMValTool/actions/runs/4382991244/jobs/7672693535

oh that's a spanner in the works - OK so it's not Py version-related then 👍 Let me have a closer look at the test then

[valeriupredoi]

here's a fun one: Py3.8 passing here, when most of the times it fails - so it's not 100% reproducible either https://github.com/ESMValGroup/ESMValTool/actions/runs/4369963301/jobs/7644357766

[schlunma]

Yeah, I guess git is somehow (more or less random) producing files/dirs that cause our tests to fail. I think ignoring the entire .git dir with all it's subdirectories in our test should be the way to go 👍

[valeriupredoi]

@schlunma you were spot-on, mate! The problem was with our test, and not pytest at all - we had a fairly poor implementation and ignored dirs were slipping through, in my case locally .github kept passing through - I fixed it good now, Maginot Line 😆

[valeriupredoi]

@schlunma @bouweandela @remi-kazeroni proves out this test was really not working properly as it was before, two-fold:

• it was looking into top-level dirs that should be ignored
• it was using dir content from dirs that were ignored at top-level, but content was still walked on and used

It is now fixed and does what it says on the tin, please re-review when you have a bit of time. Cheers to Manu for pulling the alarm lever! 🍺

[schlunma]

Thanks @valeriupredoi, this looks much better! A couple of minor comments.

[schlunma]

Great, thanks V!! 🍻

[remi-kazeroni]

Thanks a lot @valeriupredoi and @schlunma! Let's get this in to get our tests fixed and resume merging other PRs 🍺

[valeriupredoi]

awesome, cheers, guys!