enhance `CMakeMake` easyblock to check whether correct Python installation was picked up by `cmake` #3233

MartinsNadia · 2024-02-27T11:36:12Z

Sanity check to verify if Cmake apply the new policy CMP0094

removing/adding blank spaces

ocaisa · 2024-02-27T13:32:57Z

easybuild/easyblocks/generic/cmakemake.py

+        ebrootpython_path = os.getenv('EBROOTPYTHON')
+        if all(path and ebrootpython_path in path for path in [pythonExecPath, pythonIncludePath, pythonLibraryPath]):
+            self.log.info("Python related paths configured correctly.")
+            return out


Move the return statement out of the if/else block, it is hidden here.

ocaisa · 2024-02-27T13:34:54Z

easybuild/easyblocks/generic/cmakemake.py

+        self.log.info("Python include path: %s", pythonIncludePath)
+        self.log.info("Python library path: %s", pythonLibraryPath)
+        # Check if paths include EBROOTPYTHON
+        ebrootpython_path = os.getenv('EBROOTPYTHON')


This assumes that EBROOTPYTHON is set, what happens if it isn't?

I imagine that what you want to do is complain if pythonExecPath (and friends) are set but no Python dependency is found

ocaisa · 2024-02-27T13:37:41Z

easybuild/easyblocks/generic/cmakemake.py

+        self.log.info("Python executable path: %s", pythonExecPath)
+        self.log.info("Python include path: %s", pythonIncludePath)
+        self.log.info("Python library path: %s", pythonLibraryPath)


I would only print these if they have a value (so move them up into your if/elif loop)

Not only that: You could have them multiple times when you consider Python_EXECUTABLE and Python3_EXECUTABLE or the variables may be completely unset if the CMake file didn't use Python at all

Maybe create thos variables as empty lists?

ocaisa · 2024-02-27T13:42:29Z

easybuild/easyblocks/generic/cmakemake.py

+
+        # Search for Python paths in each line
+        for line in lines:
+            if line.startswith('Python3_EXECUTABLE:FILEPATH='):


It looks like these are only guaranteed for CMake 3.16+ so should probably make this check conditional on that version

They also don't cover the FindPython case: https://cmake.org/cmake/help/latest/module/FindPython.html#artifacts-specification

Yes maybe use a regex? Python[2-3]?_EXECUTABLE?

ocaisa · 2024-02-27T14:05:10Z

easybuild/easyblocks/generic/cmakemake.py

+        self.log.info("Python include path: %s", pythonIncludePath)
+        self.log.info("Python library path: %s", pythonLibraryPath)
+        # Check if paths include EBROOTPYTHON
+        ebrootpython_path = os.getenv('EBROOTPYTHON')


Suggested change

ebrootpython_path = os.getenv('EBROOTPYTHON')

ebrootpython_path = get_software_root("Python")

if pythonExecPath and not ebrootpython_path:

raise EasyBuildError("Python related path (%s) found but no Python dependency included, check log" % pythonExecPath)

boegel · 2024-02-28T07:29:55Z

easybuild/easyblocks/generic/cmakemake.py

+        # Search for Python paths in each line
+        for line in lines:
+            if line.startswith('Python3_EXECUTABLE:FILEPATH='):
+                pythonExecPath = line.split('=')[1].strip()


Small nitpicking suggestion: let's not start using CamelCase for variable names (we only use that for class names), so please use python_exec_path, python_include_path, python_library_path

Flamefire

This might be a bit more tricky than it seems...

Flamefire · 2024-03-15T14:04:13Z

easybuild/easyblocks/generic/cmakemake.py

-
+        # sanitycheck for the configuration step
+        self.log.info("Checking python paths")
+        with open('CMakeCache.txt', 'r') as file:


This should handle the case that this file may not exist. E.g. when configure_cmd != DEFAULT_CONFIGURE_CMD anything can happen.

Flamefire · 2024-03-15T14:06:41Z

easybuild/easyblocks/generic/cmakemake.py

+        # sanitycheck for the configuration step
+        self.log.info("Checking python paths")
+        with open('CMakeCache.txt', 'r') as file:
+            lines = file.readlines()


Maybe use readfile(...).split('\n')?

Flamefire · 2024-03-15T14:08:13Z

easybuild/easyblocks/generic/cmakemake.py

+
+        # Search for Python paths in each line
+        for line in lines:
+            if line.startswith('Python3_EXECUTABLE:FILEPATH='):


Yes maybe use a regex? Python[2-3]?_EXECUTABLE?

Flamefire · 2024-03-15T14:09:44Z

easybuild/easyblocks/generic/cmakemake.py

+        self.log.info("Python executable path: %s", pythonExecPath)
+        self.log.info("Python include path: %s", pythonIncludePath)
+        self.log.info("Python library path: %s", pythonLibraryPath)


Not only that: You could have them multiple times when you consider Python_EXECUTABLE and Python3_EXECUTABLE or the variables may be completely unset if the CMake file didn't use Python at all

Maybe create thos variables as empty lists?

Flamefire · 2024-03-15T14:11:06Z

easybuild/easyblocks/generic/cmakemake.py

+        self.log.info("Python library path: %s", pythonLibraryPath)
+        # Check if paths include EBROOTPYTHON
+        ebrootpython_path = os.getenv('EBROOTPYTHON')
+        if all(path and ebrootpython_path in path for path in [pythonExecPath, pythonIncludePath, pythonLibraryPath]):


See above: pythonExecPath & co might not have been set at all. And maybe path.startswith(ebrootpython_path)? We possibly need to take symlinks into account too: either of the 2 might be a resolved symlink or not.

Flamefire · 2024-03-22T14:53:40Z

easybuild/easyblocks/generic/cmakemake.py

+        except FileNotFoundError:
+            self.log.warning("CMakeCache.txt not founf. Python paths checks skipped.")
+            return 
+        # EBROOTPYTHON check 


And please remove this comment. It doesn't add any information to the code below

Flamefire

Please recheck also the use(s) of self.log.error. When raising an EasyBuildError you don't need to log the same because that will already be done.
I also thing that you don't need self.log.error but only EasyBuildError, i.e. fail if there is an error.

Another enhancement would be to gather all issues (e.g. each wrong path) in a list and then fail if that list is non-empty. This way ALL issues would be reported, not only the first found which might help in fixing this.

Flamefire · 2024-03-25T10:17:26Z

easybuild/easyblocks/generic/cmakemake.py

+        python_prefixes = r"(Python|Python2|Python3)_"
+
+        for line in lines:
+            if line.startswith(python_prefixes + r"EXECUTABLE:FILEPATH="):


This doesn't work. python_prefixes is all lowercase and you likely intended to use something like re.match. Got to look up the docs but from the top of my head: python_prefixes = "PYTHON[23]?_" above and re.match(python_prefixes + "EXECUTABLE(:FILEPATH)?=", line) here would be needed

Flamefire · 2024-03-25T10:18:13Z

easybuild/easyblocks/generic/cmakemake.py

+                python_library_path.append(line.split('=')[1].strip())
+                self.log.info("Python library path: %s", python_library_path[-1])
+
+            # Check if paths are found and validate paths for symlinks


Everything from here should be outside the loop shouldn't it?

Flamefire · 2024-03-25T10:19:30Z

easybuild/easyblocks/generic/cmakemake.py

+
+            # Check if paths are found and handle EBROOTPYTHON cases
+            if any(path for path in [python_exec_path, python_include_path, python_library_path]):
+                if not os.getenv('EBROOTPYTHON'):


This looks like a duplication to the below (ebrootpython_path etc). Maybe at least put them together? Are both even required? I.e. isn't ebrootpython_path enough/better than EBROOTPYTHON?

Flamefire · 2024-03-25T11:13:37Z

easybuild/easyblocks/generic/cmakemake.py

+                    )
+                elif not all(
+                    (path and os.getenv('EBROOTPYTHON') in path)
+                    for path in [python_exec_path, python_include_path, python_library_path]


You are checking only for $EBROOTPYTHON being in a list of paths, which likely is always false. Maybe rename the variables to be able to spot that: python_exec_path -> python_exec_paths and so on.

It might even make sense to show which path is "wrong". I.e. which of the 3 lists and which exact path is outside EB.

MartinsNadia and others added 4 commits February 19, 2024 16:44

Sanity check for python configuration

7e3b26a

Sanity check for Python

40986c8

Remove legacy code, integrate python sanity check into configure section

133e3b8

Update cmakemake.py

775dde0

removing/adding blank spaces

ocaisa reviewed Feb 27, 2024

View reviewed changes

boegel requested changes Feb 28, 2024

View reviewed changes

boegel changed the title ~~Sanity Check for Python Configuration~~ enhance CMakeMake easyblock to check whether correct Python installation was picked up by cmake Feb 28, 2024

Flamefire suggested changes Mar 15, 2024

View reviewed changes

MartinsNadia added 3 commits March 18, 2024 15:30

teste

067c4cd

Implementation of path validation for sanitycheck

16b6465

Fix identation

cfbe5e5

Flamefire reviewed Mar 22, 2024

View reviewed changes

MartinsNadia added 2 commits March 22, 2024 15:42

Formatting improvements

3d82787

Fixing whitespace

c0d5e86

Flamefire suggested changes Mar 25, 2024

View reviewed changes

MartinsNadia added 2 commits April 30, 2024 10:27

Revised version

f260f73

Revised version - whitespace

b60ecd6

Flamefire mentioned this pull request Apr 30, 2024

Improve Python check in cmakemake.py MartinsNadia/easybuild-easyblocks#2

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance `CMakeMake` easyblock to check whether correct Python installation was picked up by `cmake` #3233

enhance `CMakeMake` easyblock to check whether correct Python installation was picked up by `cmake` #3233

MartinsNadia commented Feb 27, 2024

ocaisa Feb 27, 2024

ocaisa Feb 27, 2024

ocaisa Feb 27, 2024

ocaisa Feb 27, 2024 •

edited

Flamefire Mar 15, 2024

ocaisa Feb 27, 2024

ocaisa Feb 27, 2024

Flamefire Mar 15, 2024

ocaisa Feb 27, 2024 •

edited

boegel Feb 28, 2024

Flamefire left a comment

Flamefire Mar 15, 2024

Flamefire Mar 15, 2024

Flamefire Mar 15, 2024

Flamefire Mar 15, 2024

Flamefire Mar 15, 2024

Flamefire Mar 22, 2024

Flamefire left a comment

Flamefire Mar 25, 2024

Flamefire Mar 25, 2024

Flamefire Mar 25, 2024

Flamefire Mar 25, 2024

-        ebrootpython_path = os.getenv('EBROOTPYTHON')
+        ebrootpython_path = get_software_root("Python")
+        if pythonExecPath and not ebrootpython_path:
+            raise EasyBuildError("Python related path (%s) found but no Python dependency included, check log" % pythonExecPath)

enhance CMakeMake easyblock to check whether correct Python installation was picked up by cmake #3233

Are you sure you want to change the base?

enhance CMakeMake easyblock to check whether correct Python installation was picked up by cmake #3233

Conversation

MartinsNadia commented Feb 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ocaisa Feb 27, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ocaisa Feb 27, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Flamefire left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Flamefire left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enhance `CMakeMake` easyblock to check whether correct Python installation was picked up by `cmake` #3233

enhance `CMakeMake` easyblock to check whether correct Python installation was picked up by `cmake` #3233

ocaisa Feb 27, 2024 •

edited

ocaisa Feb 27, 2024 •

edited