Oxpy & oat not working on M1 Mac

[simonecodispoti]

Hi all,
I'm still having issues with oxpy and oat, issues similar to those investigated here:

sulcgroup/oxdna_analysis_tools#22

This time I'm working on macOS Monterey 12.5, with M1 architecture. I was able to compile and test successfully oxDNA but make test_oxpy was not passed and if I try to call oat config this raises a brutal segmentation fault:

INFO: oxDNA_analysis_tools version: 2.0.3
INFO: running config.py installed at:  /Users/simo/opt/anaconda3/envs/oxdna/lib/python3.9/site-packages/oxDNA_analysis_tools/config.py
INFO: Python version: 3.9.13
INFO: Package Numpy found. Version: 1.21.5
INFO: Package MatPlotLib found. Version: 3.5.2
INFO: Package BioPython found. Version: 1.78
INFO: Package SciKit-Learn found. Version: 1.1.1
zsh: segmentation fault  oat config

As suggested I created a conda virtual enviroment with python3.9 and the requested libraries and then compiled with:

cmake .. -DPython=ON -DPYTHON_INCLUDE_DIRS=/Users/simo/opt/anaconda3/envs/oxdna/include/python3.9 -DPYTHON_EXECUTABLE=/Users/simo/opt/anaconda3/envs/oxdna/bin/python

I also tried without a virtual env (I use anaconda as package manager, the one developed for M1 arm architectures) and without specifying the path using only -DOxpySystemInstall=On, but this doesn't change anything.

Thank you in advance,
Simone

P.S. The cmake log output on the console:

-- Performing Test COMPILER_SUPPORTS_MARCH_NATIVE
-- Performing Test COMPILER_SUPPORTS_MARCH_NATIVE - Failed
-- The '-march=native' flag is not supported, compiling without it
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- Adding the targets found in the /Users/simo/oxDNA/contrib/randisi folder
-- Adding the targets found in the /Users/simo/oxDNA/contrib/romano folder
-- Adding the targets found in the /Users/simo/oxDNA/contrib/rovigatti folder
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) 
-- Adding the targets found in the /Users/simo/oxDNA/contrib/tostiguerra folder
-- pybind11 v2.2.4
-- The detected Python's include dirs are: /Users/simo/opt/anaconda3/envs/oxdna/include/python3.9
-- Enabling Python bindings
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/simo/oxDNA/build

[lorenzo-rovigatti]

Thanks for the report! What happens if you run the oxpy example by hand?

$ cd oxDNA_folder/examples/OXPY
$ python3 run.py

[simonecodispoti]

The same error:

zsh: segmentation fault python3 run.py

On the previous Linux-Ubuntu OS I have never experienced such errors.

[lorenzo-rovigatti]

Thanks! This means that the issue is definitely caused by oxpy. Unfortunately I don't have access to an M1-powered mac computer, but I'll ask around and see if I can borrow one.

[simonecodispoti]

Thanks in advance! I was pretty confident it was oxpy, because oxDNA works well.

[simonecodispoti]

Just a quick supplement: I think that all the problems originate from the first lines of the cmake log:

-- Performing Test COMPILER_SUPPORTS_MARCH_NATIVE
-- Performing Test COMPILER_SUPPORTS_MARCH_NATIVE - Failed
-- The '-march=native' flag is not supported, compiling without it

It seems that MARCH is not supporter by oxpy? No other users have ever tested ox tools on a M1 chip pc?

[lorenzo-rovigatti]

This shouldn't be an issue: cmake checks whether the flag is supported and if it's not (as in this case) then it doesn't use it.

[RodenLuo]

Hi, This might be a (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64') error. To confirm this, could you please run the following command in your terminal and paste the output?

In your oxdna env

file $(which python3)

My guess is you will get something similar to Mach-O 64-bit executable x86_64

Then, in your system python, usually it is /usr/bin/python3, so just type

file $(which /usr/bin/python3)

My guess is you will get something similar to Mach-O 64-bit executable arm64e in the output.

If this is the case, you can either create an virtual env with your system's python, or just use it (not that recommended). Or, better reinstall your conda, and pay EXTRA attention to install the arm64 version. After that, repeat the whole installation process from scratch, meaning, create new virtual env with the newly installed conda and see.

[simonecodispoti]

Thanks for the response Roden! These are the outputs for the two commands:

1. file $(which python3) in my oxdna env ---> /Users/simo/opt/anaconda3/envs/oxdna/bin/python3: Mach-O 64-bit executable arm64
2. file $(which /usr/bin/python3) in my 'base' env --->

/usr/bin/python3: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64
- Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e
- Mach-O 64-bit executable arm64e]
/usr/bin/python3 (for architecture x86_64):	Mach-O 64-bit executable x86_64
/usr/bin/python3 (for architecture arm64e):	Mach-O 64-bit executable arm64e

[RodenLuo]

Then, this is not the case. Sorry...

[RodenLuo]

I experimented a bit with an M1 from my colleague and realized that I cannot enable Debug mode and oxpy together...

$ /Applications/CMake.app/Contents/bin/cmake .. -DDebug=ON -DPython=ON -DPYTHON_INCLUDE_DIRS=/Users/yangx0i/miniconda3_arm64/envs/oxpy_arm/include/python3.10 -DPYTHON_EXECUTABLE=/Users/yangx0i/miniconda3_arm64/envs/oxpy_arm/bin/python

CMake Error at oxpy/CMakeLists.txt:23 (SET_TARGET_PROPERTIES):
  SET_TARGET_PROPERTIES Can not find target to add properties to: common

If I remove -DDebug=ON, it's fine. Without the debugging mode enabled, using CodeLLDB extension in VS Code, and the following launch.json setting:

        {
            "type": "lldb",
            "request": "launch",
            "name": "LLDB Python",
            "program": "/Users/yangx0i/miniconda3_arm64/envs/oxpy_arm/bin/python",
            "args": [
                "${file}"
            ],
            "cwd": "${workspaceFolder}",
            "stopOnEntry": true,
            "env": {
                "PYTHONPATH": "${workspaceFolder}/build"
            },
        },

I reached to this screenshot:

The beginning part of the call stack:

All assembly... I know it failed immediately at Line4 of run.py (with oxpy.Context():)

I think one way forward is to figure out how to have the debug info enabled for oxpy, which I'm really not sure..

[lorenzo-rovigatti]

Roden, thanks for trying. You can use the -DG=1 cmake switch to compile with optimisation&debug flags enabled.

[simonecodispoti]

Thank you both! While trying to fix this just a silly request/idea: because I usually run simulations "by hand" or by using shell scripts and I'm not planning to use oxpy, is there any stable version of oat not relying on ox? I have managed to run some oat scripts (some don't make use of oxpy) and they Indeed seem to work.

[lorenzo-rovigatti]

That's a question for @ErikPoppleton !

[ErikPoppleton]

Hi Simone,

The last version of oat to not use oxpy for energy calculations was 1.0.16, which can be downloaded from PyPi. However, subprocessing out the process was substantially slower, so if we can get oxpy working on M1, that would be much better (depending on the system, the newer oxpy version can be up to 100x faster than the older DNAnalysis version.

[ErikPoppleton]

One other thing to try, if you to go the oxDNA source directory and try installing oat with

python -m pip install ./analysis --no-build-isolation

Does that fix the segmentation fault? oat config doesn't try to call oxpy, so that suggests that it's a problem with the Cython file reader, possibly due to numpy array header missmatch. Also, what is your version of numpy?

[simonecodispoti]

Hi Erik, Thanks for the alternative! Yes it would be nice to try to fix oxpy : (

Following your hint inside the conda env doesn't fix the problem and oat config still rises a segmentation fault.

[RodenLuo]

Some work-in-progress.

• I noticed a similar segmentation fault (probably the same) on my intel Mac (MacBook Pro (16-inch, 2019), 2.4 GHz 8-Core Intel Core i9). The following are all based on this machine.
• I still could not get the source code up in the debugger. The methods I tried are mentioned in the end.
• It crashed at Line 2 of run.py, import oxpy, instead of my previously mentioned Line 4 with oxpy.Context():.
• Removing Line 1 import numpy as np does not by-pass the error. My numpy is 1.23.2
• Using a demo project, I am able to step into the source code...
• The error might be from the block starting from Line30 PYBIND11_MODULE(core, m) in oxpy/bindings.cpp.

I tried all the following hoping to get into the oxpy source code:

1. Add -DG=1. I can step into the oxDNA cpp source code, but not oxpy.
2. Some permutations of adding one or all of:-DG=1 in the cmake commands; adding set(CMAKE_BUILD_TYPE Debug) to oxpy/CMakeLists.txt, adding target_compile_options(core PRIVATE -g) and target_compile_options(_oxpy_lib PRIVATE -g) to oxpy/CMakeLists.txt
3. set os.environ["CPPFLAGS"] = "-g" and pip install --no-clean ./ (I did this slightly different, see below) as mentioned here. I did not try the extra_compile_args=["-g"] approach there. Because it seems to be .cpp file specific and I was not sure how many cpp files are there, and how should I pair them up with seemingly a name field before the .cpp file.

   if(OxpySystemInstall)
   		INSTALL(CODE "execute_process(COMMAND ${PYTHON_EXECUTABLE} -m pip install --no-clean 
   ${OXPY_OUTPUT_DIR})")
   	else()
   		INSTALL(CODE "execute_process(COMMAND ${PYTHON_EXECUTABLE} -m pip install --no-clean --user 
   ${OXPY_OUTPUT_DIR})")
   	endif()
   

4. Followed this tutorial (this is my first encounter with pybind, and the first few with looking inside cmake...). I added only set(CMAKE_BUILD_TYPE Debug). And the source code is up for the demo project in the video. I tried to compare the call stack difference at the assembly level, but the difference starts at the middle of the stack, instead of near the end even though I put the breakpoint (for the debuggable one) at the earliest I can.
5. One difference between the demo and oxpy is pybind11 folder. I put a brand new folder in oxpy but still no luck.
6. Comparing near the end of the call stack, screenshots below, I think the error can be located in the block starting from Line30 PYBIND11_MODULE(core, m) in oxpy/bindings.cpp. Some differences are export_ syntax (unique to oxpy), py::class_ (not exist in oxpy), def_submodule (unique to oxpy).

I'm not really sure where to go from here to get more efficient debugging. The worst case might be a few std::cout line-by-line...

[RodenLuo]

Finally, I met this issue in pybind11 (on my Intel Mac with the newest pybind11).

Ultimately, in the newest version of pybind11, the following code section cannot run through.

https://github.com/pybind/pybind11/blob/8756f16ed842e40406018df901f3219b231e2105/include/pybind11/detail/internals.h#L411-L417

    // Ensure that the GIL is held since we will need to make Python calls.
    // Cannot use py::gil_scoped_acquire here since that constructor calls get_internals.
    struct gil_scoped_acquire_local {
        gil_scoped_acquire_local() : state(PyGILState_Ensure()) {}
        ~gil_scoped_acquire_local() { PyGILState_Release(state); }
        const PyGILState_STATE state;
    } gil;

The comment says "Ensure that the GIL is held". It means somehow in oxpy, the GIL is not held.

[RodenLuo]

For more about the debugging process, please see pybind11 issue #3801.

@simonecodispoti Could you please make the following change, re-cmake (remove the build folder first), and then recompile and see?

oxDNA/oxpy/CMakeLists.txt

Line 26 in ec1be2b

TARGET_LINK_LIBRARIES(_oxpy_lib ${PYTHON_LIBRARIES} common)

↓

TARGET_LINK_LIBRARIES(_oxpy_lib common) 

[lorenzo-rovigatti]

Hey Roden, my last commit (e003f6c) makes it possible to compile oxpy with -DDebug=True. I hope this help, and thanks for all the effort you are putting into this. I hope I'll be able to try to debug it myself as soon as possible.

[RodenLuo]

So, the change I mentioned previously (remove ${PYTHON_LIBRARIES} from linking) made the run of run.py successful on my intel Mac (not yet able to test on M1, probably tomorrow). Not sure why there was no linking error after I deleted the linking. Is ${PYTHON_LIBRARIES} used by _oxpy_lib? Per my naive understanding here is that, when pybind11 is involved, all the communication between c++ and python should be (or preferred to be) handled by pybind11 and that's why there is this PyMUTEX_LOCK. But then, I don't understand why this is not causing problems on Linux.

As for the debugging process, somehow before I sleep yesterday, after reopening vscode or something I realized that I could step into bindings.cpp. That's why I was able to locate the problem in my intel case. I did a reflection after work today and found that -DG=1 flag itself is enough to step into bindings.cpp (given that a breakpoint is set there). But, I need to use the newest pybind11 from its GitHub repo. The same for my ubuntu, no source code info if not using the newest pybind11.

Pybind11 itself is a header-only library, then, my guess is there might be something missing in the CMake files in the oxDNA's pybind11. But I haven't checked it yet.

[RodenLuo]

I was using codelldb in VS Code. Because gdb is not available on M1. I have not tested gdb on Ubuntu though.

[simonecodispoti]

Thanks a lot @RodenLuo ! You are really putting a lot of effort into this and I'm really glad of that.
Unfortunately I'm not an expert in debugging codes, I'm just following your instructions!

@simonecodispoti Could you please make the following change, re-cmake (remove the build folder first), and then recompile and see?

oxDNA/oxpy/CMakeLists.txt

Line 26 in ec1be2b

TARGET_LINK_LIBRARIES(_oxpy_lib ${PYTHON_LIBRARIES} common)

↓

TARGET_LINK_LIBRARIES(_oxpy_lib common) 

Unfortunately this only fixes oat config segmentation fault and doesn't work with the other tools (the same segmentation fault appears e.g. calling oat mean for a given trajectory).

I will also try the old version of oxdna tools if necessary.

[RodenLuo]

It's working on the M1 I'm testing now. The machine is MacBook Pro (16-inch, 2021), with Apple M1 Pro chip. The whole log is below, let me know if any step is not clear.

Click me to see the whole log

# ======= version checking
$ file /Users/yangx0i/miniconda3_arm64/envs/oxpy_arm/bin/python
/Users/yangx0i/miniconda3_arm64/envs/oxpy_arm/bin/python: Mach-O 64-bit executable arm64

$ /Users/yangx0i/miniconda3_arm64/envs/oxpy_arm/bin/python -V
Python 3.10.4

$ /Applications/CMake.app/Contents/bin/cmake -version
cmake version 3.24.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).


# ======= download and edit
$ git clone https://github.com/lorenzo-rovigatti/oxDNA.git
$ vi oxDNA/oxpy/CMakeLists.txt # remove the linking as mentioned above


# ======= install
$ /Applications/CMake.app/Contents/bin/cmake .. -DDebug=True -DPython=ON -DPYTHON_INCLUDE_DIRS=/Users/yangx0i/miniconda3_arm64/envs/oxpy_arm/include/python3.10 -DPYTHON_EXECUTABLE=/Users/yangx0i/miniconda3_arm64/envs/oxpy_arm/bin/python

CMake Deprecation Warning at CMakeLists.txt:8 (cmake_policy):
  The OLD behavior for policy CMP0037 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.


-- The C compiler identification is AppleClang 13.0.0.13000027
-- The CXX compiler identification is AppleClang 13.0.0.13000027
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Compiling in Debug mode
-- Found Git: /opt/homebrew/bin/git (found version "2.37.3") 
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- Adding the targets found in the /Users/yangx0i/roden/oxDNA/contrib/randisi folder
-- Adding the targets found in the /Users/yangx0i/roden/oxDNA/contrib/romano folder
-- Adding the targets found in the /Users/yangx0i/roden/oxDNA/contrib/rovigatti folder
-- Found PkgConfig: /opt/homebrew/bin/pkg-config (found version "0.29.2") 
-- Checking for module 'gsl'
--   No package 'gsl' found
-- Adding the targets found in the /Users/yangx0i/roden/oxDNA/contrib/tostiguerra folder
-- Found PythonInterp: /Users/yangx0i/miniconda3_arm64/envs/oxpy_arm/bin/python (found suitable version "3.10.4", minimum required is "3") 
-- Found PythonLibs: /Users/yangx0i/miniconda3_arm64/envs/oxpy_arm/lib/libpython3.10.dylib
-- pybind11 v2.2.4
-- The detected Python's include dirs are: /Users/yangx0i/miniconda3_arm64/envs/oxpy_arm/include/python3.10
-- Enabling Python bindings
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- LTO enabled
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/yangx0i/roden/oxDNA/build

$ make -j98
$ make install


# ======= testing
$ ~/.local/bin/oat config
INFO: oxDNA_analysis_tools version: 2.0.3
INFO: running config.py installed at:  /Users/yangx0i/.local/lib/python3.10/site-packages/oxDNA_analysis_tools/config.py
INFO: Python version: 3.10.4
INFO: Package Numpy found. Version: 1.23.3
INFO: Package MatPlotLib found. Version: 3.5.3
INFO: Package BioPython found. Version: 1.79
INFO: Package SciKit-Learn found. Version: 1.1.2
INFO: Package oxpy found. Version: 3.4.3.dev0+g6794178.d20220912
INFO: No dependency issues found.

INFO: Analyses will be computed in chunks of 20 configurations at a time
INFO: You can modify this number by running oat config -n <number>, which will be persistent between analyses.



$ ~/.local/bin/oat mean -h
usage: mean.py [-h] [-p num_cpus] [-o output_file] [-d deviation_file]
               [-i index_file] [-a alignment_configuration]
               trajectory

Computes the mean structure of a trajectory file

positional arguments:
  trajectory            The trajectory file you wish to analyze

options:
  -h, --help            show this help message and exit
  -p num_cpus           (optional) How many cores to use
  -o output_file, --output output_file
                        The filename to save the mean structure to
  -d deviation_file, --deviations deviation_file
                        Immediatley run oat deviations from the output
  -i index_file         Compute mean structure of a subset of particles from a
                        space-separated list in the provided file
  -a alignment_configuration, --align alignment_configuration
                        The id of the configuration to align to, otherwise
                        random


$ ~/.local/bin/oat mean -o mean_all.dat hairpin.conf 
INFO: oxDNA_analysis_tools version: 2.0.3
INFO: running config.py installed at:  /Users/yangx0i/.local/lib/python3.10/site-packages/oxDNA_analysis_tools/config.py
INFO: Python version: 3.10.4
INFO: Package Numpy found. Version: 1.23.3
INFO: No dependency issues found.
INFO: Processing in blocks of 20 configurations
INFO: You can modify this number by running oat config -n <number>, which will be persistent between analyses.
Starting up 1 processes for 1 chunks
All spawned, waiting for results
--- 0.6535718441009521 seconds ---

@lorenzo-rovigatti Hi Lorenzo, Using -DDebug=True is effective, but still only if I replace oxpy/pybind11 with the latest https://github.com/pybind/pybind11, can I step into oxpy/bindings.cpp.

[simonecodispoti]

Thanks @RodenLuo ! I will try with a clean installation and following your log. Do I have to install pybind11'?

[RodenLuo]

You are very welcome! You don't have to do any extra stuff for pybind11 (unless you want to debug oxpy). Just remove ${PYTHON_LIBRARIES} from Line 26 of oxDNA/oxpy/CMakeLists.txt is enough.

[simonecodispoti]

I Tried again and now one of the 2 tests of make test_oxpy fails. nevertheless, the tools seem to work on small trajectories :) but still rise segmentation faults on larger ones :( @ErikPoppleton the how the mean tool should perform with 10000 configurations of a 20 nts strand?

[lorenzo-rovigatti]

Thank you very much Roden. I still haven't found the time to get my hands dirty with this, but I have committed the change you suggested (removing the linking of python libs).

[simonecodispoti]

I Tried again and now one of the 2 tests of make test_oxpy fails. nevertheless, the tools seem to work on small trajectories :) but still rise segmentation faults on larger ones :( @ErikPoppleton the how the mean tool should perform with 10000 configurations of a 20 nts strand?

Actually I tried with another trajectory and is running, although very slowly. I think I found the issue: the trajectories I was using were without momenta (print_momenta = False in the input file). Does mean need the whole trajectory? It shouldn't because mean configurations don't need velocities and angular momenta (@ErikPoppleton)

[RodenLuo]

Hi Lorenzo, Thanks. I just got interested in stepping into the code to learn more as only recently do I start to understand the code base, even though very superficially in many places. Feel free to handle it at your convenience.

Hi Simone, I tested on two slightly larger trajectories and they could run through (but with some differences in the resulting files compared to the run on my ubuntu). The structure has 16128 NTs. Both trajectories have 100 recorded confs (100 t = ). The first is with non-zero velocity values (422M). The second is with 0 for the last 6 columns (290M). Basically, the velocities are all zero, but the 0 is there as a placeholder. The resulting mean conf shows differences in the coordinate values compared to my ubuntu mean conf. But when looking in oxview, the difference is not noticeable.

I also have one make test_oxpy failed. My case is caused by under oxDNA/test/OXPY, after running the test, the following files are different:

> diff last_nucleotide.dat last_nucleotide_correct.dat
2,10c2,10
< 8.914564 -0.061766 1.133814
< 8.856179 0.137832 1.160710
< 8.765789 0.300055 1.090000
< 8.824286 0.355273 1.063431
< 8.824516 0.386833 1.077637
< 8.757490 0.452182 1.008346
< 8.983339 0.367106 0.999174
< 8.984770 0.363439 0.864591
< 8.886357 0.530297 0.921013
---
> 8.915883 -0.061127 1.132751
> 8.858563 0.138343 1.159861
> 8.816379 0.295659 1.044456
> 8.881306 0.325018 1.010187
> 8.910014 0.357738 1.016579
> 8.942146 0.304912 0.892514
> 9.069714 0.302830 0.841750
> 9.154019 0.356858 0.754155
> 9.164492 0.493286 0.788510

Basically, in that folder, the failed test requires the run of input, with initial conf as init.dat, and seed as 123456 to be consistent with the given last_nucleotide_correct.dat. In my case above, the last nucleotide coordinates are close but not exactly the same. I guess it might be because the random function or the random seed does not behave exactly the same on M1 and Ubuntu. The test for oxDNA part only set constraints on some column average of energy file. So, oxDNA-related tests all passed even though the generated trajectories are different. This failed test should be best answered by the original developers.

[zoombya]

@simonecodispoti I've reproduced your error with trajectory_print_momenta = False
the issue is most likely that we are parsing the entire conf expecting momenta,
as we where not aware this option existed.

[ErikPoppleton]

@zoombya It's slightly more subtle than that. The trajectory parser is breaking because it's looking for a space character after the a3 vector to split it from the momenta, but when trajectory_print_momenta = False, the a3 vector is immediately followed by a \n. This is because the trajectory_print_momenta option is newer than the oat parser, and all the tools we use to strip the momenta replace them with 0 0 0 0 0 0 rather than removing them all together to keep the output files valid input files for subsequent oxDNA simulations.

[ErikPoppleton]

There are two possible resolutions to this:

1. We change the output from trajectory_print_momenta = False to look like

x y z a1x a1y a1z a3x a3y a3z 0 0 0 0 0 0\n

2. We change the oat parser to check the first line to see if the momenta are there and if not, write a new parser that splits the a3z on \n rather than

[simonecodispoti]

Hi all!
Sorry for this delayed response, but I was not using oxDNA in the last days. Finally I can confirm you that, following @RodenLuo Log, I could successfully install oxpy & oxdna-analysis-tools. After many tests everything seems to work just fine!

From the mean side, I was processing too much configurations! Now with 2000 confs it works quite fast. @ErikPoppleton I would suggest the first option for making trajectory_print_momenta = False working with mean, because maybe other scripts (or self written scripts by users) relies on the whole trajectory!

[ErikPoppleton]

Side note on the original topic, I helped somebody get started with the software (without oxpy) on an M1 Mac yesterday and everything worked smoothly.

[lorenzo-rovigatti]

That's great to know! Can I mark it as resolved?

[ErikPoppleton]

Any final thoughts on the side discussion this spawned over trajectory_print_momenta?

[RodenLuo]

Great to know! I didn't face the issue here as I never used trajectory_print_momenta = False. But if I understand things correctly, it would waste quite some disk space for 0s if going with the first option. Probably either specify in the doc that this flag is not supposed to be piped with oat or go with the second option. Or, something like trajectory_print_momenta = Zero.

[simonecodispoti]

That's great to know! Can I mark it as resolved?

Just one last thing! Everything seems to work nicely with the exception of the clustering related to te pca script. I had no problems in performing it related to the distance script, but when I try with this:

oat pca trajectory_subset.dat mean_subset.conf pca.json -n 3 -c

I get the following traceback of errors:

Traceback (most recent call last):
  File "/Users/simo/.local/bin/oat", line 8, in <module>
    sys.exit(main())
  File "/Users/simo/.local/lib/python3.9/site-packages/oxDNA_analysis_tools/entry.py", line 24, in main
    to_call.main()
  File "/Users/simo/.local/lib/python3.9/site-packages/oxDNA_analysis_tools/pca.py", line 258, in main
    labs = perform_DBSCAN(traj_info, top_info, coordinates, "euclidean", 12, 8)
  File "/Users/simo/.local/lib/python3.9/site-packages/oxDNA_analysis_tools/clustering.py", line 208, in perform_DBSCAN
    db = DBSCAN(eps=eps, min_samples=min_samples, metric=metric).fit(op) 
  File "/Users/simo/opt/anaconda3/envs/oxdna/lib/python3.9/site-packages/sklearn/cluster/_dbscan.py", line 406, in fit
    neighborhoods = neighbors_model.radius_neighbors(X, return_distance=False)
  File "/Users/simo/opt/anaconda3/envs/oxdna/lib/python3.9/site-packages/sklearn/neighbors/_base.py", line 1097, in radius_neighbors
    results = PairwiseDistancesRadiusNeighborhood.compute(
  File "sklearn/metrics/_pairwise_distances_reduction.pyx", line 1346, in sklearn.metrics._pairwise_distances_reduction.PairwiseDistancesRadiusNeighborhood.compute
  File "/Users/simo/opt/anaconda3/envs/oxdna/lib/python3.9/site-packages/sklearn/utils/fixes.py", line 151, in threadpool_limits
    return threadpoolctl.threadpool_limits(limits=limits, user_api=user_api)
  File "/Users/simo/opt/anaconda3/envs/oxdna/lib/python3.9/site-packages/threadpoolctl.py", line 171, in __init__
    self._original_info = self._set_threadpool_limits()
  File "/Users/simo/opt/anaconda3/envs/oxdna/lib/python3.9/site-packages/threadpoolctl.py", line 268, in _set_threadpool_limits
    modules = _ThreadpoolInfo(prefixes=self._prefixes,
  File "/Users/simo/opt/anaconda3/envs/oxdna/lib/python3.9/site-packages/threadpoolctl.py", line 340, in __init__
    self._load_modules()
  File "/Users/simo/opt/anaconda3/envs/oxdna/lib/python3.9/site-packages/threadpoolctl.py", line 371, in _load_modules
    self._find_modules_with_dyld()
  File "/Users/simo/opt/anaconda3/envs/oxdna/lib/python3.9/site-packages/threadpoolctl.py", line 428, in _find_modules_with_dyld
    self._make_module_from_path(filepath)
  File "/Users/simo/opt/anaconda3/envs/oxdna/lib/python3.9/site-packages/threadpoolctl.py", line 515, in _make_module_from_path
    module = module_class(filepath, prefix, user_api, internal_api)
  File "/Users/simo/opt/anaconda3/envs/oxdna/lib/python3.9/site-packages/threadpoolctl.py", line 606, in __init__
    self.version = self.get_version()
  File "/Users/simo/opt/anaconda3/envs/oxdna/lib/python3.9/site-packages/threadpoolctl.py", line 646, in get_version
    config = get_config().split()
AttributeError: 'NoneType' object has no attribute 'split'

The pca script alone works nicely, and so the clustering via distance, so I believe it is something trivial happening here?

Thanks, this should be the last fix ☺️

[ErikPoppleton]

Hi Simone,
Based on that error I suspect that it didn't find any clusters. Can you try running the serialized input data that clustering produces and run it through oat clustering -e 100 cluster_data.json? Could you also send me the cluster_data.json file so I can add a helpful error message when this happens?

[simonecodispoti]

Hi, thanks Erik,
I'm attaching the file related to the error in txt format (json not supported here ...)

cluster_data.txt

with oat clustering -e 100 cluster_data.json I still get the same error.

[ErikPoppleton]

Commit 6542715 fixes the problem with oat not being able to read trajectory files with the velocities removed. Thank you @zoombya for the suggestion how to make it clean.

[ErikPoppleton]

@simonecodispoti I cleaned up the clustering a bit, but couldn't reproduce your error with a toy system. If you're still running into this, could you please open up a new issue and include a small trajectory file which generates the error for you?