xmipp_volume_align with option --frm does not work after the recent update #294

MohamadHarastani · 2020-06-29T12:50:46Z

There is an issue with xmipp_volume_align, that looks python-binding related.

To reproduce:

xmipp_volume_align --i1 volume1.spi --i2 volume2.spi --frm

gives the following error:
XMIPP_ERROR 60: Not classified error.
Cannot import sh_alignment.
File: libraries/interface/frm.cpp line: 85

I would appreciate your help in this regards.

Also, I have noticed that recompiling xmipp (xmipp-bundle / developers way) causes Scipion2 to fail starting, is this normal?

Kindly,
Mohamad

DStrelak · 2020-06-29T13:48:35Z

Also, I have noticed that recompiling xmipp (xmipp-bundle / developers way) causes Scipion2 to fail starting, is this normal?

Xmipp in devel version is now migrated to python 3 (and expected to be used with scipion 3).
If you need python 2 version, use following version: https://github.com/I2PC/xmipp/releases/tag/v3.19.04-Apollo-DEVEL

MohamadHarastani · 2020-06-29T14:54:24Z

Also, I have noticed that recompiling xmipp (xmipp-bundle / developers way) causes Scipion2 to fail starting, is this normal?

Xmipp in devel version is now migrated to python 3 (and expected to be used with scipion 3).
If you need python 2 version, use following version: https://github.com/I2PC/xmipp/releases/tag/v3.19.04-Apollo-DEVEL

Thanks for the fast reply. I moved to Scipion3 already and I am happy with it, with few bugs to report. Thanks for all the Scipion/Xmipp team for your efforts and support.
I don't need to stay in Scipion2 once the issue I reported is solved (I heavily depend on xmipp_volume_align --frm). I tested the version in the link and it didn't compile on my computer. Here is the log.txt after executing "./xmipp >> log.txt"
log.txt

DStrelak · 2020-06-30T08:53:08Z

I tested the version in the link and it didn't compile on my computer.

Thanks for reporting that. I've created separate issue for that, see #295 for a workaround.

MohamadHarastani · 2020-07-01T11:56:45Z

I tested the version in the link and it didn't compile on my computer.

Thanks for reporting that. I've created separate issue for that, see #295 for a workaround.

Thanks, I confirm that xmipp_volume_align works there (for python2).

DStrelak · 2020-07-07T16:01:41Z

@MohamadHarastani
What is your python version? I can confirm that the xmipp_volume_align with --frm flag does not work, but I get different errors (for current devel, i.e. python3 version).
I get segmentation fault for Python 2.7.17, and for version 3.8.3 I get:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = '/home/david/git/scipion3/xmipp-bundle/build/bindings/python:/home/david/git/scipion3/xmipp-bundle/build/pylib:'
  program name = '\U6d6f682f\U61642f65\U2f646976\U696e696d\U646e6f63\U652f3261\U2f73766e\U70696373\U336e6f69\U6e69622f\U7479702f\U0a6e6f68'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = ''
  sys.base_prefix = '/tmp/build/80754af9/python_1593706424329/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.base_exec_prefix = '/tmp/build/80754af9/python_1593706424329/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.executable = ''
  sys.prefix = '/tmp/build/80754af9/python_1593706424329/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.exec_prefix = '/tmp/build/80754af9/python_1593706424329/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.path = [
    '/home/david/git/scipion3/xmipp-bundle/build/bindings/python',
    '/home/david/git/scipion3/xmipp-bundle/build/pylib',
    '',
    '/tmp/build/80754af9/python_1593706424329/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python38.zip',
    '/tmp/build/80754af9/python_1593706424329/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.8',
    '/tmp/build/80754af9/python_1593706424329/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f44e4e00740 (most recent call first):
<no Python frame>

MohamadHarastani · 2020-07-08T11:27:03Z

@DStrelak
I don't know if I understood the question correctly, if the following doesn't give you an answer, please let me know.

mohamad@mohamad:~$ echo $PYTHONPATH 
/home/mohamad/xmipp-bundle/xmipp/build/bindings/python:/home/mohamad/xmipp-bundle/xmipp/build/pylib:
mohamad@mohamad:~$ echo $SCIPION_HOME 
/home/mohamad/scipion3
mohamad@mohamad:~$ python --version
Python 2.7.17
mohamad@mohamad:~$ python3 --version
Python 3.6.9
mohamad@mohamad:~$ which python && which python3
/usr/bin/python
/usr/bin/python3

The error I get:

mohamad@mohamad:~/Work$ xmipp_volume_align --i1 volume1.spi --i2 volume2.spi --frm
XMIPP_ERROR 60:  Not classified error.
Cannot import sh_alignment.
File: libraries/interface/frm.cpp line: 85

I get the same error on another computer as well

DStrelak · 2020-07-10T09:54:25Z

Hi @MohamadHarastani ,
I have just pushed the fix. With the example you sent me some time ago, I got identical results.
However, I am a bit worried that there still might be some inconsistencies, as there has been a rather big change between python 2 and 3 in the way of handling integer division. Python 2 performs floor division, while Python 3 performs real division. This also affected numpy and (probably) other packages, so we might see some deviations.

Since you have more experience with this program, can I ask you to test the code in this branch with more examples, and send me those files that return different results (between version 2 and 3)?

MohamadHarastani · 2020-07-13T23:46:46Z

Hi @DStrelak ,
I have just checked the fix. I analyzed 100 volumes, and I got identical results as previous.
I see that you did new fixes, nevertheless, I will continue testing and report any inconsistency.
Thanks a lot for your efforts!
Mohamad

DStrelak · 2020-07-14T07:53:59Z

Hi @MohamadHarastani ,
I'm glad that the results did not change.
Never mind those latest changes, I've been just throwing code from one pile to another.
May we consider this fixed?

MohamadHarastani · 2020-07-14T21:17:20Z

Hi @MohamadHarastani ,
I'm glad that the results did not change.
Never mind those latest changes, I've been just throwing code from one pile to another.
May we consider this fixed?

Yes, we may consider it fixed.
Cheers

MohamadHarastani · 2020-07-15T22:24:51Z

After some tests of this program, I found two integer division issues here:
https://github.com/I2PC/xmipp/blob/devel/src/xmipp/external/sh_alignment/python/tompy/filter.py#L171
Lines 171 - 172 should be replaced by:
filter_vol[size[0]//2, :, :] = 0
filter_vol[size[0]//2, :, size[2]//2] = 1
The issue can be re-closed after these fixes.

DStrelak · 2020-07-16T08:39:03Z

Thanks for spotting those. See #301

MohamadHarastani · 2021-11-23T07:54:31Z

Hello again,
I found out that xmipp_volume_align --frm is not working in the new release.

 -bash-4.2$ xmipp_volume_align --i1 volume1.spi --i2 volume2.spi --frm
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = '/home/guest/xmipp-bundle/xmipp/build/bindings/python:/home/guest/xmipp-bundle/xmipp/build/pylib:'
  program name = '/usr/bin/python\x0a'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/bin/python\n'
  sys.base_prefix = '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.base_exec_prefix = '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.executable = '/usr/bin/python\n'
  sys.prefix = '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.exec_prefix = '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.path = [
    '/home/guest/xmipp-bundle/xmipp/build/bindings/python',
    '/home/guest/xmipp-bundle/xmipp/build/pylib',
    '',
    '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python38.zip',
    '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.8',
    '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Consequently, xmipp_volumeset_align, xmipp_nma_alignment_vol and there mpi versions are not working neither.

Thanks in advance,

Regards
Mohamad

MohamadHarastani · 2021-11-23T08:41:37Z

I found a way to run the program on the terminal by activating the conda environment of scipion3. Still, xmipp_volumeset_align and xmipp_nma_alignment_vol are failing without a clear message (at least to me) to debug the problem.
Probably the easiest way to reproduce is to run the continuousflex tests:
scipion3 tests continuousflex.tests.test_workflow_HEMNMA3D

DStrelak · 2021-11-23T09:08:47Z

Hello again, I found out that xmipp_volume_align --frm is not working in the new release.
...
ModuleNotFoundError: No module named 'encodings'

Hi @MohamadHarastani ,
this looks like a problem of your Python installation (see e.g. SO).
That would also explain why the error disappears when you use Conda environment (that one has its own Python installation).

MohamadHarastani · 2021-11-23T09:11:55Z

Thanks. I just need to find why the other programs are failing (xmipp_volumeset_align and xmipp_nma_alignment_vol, and their mpi versions).
I will provide an isolated test that reproduces the error asap.
Thanks for the help

MohamadHarastani · 2021-11-23T09:28:21Z

An isolated test (a minute):
xmipp_volumeset_align -i volumes.xmd -o output.xmd --ref ref.vol
https://filesender.renater.fr/?s=download&token=6db36ed6-ffe4-48d9-9c16-36ae1982e7ad

I tested with switching between the old and new xmipp versions. The old one works (although I get something about the endianness). The new, I get Segmentation fault (core dumped)

DStrelak · 2021-11-23T09:31:17Z

I can confirm that this test fails with

Signal: Segmentation fault (11)
Signal code: Address not mapped (1)
Failing at address: 0xb8

At first glance, the problem is probably caused by some problem at the MetaData processing:

[ 3] lib/libXmippCore.so(_ZN10MetaDataDb4initERKSt6vectorI7MDLabelSaIS1_EE+0x5c)[0x7f9d8e1e5e08]
[ 4] lib/libXmippCore.so(_ZN10MetaDataDb12copyMetadataERKS_b+0x41)[0x7f9d8e1e5e75]
[ 5] lib/libXmippCore.so(_ZN10MetaDataDbaSERKS_+0x2c)[0x7f9d8e1e8e32]
[ 6] lib/libXmipp.so(_ZN19ProgNmaAlignmentVol15createWorkFilesEv+0x9a)[0x7f9d8f55cf96]
[ 7] lib/libXmippParallel.so(_ZN13MpiProgNMAVol15createWorkFilesEv+0x78)[0x7f9d8dfc79ea]
[ 8] lib/libXmipp.so(_ZN19ProgNmaAlignmentVol10preProcessEv+0xc0)[0x7f9d8f55d34c]
[ 9] lib/libXmippCore.so(_ZN20XmippMetadataProgram3runEv+0x85)[0x7f9d8e139657]

dmaluenda assigned ajimoreno, DStrelak and DavidHerreros and unassigned ajimoreno Jun 29, 2020

DStrelak added the in progress label Jul 7, 2020

DStrelak removed the in progress label Jul 15, 2020

DStrelak closed this as completed Jul 15, 2020

MohamadHarastani reopened this Jul 15, 2020

DStrelak closed this as completed Jul 16, 2020

MohamadHarastani reopened this Nov 23, 2021

DStrelak added bug Something isn't working confirmed labels Nov 23, 2021

DStrelak assigned albertmena Nov 23, 2021

DStrelak mentioned this issue Nov 25, 2021

Multiple programs: Adding a common implementation of the rerun #475

Closed

DStrelak mentioned this issue Dec 3, 2021

Multiple programs: Adding a common implementation of the rerun #484

Merged

DStrelak closed this as completed in #484 Dec 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

xmipp_volume_align with option --frm does not work after the recent update #294

xmipp_volume_align with option --frm does not work after the recent update #294

MohamadHarastani commented Jun 29, 2020

DStrelak commented Jun 29, 2020

MohamadHarastani commented Jun 29, 2020

DStrelak commented Jun 30, 2020

MohamadHarastani commented Jul 1, 2020

DStrelak commented Jul 7, 2020 •

edited

Loading

MohamadHarastani commented Jul 8, 2020

DStrelak commented Jul 10, 2020

MohamadHarastani commented Jul 13, 2020

DStrelak commented Jul 14, 2020

MohamadHarastani commented Jul 14, 2020

MohamadHarastani commented Jul 15, 2020

DStrelak commented Jul 16, 2020

MohamadHarastani commented Nov 23, 2021

MohamadHarastani commented Nov 23, 2021 •

edited

Loading

DStrelak commented Nov 23, 2021

MohamadHarastani commented Nov 23, 2021

MohamadHarastani commented Nov 23, 2021 •

edited

Loading

DStrelak commented Nov 23, 2021

xmipp_volume_align with option --frm does not work after the recent update #294

xmipp_volume_align with option --frm does not work after the recent update #294

Comments

MohamadHarastani commented Jun 29, 2020

DStrelak commented Jun 29, 2020

MohamadHarastani commented Jun 29, 2020

DStrelak commented Jun 30, 2020

MohamadHarastani commented Jul 1, 2020

DStrelak commented Jul 7, 2020 • edited Loading

MohamadHarastani commented Jul 8, 2020

DStrelak commented Jul 10, 2020

MohamadHarastani commented Jul 13, 2020

DStrelak commented Jul 14, 2020

MohamadHarastani commented Jul 14, 2020

MohamadHarastani commented Jul 15, 2020

DStrelak commented Jul 16, 2020

MohamadHarastani commented Nov 23, 2021

MohamadHarastani commented Nov 23, 2021 • edited Loading

DStrelak commented Nov 23, 2021

MohamadHarastani commented Nov 23, 2021

MohamadHarastani commented Nov 23, 2021 • edited Loading

DStrelak commented Nov 23, 2021

DStrelak commented Jul 7, 2020 •

edited

Loading

MohamadHarastani commented Nov 23, 2021 •

edited

Loading

MohamadHarastani commented Nov 23, 2021 •

edited

Loading