Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python segmentation fault on Ubuntu 22.04 CI #8

Closed
S-Dafarra opened this issue Sep 12, 2023 · 10 comments
Closed

Python segmentation fault on Ubuntu 22.04 CI #8

S-Dafarra opened this issue Sep 12, 2023 · 10 comments

Comments

@S-Dafarra
Copy link
Member

The latest commit in my development branch failed in CI: https://github.com/ami-iit/hippopt/actions/runs/6160272586

Here the error

2023-09-12T13:39:20.2735152Z ##[group]Run pytest
2023-09-12T13:39:20.2735466Z �[36;1mpytest�[0m
2023-09-12T13:39:20.2790404Z shell: /usr/bin/bash -l {0}
2023-09-12T13:39:20.2790651Z env:
2023-09-12T13:39:20.2790882Z   INPUT_RUN_POST: true
2023-09-12T13:39:20.2791144Z   CONDA: /usr/share/miniconda3
2023-09-12T13:39:20.2791423Z   CONDA_PKGS_DIR: /home/runner/conda_pkgs_dir
2023-09-12T13:39:20.2791699Z ##[endgroup]
2023-09-12T13:39:21.7615135Z Fatal Python error: Segmentation fault
2023-09-12T13:39:21.7615653Z 
2023-09-12T13:39:21.7616004Z Current thread 0x00007f938a108740 (most recent call first):
2023-09-12T13:39:21.7617004Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/casadi/casadi.py", line 47783 in solve
2023-09-12T13:39:21.7617643Z   File "/home/runner/work/hippopt/hippopt/src/hippopt/base/opti_solver.py", line 572 in solve
2023-09-12T13:39:21.7618306Z   File "/home/runner/work/hippopt/hippopt/src/hippopt/base/multiple_shooting_solver.py", line 883 in solve
2023-09-12T13:39:21.7619045Z   File "/home/runner/work/hippopt/hippopt/src/hippopt/base/problem.py", line 163 in solve
2023-09-12T13:39:21.7619783Z   File "/home/runner/work/hippopt/hippopt/test/test_multiple_shooting.py", line 323 in test_multiple_shooting
2023-09-12T13:39:21.7620920Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
2023-09-12T13:39:21.7621801Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
2023-09-12T13:39:21.7623072Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
2023-09-12T13:39:21.7623970Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
2023-09-12T13:39:21.7624750Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest
2023-09-12T13:39:21.7625581Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
2023-09-12T13:39:21.7626400Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
2023-09-12T13:39:21.7627191Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
2023-09-12T13:39:21.7627960Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
2023-09-12T13:39:21.7628739Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda>
2023-09-12T13:39:21.7629529Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call
2023-09-12T13:39:21.7630337Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
2023-09-12T13:39:21.7631160Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report
2023-09-12T13:39:21.7631976Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol
2023-09-12T13:39:21.7632812Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
2023-09-12T13:39:21.7633624Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
2023-09-12T13:39:21.7634413Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
2023-09-12T13:39:21.7635198Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
2023-09-12T13:39:21.7636003Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
2023-09-12T13:39:21.7637048Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
2023-09-12T13:39:21.7637861Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
2023-09-12T13:39:21.7638651Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
2023-09-12T13:39:21.7639896Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main
2023-09-12T13:39:21.7640673Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session
2023-09-12T13:39:21.7641488Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
2023-09-12T13:39:21.7642287Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
2023-09-12T13:39:21.7643050Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
2023-09-12T13:39:21.7643825Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
2023-09-12T13:39:21.7644606Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169 in main
2023-09-12T13:39:21.7645410Z   File "/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192 in console_main
2023-09-12T13:39:21.7646014Z   File "/usr/share/miniconda3/envs/test/bin/pytest", line 10 in <module>
2023-09-12T13:39:21.7646537Z 
2023-09-12T13:39:21.7647426Z Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, casadi._casadi, yaml._yaml, lxml._elementpath, lxml.etree (total: 17)
2023-09-12T13:39:23.7071508Z /home/runner/work/_temp/de045181-ba10-47d2-a765-f7106b064fdc.sh: line 1:  2318 Segmentation fault      (core dumped) pytest
2023-09-12T13:39:23.7072061Z ............
2023-09-12T13:39:23.7079340Z ##[error]Process completed with exit code 139.

I don't think this is related to the code itself. Moreover, it is not failing on Windows and Mac. I was not committing for a while, so maybe there is some dependency change that is causing issues in Linux.
@S-Dafarra
Copy link
Member Author

It is failing also in the main version: https://github.com/ami-iit/hippopt/actions/runs/6160894042/job/16718863954

@traversaro
Copy link
Contributor

I am able to get the test to work by downgrading ipopt to from ipopt 3.14.12 he6d3896_1 to ipopt 3.14.12 _0

@S-Dafarra
Copy link
Member Author

Thanks a lot @traversaro. Any suggestion on how to tackle this?

@traversaro
Copy link
Contributor

I was able to reduce the failing test to:

import dataclasses

import casadi as cs
import numpy as np
import pytest

from hippopt import (
    ContinuousVariable,
    ExpressionType,
    OptimizationObject,
    OptimizationProblem,
    Parameter,
    StorageType,
    default_storage_field,
)


@dataclasses.dataclass
class MyTestVar(OptimizationObject):
    variable: StorageType = default_storage_field(ContinuousVariable)
    size: dataclasses.InitVar[int] = dataclasses.field(default=3)

    def __post_init__(self, size: int = 3):
        self.variable = np.zeros(size)


def test_opti_solver():
    size = 4
    problem = OptimizationProblem()
    var = problem.generate_optimization_objects(input_structure=MyTestVar(size=size))
    np.random.seed(123)
    a = 10.0 * np.random.rand(size) + 0.01
    b = 20.0 * np.random.rand(size) - 10.0
    c = 20.0 * np.random.rand(size) - 10.0

    problem.add_expression(
        mode=ExpressionType.minimize,
        expression=(
            a[k] * cs.power(var.variable[k], 2) + b[k] * var.variable[k]
            for k in range(size)
        ),
    )

    problem.add_expression(
        mode=ExpressionType.subject_to,
        expression=(var.variable[k] >= c[k] for k in range(size)),  # noqa
    )

    output = problem.solver().solve()

Do you have any idea on how to enable more casadi and/or ipopt verbosity?

@traversaro
Copy link
Contributor

I reproduced the problem with just a simple ipopt example: https://github.com/casadi/casadi/blob/7f8f055024ed6d86532612f1b706195c7d41074a/docs/examples/python/ipopt_nl.py , so probably we can move the discussion in https://github.com/conda-forge/casadi-feedstock . As a workaround, you can add ipopt=3.14.12=*_0 to the installation instructions/CI to get the tests back on track.

@S-Dafarra
Copy link
Member Author

Sounds good! Thanks a lot!

S-Dafarra added a commit that referenced this issue Sep 12, 2023
@S-Dafarra
Added workaround for #8
dd1d2c4
@traversaro traversaro mentioned this issue Sep 12, 2023
Closed
1 task
@S-Dafarra
Copy link
Member Author

As mentioned by @traversaro in conda-forge/casadi-feedstock#91 (comment), a possible workaround is to force libgomp to the 12.* version. As a matter of fact, this is necessary only on Ubuntu as it is not a dependency for Mac or Windows. Related commit: fbbf415

@traversaro
Copy link
Contributor

traversaro commented Sep 20, 2023 •
edited
Loading

libgomp-13.2.0-h807b86a_2 build of gomp has just been uploaded with the fix for this issue (see conda-forge/ctng-compilers-feedstock#117). I think that in 10/20 minutes it will be propagated in the CDN, and so in theory we can remove the CI workaround.

S-Dafarra added a commit that referenced this issue Sep 22, 2023
@S-Dafarra
Removed workaround for #8
42f7999
@S-Dafarra
Copy link
Member Author

Removed workaround with 42f7999. CI is happy. Thanks a lot @traversaro! Closing

@traversaro
Copy link
Contributor

Just as a future warning, it seems that anyhow something fishy is going on w.r.t. the environ variable not being passed correctly in Python, see conda-forge/casadi-feedstock#93 . The linked bug is on the conda-forge side, but also apt packages on recent Debian and Ubuntu distros (not 22.04, fortunately) seems to be affected. I guess this could affect the correct propagation of OMP_ env variable, so if you see anything fishy related to that it could be related to this problem.

S-Dafarra added a commit that referenced this issue Jan 15, 2024
@S-Dafarra
Added workaround for #8
0ab71b9
S-Dafarra added a commit that referenced this issue Jan 15, 2024
@S-Dafarra
Removed workaround for #8
05d4758
S-Dafarra added a commit that referenced this issue Jan 15, 2024
@S-Dafarra
Added workaround for #8
740c12d
S-Dafarra added a commit that referenced this issue Jan 15, 2024
@S-Dafarra
Removed workaround for #8
ca3ab20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@traversaro @S-Dafarra