Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

segfault after https://example.com/accounts/saml/example/acs/ #3593

Closed
brianray opened this issue Jan 15, 2024 · 26 comments
Closed

segfault after https://example.com/accounts/saml/example/acs/ #3593

brianray opened this issue Jan 15, 2024 · 26 comments

Comments

@brianray
Copy link

Getting a weird segfault after https://example.com/accounts/saml/example/acs/ is POST.

Looking at the SAML-tracer (chrome extension) everything looks good, as expected. My guess is that one of the dependencies like libxmlsec1... Not really sure. I am having a hard time figuring how what is segfault-ing. Can't run on my local machine as I don't have a SSO config for that to debug.

@brianray
Copy link
Author

Nevermind, this is fixed here 0c49379

@brianray
Copy link
Author

sorry the issue still remains with code that is in main. The segfault happens here:

The exception never is handled as it sefaults. My guess it happens inside of the OneLogin_Saml2_Auth https://github.com/SAML-Toolkits/python3-saml/blob/master/src/onelogin/saml2/auth.py which is a different project.

@brianray brianray reopened this Jan 16, 2024
@pennersr
Copy link
Owner

Nevermind, this is fixed here 0c49379

That is only about requiring POST requests -- it is unlikely that it fixes your issue as your scenario involves POST.

The exception never is handled as it sefaults.

A segfault really points to an issue elsewhere, as allauth itself, being pure python code, typically does not cause segfaults. So this needs to be fixed upstream. Having said that, if there are any workarounds that can be put in place to avoid the segfault, let me know... but other than that this really is not actionable on this end.

@pennersr
Copy link
Owner

There are a few mentions of segfaults here: https://github.com/search?q=repo%3ASAML-Toolkits%2Fpython3-saml+segfault&type=issues

@pennersr
Copy link
Owner

Can you try downgrading lxml as is mentioned here? SAML-Toolkits/python3-saml#389

@pennersr pennersr closed this as not planned Won't fix, can't repro, duplicate, stale Jan 16, 2024
@nosnilmot
Copy link

Use pip install --no-binary lxml lxml to install lxml from source and link to system libxml2, same as xmlsec, to avoid conflicting libxml2 libraries.
See also https://bugs.launchpad.net/lxml/+bug/1960668#yui_3_10_3_1_1705879387759_1639

@vinodpandey
Copy link

vinodpandey commented Jan 26, 2024

We are also getting the same segfault error (Process finished with exit code 245). The segfault happens in the same auth.process_response() method.

Installed lxml and xmlsec both from source. But, the issue still persists. It happens on about 30-40% of the POST requests. Rest of the time the intergration works as expected.

pip install --no-binary lxml lxml
pip install --no-binary xmlsec xmlsec

error log



Fatal Python error: Segmentation fault

Current thread 0x000000030bc96000 (most recent call first):
  File "/venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 54 in wrapped_view
  File "/venv/lib/python3.10/site-packages/django/utils/decorators.py", line 46 in _wrapper
  File "/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 84 in view
  File "/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197 in _get_response
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/cachalot/panels.py", line 41 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/logging.py", line 77 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/staticfiles.py", line 117 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/headers.py", line 50 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/timer.py", line 65 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/middleware.py", line 57 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/allauth/account/middleware.py", line 14 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/apps/util/middleware.py", line 48 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 140 in get_response
  File "/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 132 in __call__
  File "/venv/lib/python3.10/site-packages/django/contrib/staticfiles/handlers.py", line 80 in __call__
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/wsgiref/handlers.py", line 137 in run
  File "/venv/lib/python3.10/site-packages/django/core/servers/basehttp.py", line 227 in handle_one_request
  File "/venv/lib/python3.10/site-packages/django/core/servers/basehttp.py", line 202 in handle
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/socketserver.py", line 747 in __init__
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/socketserver.py", line 360 in finish_request
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/socketserver.py", line 683 in process_request_thread
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 953 in run
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x000000030ac93000 (most recent call first):
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/selectors.py", line 416 in select
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/socketserver.py", line 232 in serve_forever
  File "/venv/lib/python3.10/site-packages/django/core/servers/basehttp.py", line 246 in run
  File "/venv/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 158 in inner_run
  File "/venv/lib/python3.10/site-packages/django/utils/autoreload.py", line 64 in wrapper
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 953 in run
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00000002044a82c0 (most recent call first):
  File "/venv/lib/python3.10/site-packages/django/utils/autoreload.py", line 405 in tick
  File "/venv/lib/python3.10/site-packages/django/utils/autoreload.py", line 350 in run_loop
  File "/venv/lib/python3.10/site-packages/django/utils/autoreload.py", line 344 in run
  File "/venv/lib/python3.10/site-packages/django/utils/autoreload.py", line 661 in start_django
  File "/venv/lib/python3.10/site-packages/django/utils/autoreload.py", line 680 in run_with_reloader
  File "/venv/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 118 in run
  File "/venv/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 111 in handle
  File "/venv/lib/python3.10/site-packages/django/core/management/base.py", line 460 in execute
  File "/venv/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 74 in execute
  File "/venv/lib/python3.10/site-packages/django/core/management/base.py", line 414 in run_from_argv
  File "/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 440 in execute
  File "/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 446 in execute_from_command_line
  File "/manage.py", line 39 in <module>

Extension modules: yaml._yaml, charset_normalizer.md, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, MySQLdb._mysql, markupsafe._speedups, lxml._elementpath, lxml.etree, PIL._imaging, _cffi_backend, xmlsec (total: 22)


@pennersr
Copy link
Owner

Can you capture one of the POST requests that causes the segfault and turn it into a small example project demonstrating the issue?

@vinodpandey
Copy link

sure @pennersr, let me create a small example project to replicate this issue.

@vinodpandey
Copy link

Hi @pennersr , I have create a sample project at https://github.com/vinodpandey/saml-allauth-demo.

Below are the steps to replicate the issues:

  • Python version for below setup is 3.10.12
  • setup code
git clone https://github.com/vinodpandey/saml-allauth-demo.git
cd saml-allauth-demo

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
  • run server (no need to run migrations. sqlite file is already checked-in in the repository)
python manage.py runserver

  • access http://localhost:8000 and click on SSO Login. This will redirect to https://mocksaml.com/ IdP authentication page. Click on Sign In and it will redirect back our home page.
  • Click on Logout button.
  • Repeat above 2 steps for approx 10 times. The segfault occurs for me after 4-5 tries.

I have uploaded video at https://youtu.be/NV2FokoH1zI which shows the above repository setup steps and replicates the segfault issue.

In above same project, I ran below commands and tested it again. But, even after that, the issue occurs after 4-5 tries.

pip uninstall lxml
pip install --no-binary lxml lxml==4.9.3

@pennersr
Copy link
Owner

For me, the crash does not reproduce. Though, I am running nix with:

xmlsec-1.2.34  XML Security Library in C based on libxml2
libxml2-2.11.5  XML parsing library for C

And relevant pip freeze output:

lxml==4.9.3
xmlsec==1.3.13

Can you check what version of xmlsec / libxml2 you are using?

@vinodpandey
Copy link

vinodpandey commented Jan 27, 2024

My software versions in Mac are below

libxml2: 2.12.4
xmlsec1: 1.2.37

pip freeze output:

lxml==4.9.3
xmlsec==1.3.13

I created a docker image with Ubuntu 20.04 in https://github.com/vinodpandey/saml-allauth-demo and the issue didn't reproduce for me also. In Ubuntu 20.04, it was working correctly.

It seems the issue is limited to Macbook M2 only. I am debugging the issue and will update if I find the root cause/fix for this.

@nosnilmot
Copy link

Good news (?), I am lucky enough to also have an M2 mac, and I was able to reproduce the crash with the demo provided by @vinodpandey

I believe the problem is again caused by mismatches in version of libxml2 that the native library libxmlsec1 was linked against at install time vs. the version it loads at runtime.

I too have libxml2 2.12.4 and libxmlsec1 1.2.37 installed but crucially when libxmlsec1 was installed (in June 2023) it was almost certainly linked against libxml2 2.11.x.

Reinstalling libxmlsec1 to link against your current libxml2 will likely fix the crashes. That may be harder than you want on macos / brew because python-xmlsec (which appears to be desperately lacking in maintenance) is not compatible with current libxmlsec1 available in brew (1.3.2).

Assuming you already worked around that to get libxmlsec1 1.2.37, then brew reinstall libxmlsec1@1.2.37 may be sufficient, but you may also need to edit the formula as the source is now in the older-releases directory. Update to 1.2.39 is also possible, and removes the need for the configure patch carried by the brew formula.

@vinodpandey
Copy link

@nosnilmot I uninstalled both libxmlsec1 and libxml2 and after that installed libxmlsec1 1.2.37 using xmlsec/python-xmlsec#254 (comment) which itself re-installed libxml2 as it's dependency. I re-created the virtual environment and re-installed dependencies using pip install -r requirements.txt --no-cache-dir.

I found a workaround for fixing the crashes in Mac M2. You are correct, the issue is occuring because of version mismatch only.

After running pip install lxml==5.1.0, the crash doesn't occur (i tried 20 iterations of login/logout). When I installed lxml==5.1.0, instead of using system libxml2, it was using it's own bundled libxml2. Since it's bundled libxml2 is 2.12.3 and brew libxml2 is 2.12.4, it co-incidentally matches the version with brew libxml2 version and this might have fixed the crashes.

Below is my analysis regarding libxml2 version.

In my Mac M2, it has it's own version of libxml2 which is 2.9.13 and brew installed version is 2.12.4

xmllint --version
xmllint: using libxml version 20913

/usr/local/opt/libxml2/bin/xmllint --version
/usr/local/opt/libxml2/bin/xmllint: using libxml version 21204

When using pip install lxml==4.9.3, bundled lxml2 is used. This is the same output which I get in Ubuntu 20.04 where it is working correctly.

python -c "import lxml.etree as etree;print(etree.LXML_VERSION);print(etree.LIBXML_VERSION);print(etree.LIBXML_COMPILED_VERSION);"

(4, 9, 3, 0)
(2, 10, 3)
(2, 10, 3)

When using pip install --force-reinstall --no-binary lxml lxml==4.9.3 --no-cache-dir, Both LIBXML_VERSION and LIBXML_COMPILED_VERSION are same (same as system installed lxml2).

python -c "import lxml.etree as etree;print(etree.LXML_VERSION);print(etree.LIBXML_VERSION);print(etree.LIBXML_COMPILED_VERSION);"

(4, 9, 3, 0)
(2, 9, 13)
(2, 9, 13)

Now, after running the tests again, the crash still occurs.

I upgraded lxml lxml==5.1.0 and the issue didn't occur after that. lxml is using it's bunduled libxml2 library.

python -c "import lxml.etree as etree;print(etree.LXML_VERSION);print(etree.LIBXML_VERSION);print(etree.LIBXML_COMPILED_VERSION);"

(5, 1, 0, 0)
(2, 12, 3)
(2, 12, 3)

In Ubuntu 20.04, if I upgrade lxml to lxml==5.1.0, the crashes start appearing in Ubuntu.

pkg-config --modversion libxml-2.0
2.9.10

xmlsec1-config --version
1.2.28

python -c "import lxml.etree as etree;print(etree.LXML_VERSION);print(etree.LIBXML_VERSION);print(etree.LIBXML_COMPILED_VERSION);"

(5, 1, 0, 0)
(2, 12, 3)
(2, 12, 3)

I couldn't find a way to make pip install --force-reinstall --no-binary lxml lxml==4.9.3 --no-cache-dir to use brew installed libxml2 (2.12.4) instead of system libxml2 (2.9.13).

I tried below commands, but that resulted in mismatched LIBXML_VERSION and LIBXML_COMPILED_VERSION. The crashes continued to occur after that.


export PATH="/usr/local/opt/libxml2/bin:$PATH"
export LDFLAGS="-L/usr/local/opt/libxml2/lib"
export CPPFLAGS="-I/usr/local/opt/libxml2/include"
export PKG_CONFIG_PATH="/usr/local/opt/libxml2/lib/pkgconfig"

pip install --force-reinstall --no-binary lxml lxml==4.9.3 --no-cache-dir


python -c "import lxml.etree as etree;print(etree.LXML_VERSION);print(etree.LIBXML_VERSION);print(etree.LIBXML_COMPILED_VERSION);"

(4, 9, 3, 0)
(2, 9, 13)
(2, 12, 4)

@nosnilmot
Copy link

It is important that all components (lxml, libxmlsec1 and python-xmlsec) are using (linked with) the same version of libxml2. Because you are using brew and libxmlsec1 installed by brew, this means it should be libxml2 from brew that all components use.

lxml binaries bundle a copy of libxml2 , so when used with xmlsec, lxml must be built from source by installing with pip install --no-binary lxml lxml to have best chance it uses same version as xmlsec.

Both lxml and python-xmlsec will use pkg-config to find libraries to link to, so the correct brew libxml2 path must be set in PKG_CONFIG_PATH before installing.

I do not know why your lxml is loading system libxml2 instead of from brew. I wonder if it is because you have brew in /usr/local rather than the normal /opt/homebrew for arm64. why do you have that? are you actually using x86 brew on arm64? Does brew doctor report good health? My setup, with brew in /opt/homebrew, reports:

$ python -c "import lxml.etree as etree;print(etree.LXML_VERSION);print(etree.LIBXML_VERSION);print(etree.LIBXML_COMPILED_VERSION);"
(4, 9, 3, 0)
(2, 12, 4)
(2, 12, 4)

I can force same result as you only if I override DYLD_LIBRARY_PATH, do you do this?

$ DYLD_LIBRARY_PATH=/usr/lib python -c "import lxml.etree as etree;print(etree.LXML_VERSION);print(etree.LIBXML_VERSION);print(etree.LIBXML_COMPILED_VERSION);"
(4, 9, 3, 0)
(2, 9, 13)
(2, 12, 4)

I believe this is a reliable way to get working lxml, libxmlsec1 and python-xmlsec installed on macos:

  1. Install libxml2 from brew:
    brew install libxml2
  2. Install libxmlsec1 version 1.2.x with brew
    This is out of scope of these instructions but I'm trying to make it easier.
  3. Install python packages lxml and pyhton-xmlsec ensuring they link to correct libxml2:
    export PKG_CONFIG_PATH="$(brew --prefix)/opt/libxml2/lib/pkgconfig"
    pip install --force-reinstall --no-cache-dir --no-binary lxml lxml xmlsec

--force-reinstall and --no-cache-dir are not needed if this is a clean install.

@vinodpandey
Copy link

Thanks @nosnilmot for helping to debug the issue. I re-installed homebrew and rechecked the steps above. I was still getting the same mismatched etree.LIBXML_VERSION and etree.LIBXML_COMPILED_VERSION. I debugged futher and found that the commands are behaving differently with different versions of python. I am using Python 3.10.12 where it is not working correctly whereas in Python 3.9.6 I am getting the same results as yours (lxml is using the brew version libxml2). It is working correctly in Python 3.10.13 also.

Although, pip install --no-binary lxml lxml was the correct solution. But, it was not working for me on Macbook M2 because I was on Python 3.10.12

In Python 3.9.6 on my Mac it works correctly.

python -V
Python 3.9.6

export LDFLAGS="-L/opt/homebrew/opt/libxml2/lib"
export CPPFLAGS="-I/opt/homebrew/opt/libxml2/include"
export PKG_CONFIG_PATH="/opt/homebrew/opt/libxml2/lib/pkgconfig"

python -m venv venv
source venv/bin/activate

pip install --no-binary lxml lxml==4.9.3

python -c "import lxml.etree as etree;print(etree.LXML_VERSION);print(etree.LIBXML_VERSION);print(etree.LIBXML_COMPILED_VERSION);"

(4, 9, 3, 0)
(2, 12, 4)
(2, 12, 4)


In Python 3.10.12, executing the same command gives mismatched version of LIBXML_VERSION and LIBXML_COMPILED_VERSION.


python -c "import lxml.etree as etree;print(etree.LXML_VERSION);print(etree.LIBXML_VERSION);print(etree.LIBXML_COMPILED_VERSION);"
(4, 9, 3, 0)
(2, 9, 13)
(2, 12, 4)

@nosnilmot
Copy link

You still have something odd in your system configuration. I have no idea why it only shows up for python 3.10.12 for you - maybe that's installed differently? (current brew 3.10 is 3.10.13)
Did you check if you have set DYLD_LIBRARY_PATH?

$ python --version
Python 3.10.13

$ python -c "import lxml.etree as etree;print(etree.LXML_VERSION);print(etree.LIBXML_VERSION);print(etree.LIBXML_COMPILED_VERSION);"
(4, 9, 3, 0)
(2, 12, 4)
(2, 12, 4)

@vinodpandey
Copy link

I am using https://github.com/pyenv/pyenv for installing multiple python versions. Both Python 3.10.13 and Python 3.10.12 were installed using pyenv instead of brew.

I have not set the DYLD_LIBRARY_PATH. I checked in pyenv installed Python 3.10.13 and found that I get the mismatched version after forcing DYLD_LIBRARY_PATH

python -V
Python 3.10.13

python -c "import lxml.etree as etree;print(etree.LXML_VERSION);print(etree.LIBXML_VERSION);print(etree.LIBXML_COMPILED_VERSION);"
(4, 9, 3, 0)
(2, 12, 4)
(2, 12, 4)


DYLD_LIBRARY_PATH=/usr/lib python -c "import lxml.etree as etree;print(etree.LXML_VERSION);print(etree.LIBXML_VERSION);print(etree.LIBXML_COMPILED_VERSION);"
(4, 9, 3, 0)
(2, 9, 13)
(2, 12, 4)

@nosnilmot
Copy link

I cannot reproduce. Perhaps reinstall (pyenv install --force 3.10.12) if you did not do so after you re-installed brew ?

$ pyenv install 3.10.12
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.10.12.tar.xz...
-> https://www.python.org/ftp/python/3.10.12/Python-3.10.12.tar.xz
Installing Python-3.10.12...
python-build: use tcl-tk from homebrew
python-build: use readline from homebrew
python-build: use ncurses from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.10.12 to /Users/stu/.pyenv/versions/3.10.12

$ python --version
Python 3.10.12

$ python -m venv venv.pyenv
$ . venv.pyenv/bin/activate
$ pip install --no-cache-dir --no-binary lxml lxml==4.9.3
...

$ python -c "import lxml.etree as etree;print(etree.LXML_VERSION);print(etree.LIBXML_VERSION);print(etree.LIBXML_COMPILED_VERSION);"
(4, 9, 3, 0)
(2, 12, 4)
(2, 12, 4)

@vinodpandey
Copy link

I used Mac Migration Assistant software to migrate from Macbook Pro x86 to Macbook M2. This might be the reason why my setup is working weirdly as the architecture changed from x86 to ARM. Below is my output after pyenv install --force 3.10.12

pyenv install --force 3.10.12

python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.10.12.tar.xz...
-> https://www.python.org/ftp/python/3.10.12/Python-3.10.12.tar.xz
Installing Python-3.10.12...
python-build: use readline from homebrew
python-build: use ncurses from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.10.12 to /Users/vinodpandey/.pyenv/versions/3.10.12

python --version
Python 3.10.12

python -m venv venv.pyenv

. venv.pyenv/bin/activate
pip install --no-cache-dir --no-binary lxml lxml==4.9.3

python -c "import lxml.etree as etree;print(etree.LXML_VERSION);print(etree.LIBXML_VERSION);print(etree.LIBXML_COMPILED_VERSION);"
(4, 9, 3, 0)
(2, 9, 13)
(2, 9, 13)

@nosnilmot
Copy link

I think you forgot to set PKG_CONFIG_PATH="/opt/homebrew/opt/libxml2/lib/pkgconfig" before installing lxml this time (sorry I already had it set in my env when I ran the above).

@vinodpandey
Copy link

Thanks a lot @nosnilmot. It's working correctly for me. I can sleep better now 😄.

Below is the combination that worked for me. Without LDFLAGS and CPPFLAGS I was still getting the mismatched version.


$ export LDFLAGS="-L/opt/homebrew/opt/libxml2/lib"
$ export CPPFLAGS="-I/opt/homebrew/opt/libxml2/include"
$ export PKG_CONFIG_PATH="/opt/homebrew/opt/libxml2/lib/pkgconfig"

$ pyenv install --force 3.10.12

python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.10.12.tar.xz...
-> https://www.python.org/ftp/python/3.10.12/Python-3.10.12.tar.xz
Installing Python-3.10.12...
python-build: use readline from homebrew
python-build: use ncurses from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.10.12 to /Users/vinodpandey/.pyenv/versions/3.10.12

$ pyenv local 3.10.12

$ python --version
Python 3.10.12

$ python -m venv venv.pyenv

$ . venv.pyenv/bin/activate
$ pip install --no-cache-dir --no-binary lxml lxml==4.9.3

$ python -c "import lxml.etree as etree;print(etree.LXML_VERSION);print(etree.LIBXML_VERSION);print(etree.LIBXML_COMPILED_VERSION);"

(4, 9, 3, 0)
(2, 12, 4)
(2, 12, 4)

@ryanhiebert
Copy link

python3-saml depends on xmlsec and lxml. Because xmlsec also depends on lxml, this feels like a problem that xmlsec should resolve. It seems that there is some activity on the xmlsec repo toward this: xmlsec/python-xmlsec#283, though it's not clear whether xmlsec is maintained well enough to address this or whether it will need to be forked.

@banagale
Copy link

This is not an allauth issue, but I also needed the saml extra on allauth and ran into issues on an Apple Silicon mac building xmlsec1. Some notes for the future weary traveler:

  1. brew install xmlsec1 won't do it, at least not right now.
  2. This comment has a sequence that worked for me:
export DESIRED_SHA="7f35e6ede954326a10949891af2dba47bbe1fc17"
wget -O /tmp/libxmlsec1.rb "https://raw.githubusercontent.com/Homebrew/homebrew-core/${DESIRED_SHA}/Formula/libxmlsec1.rb"
brew install --formula /tmp/libxmlsec1.rb

Then, poetry update succeeds even if it includes this line:

django-allauth = { version = "^0.61.1", extras = ["saml"] }

@vdboor
Copy link

vdboor commented Apr 22, 2024

Looks like xmlsec 1.3.14 addressed this issue!

@banagale
Copy link

banagale commented Apr 22, 2024

Looks like xmlsec 1.3.14 addressed this issue!

Will try to validate this, thanks for the heads-up, @vdboor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants