Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

installation issues with Parsons 3.10 on M1 Mac #744

Closed
shaunagm opened this issue Sep 15, 2022 · 10 comments
Closed

installation issues with Parsons 3.10 on M1 Mac #744

shaunagm opened this issue Sep 15, 2022 · 10 comments
Labels
bug Impact - something is currently broken in Parsons and needs to be fixed low priority Priority - this is a nice-to-have, is non-urgent, and/or has a minor overall imapact on Parsons needs more info Status - the issue is not described in enough detail to act on right away

Comments

@shaunagm
Copy link
Collaborator

shaunagm commented Sep 15, 2022 •
edited

Parsons doesn't work with Python 3.10 (on M1 Mac and probably elsewhere). Specifically seems to be an issue with new directory structures in the Collections standard library in 3.10 conflicting with older dependencies. See this StackOverflow page. The packages we ran into trouble with were urllib3, botocore, and gspread.

To Reproduce

Using a Mac (or possibly other OS) and Python 3.10, try installing Parsons. It should install, but when trying to use it and importing anything from Parsons you'll get an issue like ImportError: cannot import name 'Mapping' from 'collections'.

Sorry I don't have a full stacktrace, but these are errors from dependencies, not a fix that can be made in Parsons itself. Found issues with urllib3, botocore and gspread before giving up and switching to Python 3.9.

Your Environment

Wasn't mine, but: Mac with M1 chip, using Python 3.10, Parsons 0.21.0.

Priority

High priority (or lower the priority but remove statements saying we support 3.10)
@shaunagm shaunagm added bug Impact - something is currently broken in Parsons and needs to be fixed high priority Priority - addressing this is an urgent need for a broad swath of Parsons users labels Sep 15, 2022
@alxmrs
Copy link
Contributor

alxmrs commented Sep 16, 2022

I'm interested in working on this.

@alxmrs
Copy link
Contributor

alxmrs commented Sep 16, 2022

I'm hitting an additional error on install. I'm on an M1 in a fresh conda environment with Python 3.10.4, and on pip install I get:

Stack Trace 
  Running setup.py install for lxml ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for lxml did not run successfully.
  │ exit code: 1
  ╰─> [88 lines of output]
      Building lxml version 4.9.1.
      Building without Cython.
      Building against libxml2 2.9.4 and libxslt 1.1.29
      running install
      /opt/homebrew/Caskroom/miniconda/base/envs/parsons310/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.9-x86_64-cpython-310
      creating build/lib.macosx-10.9-x86_64-cpython-310/lxml
      copying src/lxml/_elementpath.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml
      copying src/lxml/sax.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml
      copying src/lxml/pyclasslookup.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml
      copying src/lxml/__init__.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml
      copying src/lxml/builder.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml
      copying src/lxml/doctestcompare.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml
      copying src/lxml/usedoctest.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml
      copying src/lxml/cssselect.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml
      copying src/lxml/ElementInclude.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml
      creating build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/__init__.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      creating build/lib.macosx-10.9-x86_64-cpython-310/lxml/html
      copying src/lxml/html/soupparser.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/html
      copying src/lxml/html/defs.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/html
      copying src/lxml/html/_setmixin.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/html
      copying src/lxml/html/clean.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/html
      copying src/lxml/html/_diffcommand.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/html
      copying src/lxml/html/html5parser.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/html
      copying src/lxml/html/__init__.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/html
      copying src/lxml/html/formfill.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/html
      copying src/lxml/html/builder.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/html
      copying src/lxml/html/ElementSoup.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/html
      copying src/lxml/html/_html5builder.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/html
      copying src/lxml/html/usedoctest.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/html
      copying src/lxml/html/diff.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/html
      creating build/lib.macosx-10.9-x86_64-cpython-310/lxml/isoschematron
      copying src/lxml/isoschematron/__init__.py -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/isoschematron
      copying src/lxml/etree.h -> build/lib.macosx-10.9-x86_64-cpython-310/lxml
      copying src/lxml/etree_api.h -> build/lib.macosx-10.9-x86_64-cpython-310/lxml
      copying src/lxml/lxml.etree.h -> build/lib.macosx-10.9-x86_64-cpython-310/lxml
      copying src/lxml/lxml.etree_api.h -> build/lib.macosx-10.9-x86_64-cpython-310/lxml
      copying src/lxml/includes/xmlerror.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/c14n.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/xmlschema.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/__init__.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/schematron.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/tree.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/uri.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/etreepublic.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/xpath.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/htmlparser.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/xslt.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/config.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/xmlparser.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/xinclude.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/dtdvalid.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/relaxng.pxd -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/lxml-version.h -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      copying src/lxml/includes/etree_defs.h -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/includes
      creating build/lib.macosx-10.9-x86_64-cpython-310/lxml/isoschematron/resources
      creating build/lib.macosx-10.9-x86_64-cpython-310/lxml/isoschematron/resources/rng
      copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/isoschematron/resources/rng
      creating build/lib.macosx-10.9-x86_64-cpython-310/lxml/isoschematron/resources/xsl
      copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/isoschematron/resources/xsl
      copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/isoschematron/resources/xsl
      creating build/lib.macosx-10.9-x86_64-cpython-310/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.macosx-10.9-x86_64-cpython-310/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      running build_ext
      building 'lxml.etree' extension
      creating build/temp.macosx-10.9-x86_64-cpython-310
      creating build/temp.macosx-10.9-x86_64-cpython-310/src
      creating build/temp.macosx-10.9-x86_64-cpython-310/src/lxml
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/homebrew/Caskroom/miniconda/base/envs/parsons310/include -fPIC -O2 -isystem /opt/homebrew/Caskroom/miniconda/base/envs/parsons310/include -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc/lxml/includes -I/opt/homebrew/Caskroom/miniconda/base/envs/parsons310/include/python3.10 -c src/lxml/etree.c -o build/temp.macosx-10.9-x86_64-cpython-310/src/lxml/etree.o -w -flat_namespace
      xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
      Compile failed: command '/usr/bin/clang' failed with exit code 1
      cc -I/usr/include/libxml2 -c /var/folders/h7/c4_scx2s3nbg1x36qln787mm0000gn/T/xmlXPathInit0n4q3w2c.c -o var/folders/h7/c4_scx2s3nbg1x36qln787mm0000gn/T/xmlXPathInit0n4q3w2c.o
      xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
      *********************************************************************************
      Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
      Perhaps try: xcode-select --install
      *********************************************************************************
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> lxml

This seems to be related to eerohele/sublime-lxml#11; I'll try the xcode-select --install suggestion to see if this gets me past the initial error.

@alxmrs
Copy link
Contributor

alxmrs commented Sep 16, 2022

xcode-select --install does seem to fix this issue.

@alxmrs
Copy link
Contributor

alxmrs commented Sep 16, 2022

I haven't been able to reproduce this error on my M1 mac. The steps I've take are:

  1. Cloned a fork of the repo (at version 0.21.0 / 1411927).
  2. Create a clean Python 3.10.4 environment with Anaconda
  3. Install xcode-select
  4. pip install -e . (which defaults to the locked versions in requirements.txt).
  5. Ran tests with pytest.

All tests ran without a hitch. In addition, to try to trigger the issues with botocore, I also ran the following in a REPL:

from parsons import S3

These worked fine for me, given the above steps.

@shaunagm
Copy link
Collaborator Author

Thanks Alex! I haven't been able to reproduce either. I'm meeting again with the person who discovered this issue to see if we can reproduce on her machine, but for now it's best to hold off. Luckily we have 90+ other issues to work on. 😂 LMK if you'd like to work on something else but need help finding something.

Thanks for info on the other error, I will update our installation guide!

@alxmrs
Copy link
Contributor

alxmrs commented Sep 17, 2022

Thanks Shauna! I will look around for other issues.

@shaunagm
Copy link
Collaborator Author

Okay, update. The specific version that's causing errors is 3.10.6. Here's a stacktrace of the error - parsons installs fine but when you try to import it in the interpreter, you get this:

>>> from parsons import Table

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/Users/username/temp-venv/lib/python3.10/site-packages/parsons/__init__.py", line 14, in <module>
  from parsons.databases.redshift.redshift import Redshift
 File "/Users/username/temp-venv/lib/python3.10/site-packages/parsons/databases/redshift/__init__.py", line 1, in <module>
  from parsons.databases.redshift.redshift import Redshift
 File "/Users/username/temp-venv/lib/python3.10/site-packages/parsons/databases/redshift/redshift.py", line 2, in <module>
  from parsons.databases.redshift.rs_copy_table import RedshiftCopyTable
 File "/Users/username/temp-venv/lib/python3.10/site-packages/parsons/databases/redshift/rs_copy_table.py", line 2, in <module>
  from parsons.aws.s3 import S3
 File "/Users/username/temp-venv/lib/python3.10/site-packages/parsons/aws/__init__.py", line 1, in <module>
  from parsons.aws.s3 import S3
 File "/Users/username/temp-venv/lib/python3.10/site-packages/parsons/aws/s3.py", line 2, in <module>
  import boto3
 File "/Users/username/temp-venv/lib/python3.10/site-packages/boto3/__init__.py", line 16, in <module>
  from boto3.session import Session
 File "/Users/username/temp-venv/lib/python3.10/site-packages/boto3/session.py", line 17, in <module>
  import botocore.session
 File "/Users/username/temp-venv/lib/python3.10/site-packages/botocore/session.py", line 29, in <module>
  import botocore.configloader
 File "/Users/username/temp-venv/lib/python3.10/site-packages/botocore/configloader.py", line 19, in <module>
  from botocore.compat import six
 File "/Users/username/temp-venv/lib/python3.10/site-packages/botocore/compat.py", line 25, in <module>
  from botocore.exceptions import MD5UnavailableError
 File "/Users/username/temp-venv/lib/python3.10/site-packages/botocore/exceptions.py", line 15, in <module>
  from botocore.vendored import requests
 File "/Users/username/temp-venv/lib/python3.10/site-packages/botocore/vendored/requests/__init__.py", line 58, in <module>
  from . import utils
 File "/Users/username/temp-venv/lib/python3.10/site-packages/botocore/vendored/requests/utils.py", line 26, in <module>
  from .compat import parse_http_list as _parse_list_header
 File "/Users/username/temp-venv/lib/python3.10/site-packages/botocore/vendored/requests/compat.py", line 7, in <module>
  from .packages import chardet
 File "/Users/username/temp-venv/lib/python3.10/site-packages/botocore/vendored/requests/packages/__init__.py", line 3, in <module>
  from . import urllib3
 File "/Users/username/temp-venv/lib/python3.10/site-packages/botocore/vendored/requests/packages/urllib3/__init__.py", line 10, in <module>
  from .connectionpool import (
 File "/Users/username/temp-venv/lib/python3.10/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 38, in <module>
  from .response import HTTPResponse
 File "/Users/username/temp-venv/lib/python3.10/site-packages/botocore/vendored/requests/packages/urllib3/response.py", line 9, in <module>
  from ._collections import HTTPHeaderDict
 File "/Users/username/temp-venv/lib/python3.10/site-packages/botocore/vendored/requests/packages/urllib3/_collections.py", line 1, in <module>
  from collections import Mapping, MutableMapping
ImportError: cannot import name 'Mapping' from 'collections' (/opt/homebrew/Cellar/python@3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/collections/__init__.py)

Installing the most recent version of botocore fixes it, but then there's an error in gspread, etc.

@alxmrs
Copy link
Contributor

alxmrs commented Sep 22, 2022

I'm not able to reproduce on my machine. I've used two different ways to create a Python 3.10.6 environment (one via conda / conda-forge, another via homebrew + venv), and each time the library works as expected.

Do you have steps to reproduce? What is the environment like in which this error occurs?

parsons-3_10_6-via-brew

parsons-3_10_6-via-conda-forge

@alxmrs
Copy link
Contributor

alxmrs commented Sep 22, 2022

Here are the commands I've run for the above two methods:

conda

  1. conda create -n parsons3106 python=3.10.6 -c conda-forge
  2. conda activate parsons3106
  3. git checkout main
  4. git pull upstream main
  5. pip install -e .

homebrew + venv

  1. conda deactivate (as many times as needed to get to base python)
  2. brew install python@3.10
  3. python3 -m venv parsons310
  4. source ./parsons310/bin/activate
  5. pip install -e .

@shaunagm
Copy link
Collaborator Author

I'm going to close this as not reproducible, and open it again if anyone else runs into it. The issue is on someone else's machine and I don't want to keep bothering them to do debugging sessions when we've already figured out a workaround for her machine.

@shaunagm shaunagm added low priority Priority - this is a nice-to-have, is non-urgent, and/or has a minor overall imapact on Parsons needs more info Status - the issue is not described in enough detail to act on right away and removed high priority Priority - addressing this is an urgent need for a broad swath of Parsons users labels Sep 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Impact - something is currently broken in Parsons and needs to be fixed low priority Priority - this is a nice-to-have, is non-urgent, and/or has a minor overall imapact on Parsons needs more info Status - the issue is not described in enough detail to act on right away
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@shaunagm @alxmrs