Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MacOS Big Sur dlsym(RTLD_DEFAULT, kIOMasterPortDefault): symbol not found (ESPTOOL-82) #540

Closed
jeksys opened this issue Jun 25, 2020 · 94 comments

Comments

@jeksys
Copy link

jeksys commented Jun 25, 2020

  • Operating system: MacOS Big Sur
  • Python version: Python 3.8.1

Full esptool.py command line as run:

esptool.py chip_id

Full output from esptool.py

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/esptool.py", line 57, in <module>
    import serial.tools.list_ports as list_ports
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/serial/tools/list_ports.py", line 29, in <module>
    from serial.tools.list_ports_posix import comports
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/serial/tools/list_ports_posix.py", line 31, in <module>
    from serial.tools.list_ports_osx import comports
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/serial/tools/list_ports_osx.py", line 32, in <module>
    kIOMasterPortDefault = ctypes.c_void_p.in_dll(iokit, "kIOMasterPortDefault")
ValueError: dlsym(RTLD_DEFAULT, kIOMasterPortDefault): symbol not found

What is the expected behaviour?

Connected device info

Do you have any other information from investigating this?

I worked before the MacOS upgrade

Is there any other information you can think of which will help us reproduce this problem?

@jeksys jeksys added the bug label Jun 25, 2020
@VladislavSmolyanoy
Copy link

Same issue here!

@projectgus
Copy link
Contributor

Hi @jeksys @SwiftySquirrel ,

Thanks for letting us know about this.

esptool relies on pyserial for serial port functionality, and this is where the exception is coming from. Ultimately whatever fix is required for MacOS Big Sur support will need to be applied there and then esptool will hopefully start working again.

Until then, the best we can do in esptool is apply a workaround so it's still possible to use esptool with a specific port specified on the command line, in the case that importing this utility function fails.

@jeksys
Copy link
Author

jeksys commented Jun 26, 2020

@projectgus
Thank you for the suggestion. I think the additional parameter will work for some cases.
Though I think Arduio studio will still fail.
When do you think it is possible to make a release with a fix?

I reported the issue to pyserial
Though I think Arduio studio will still fail.

@ehagerty
Copy link

same issue here. forgive me if I'm being thick, but just putting --port /dev/tty.usbserial-xxxxxxxx flash-id for example gives the same error. did I misunderstand the suggestion of forcing the port parameter? Thanks so much, I should know better than to upgrade my primary machine!

@projectgus
Copy link
Contributor

Yes, sorry I didn't explain clearly. The workaround is a potential one, it still involves a code change in esptool.

As a very quick hack you can try commenting this line in esptool.py:

  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/esptool.py", line 57, in <module>
    import serial.tools.list_ports as list_ports

... and then always use the --port argument

...

I'm trying to get my hands on a Big Sur Preview install to do some more debugging.

@xrolfex
Copy link

xrolfex commented Jun 27, 2020

@projectgus I can confirm that if you comment out the try: import block per your suggestion above it does work on Big Sur.

@VladislavSmolyanoy
Copy link

@xrolfex It doesn't for me...

@XNinety9
Copy link

Same issue here when using esptool through PlatformIO:
image

@xrolfex
Copy link

xrolfex commented Jun 28, 2020

@p-vernaeckt Are you explicitly using the —port option after commenting out the four lines?

image

@ehagerty
Copy link

ehagerty commented Jun 30, 2020

Hi, sorry for the delay, can confirm:

OS11 Beta (20A4299v)
pyenv 1.2.19
pyenv-virtualenv 1.1.5
python 3.8.2
micropython esp32-idf4-20191220-v1.12.bin
esp32 (huzzah32 esp32-wroom32)

commenting out lines 56-61 of esptool.py and using the explicit --port works. Thank you!

@JuanEL0
Copy link

JuanEL0 commented Jul 4, 2020

Hi, I have the same issue on my mac with BigSur.

I'm using Arduino IDE and when I try to verify my code, it shows the same message.

I am totally new in this world and don't know how to configure the port from the Arduino IDE so I can not verify and load my code into my ESP8266.

If somebody has any idea about how to do it, I would appreciate it.

@jeksys
Copy link
Author

jeksys commented Jul 8, 2020

The suggested fix worked for me.

  1. Comment below lines in the file "/Library/Frameworks/Python.framework/Versions/3.8/bin/esptool.py"
#try:
#    import serial.tools.list_ports as list_ports
#except ImportError:
#    print("The installed version (%s) of pyserial appears to be too old for esptool.py (Python interpreter %s). "
#          "Check the README for installation instructions." % (sys.VERSION, sys.executable))
#    raise
  1. List all available ports
    ls /dev/tty.*

  2. Use esptool.py with --port command
    esptool.py --port /dev/tty.usbserial-01EED907 --baud 921600 erase_flash

  3. Update esptool.py or env variable with a default port so esptool.py can be called without a parameter.
    TODO

@hpwit
Copy link

hpwit commented Jul 26, 2020

A have a question which is linked to that I have python 3.7.7 installed and that is my current python but esptools via Arduino keep on trying to get to 2.7
File "esptool.py", line 57, in
File "/Library/Python/2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
how to change that
i have reinstalled Arduino and also es-tools

@ehagerty
Copy link

NB, in case anyone is interested, this works for the v3.0-dev version of esptool.py in case you are using that with your s2 modules...

@shurshun
Copy link

NB, in case anyone is interested, this works for the v3.0-dev version of esptool.py in case you are using that with your s2 modules...

I guess we just have to find v3.0-dev version of esptool.py... Could you provide a link to this version, please?

@acmdorio
Copy link

Yes, sorry I didn't explain clearly. The workaround is a potential one, it still involves a code change in esptool.

As a very quick hack you can try commenting this line in esptool.py:

  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/esptool.py", line 57, in <module>
    import serial.tools.list_ports as list_ports

... and then always use the --port argument

...

I'm trying to get my hands on a Big Sur Preview install to do some more debugging.

@projectgus Thanks, it works for me on BS 11.0 Beta (20A5354i)! By the way, a big thanks for your teamwork on this tool!
Maybe you should put this in esptool.py code as a test plus a warning before using the line 57, so other users barely note this issue. And brew update, then upgrade keep things working flawlessly. Just a humble idea. ;)

antmak pushed a commit that referenced this issue Aug 27, 2020
…view

References:
#540
pyserial/pyserial#509

The fix will probably come in a Python update before the Big Sur final release,
there is a bundle of PRs including an apparent fix for the underlying issue here:
https://bugs.python.org/issue41100
@projectgus
Copy link
Contributor

projectgus commented Aug 27, 2020

Commit cb2060c is a workaround for this issue, so that esptool.py can run (without being able to detect the port). If you pull the latest master branch version of esptool.py (or follow the instructions in the README to install in "Development Mode" via GitHub) then no more modifications of the code should be required

It looks like the actual fix will come in a Python update once this issue is resolved: https://bugs.python.org/issue41100

Will keep this issue open, at least until the upstream issue is resolved.

@R0dri
Copy link

R0dri commented Aug 27, 2020

@projectgus Thankyou for the workaround. I can confirm esptool.py chip_id now catches the error instead of an abrupt termination of the process.

I've been trying to implement this to use arduino-cli or Arduinos own IDE with no luck. Inspecting the JSON board definition files i found arduino tries to compile the code not with python+esptool.py but with a binary stored in $arduino_packages/esp32/esptool_py/2.6.1/esptool which I guess its a compilation out of esptool.py.

I have never used esptool standalone and I don't know where to go next. Of course, I know I could wait for the actual python fix and keep using my raspberry pi for now. But I would love to learn a little more in the area. What would you recommend?

@projectgus
Copy link
Contributor

projectgus commented Aug 27, 2020

@R0dri I'm not actually familiar with arduino-esp32 enough to give you an exact answer. However if you can get esptool.py master to run locally (by following the steps in the README for a development install) then you can probably update the Arduino platform.txt file here to specify a new path and command to run:
https://github.com/espressif/arduino-esp32/blob/ed96d2a1b70ea02d925435d06b71617bd451ffec/platform.txt#L6

@exrector
Copy link

exrector commented Sep 4, 2020

Hey. I tried to apply a workaround and I got an error. What advice can you give? 😭

bash-3.2$ git clone https://github.com/espressif/esptool.git xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun bash-3.2$

@petykowski
Copy link

Hey. I tried to apply a workaround and I got an error. What advice can you give? 😭

`bash-3.2$ git clone https://github.com/espressif/esptool.git

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

bash-3.2$ `

This message is unrelated to the fix provided by Expressif. This error indicates that you need to update to the latest Xcode developer tools build.

@exrector
Copy link

exrector commented Sep 4, 2020

Xcode
Thank you. But I have checked the versions and I have the latest 11.7.
Big Sur beta 11.0 (20A5364e)
Any variants? 😇
PS
I will try to install and update additional tools and Command Line Tools

@petykowski
Copy link

petykowski commented Sep 5, 2020

@exrector I am running the same configuration as described in your comment. Funny enough, I was experiencing error when attempting to commit to an unrelated repo. To resolve the issue, you should download the Apple Command Line Tools via the Developer Downloads portal, or via the direct link to the latest Xcode 12 tools here.

@AndreasRavn
Copy link

AndreasRavn commented Sep 8, 2020

I've been trying to implement this to use arduino-cli or Arduinos own IDE with no luck. Inspecting the JSON board definition files i found arduino tries to compile the code not with python+esptool.py but with a binary stored in $arduino_packages/esp32/esptool_py/2.6.1/esptool which I guess its a compilation out of esptool.py.

I have never used esptool standalone and I don't know where to go next. Of course, I know I could wait for the actual python fix and keep using my raspberry pi for now. But I would love to learn a little more in the area. What would you recommend?

I changed line 6 and 7 in platform.txt from

tools.esptool_py.path={runtime.platform.path}/tools/esptool
tools.esptool_py.cmd=esptool

to

tools.esptool_py.path={runtime.platform.path}/tools
tools.esptool_py.cmd=esptool.py

that did the trick for me, the arduino IDE is now using esptool.py instead of the compiled binary in the subfolder.

@dobairoland
Copy link
Collaborator

@evoegelin All of your issues are Arduino related, right? You don't experience these with ESP-IDF directly?

The Big Sur workaround is available in the esptool's release/v2 branch as well.

@evoegelin
Copy link

evoegelin commented Dec 7, 2020

@evoegelin All of your issues are Arduino related, right? You don't experience these with ESP-IDF directly?

The Big Sur workaround is available in the esptool's release/v2 branch as well.

Yes this all is only Arduino IDE related on a Mac after update to Big Sur.
With the exception of the SiLabs 210x USB-UART Driver, this is a general bug to Big Sur.

@dobairoland
Copy link
Collaborator

Yes this all is only Arduino IDE related on a Mac after update to Big Sur.

In this case I recommend to raise an issue there (if there isn't one already) and request to update to newer esptool.

With the exception of the SiLabs 210x USB-UART Driver, this is a general bug to Big Sur.

Unfortunately, we need to wait with this for a solution in Big Sur.

@me-no-dev
Copy link
Member

I released 1.0.5-rc4 today that includes a rebuilt esptool for Mac. Make sure you uninstall the core first (esptool version is the same so it needs to first wipe all)

@dobairoland
Copy link
Collaborator

Great! Thank you @me-no-dev.

@Elijahg
Copy link

Elijahg commented Dec 7, 2020

With the exception of the SiLabs 210x USB-UART Driver, this is a general bug to Big Sur.

I have a Heltec ESP32 dev board and it has a CP2102 UART which works fine with Big Sur. Installing the Silicon Labs drier may well break loading of the Apple SILabs driver as it did with a FTDI FT2232 I was using. So probably best to uninstall all traces of the SILabs kexts (could be in /Library/StagedExtensions or /Library/Extensions) and only use the Apple driver.

image

@evoegelin
Copy link

evoegelin commented Dec 8, 2020

Thank you @me-no-dev.

I erased Arduino from my Mac and installed it new from the scratch. It's working properly. Thank you.
And using the Apple Driver for Hellten boards works fine too.

Thank you all.

@github-actions github-actions bot changed the title MacOS Big Sur dlsym(RTLD_DEFAULT, kIOMasterPortDefault): symbol not found MacOS Big Sur dlsym(RTLD_DEFAULT, kIOMasterPortDefault): symbol not found (ESPTOOL-82) Dec 8, 2020
@omski
Copy link

omski commented Dec 8, 2020

If anyone is still struggling to get pyserial updated on platformio. I found that I need to update it within the platformio python env itself (hint as per Arduino comments above). Once that was done it compiled again.

$ cd /Users/qoda/.platformio/
$ . penv/bin/activate
(penv) $ which pip
/Users/qoda/.platformio/penv/bin/pip
(penv) $ /Users/qoda/.platformio/penv/bin/pip install -U --pre pyserial
Collecting pyserial
  Using cached pyserial-3.5-py2.py3-none-any.whl (90 kB)
Installing collected packages: pyserial
  Attempting uninstall: pyserial
    Found existing installation: pyserial 3.4
    Uninstalling pyserial-3.4:
      Successfully uninstalled pyserial-3.4
Successfully installed pyserial-3.5

This solved it for me on MacOs Big Sur with VSCode and Platform.io. No need to edit any line of python code anymore.

@dobairoland
Copy link
Collaborator

Thank you all for the responses. I'm closing this now because I consider it solved as I described before. Feel free to ping me if you disagree.

@chris-schmitz
Copy link

Thanks @me-no-dev, the 1.0.5-rc4 update got this working for my mac running big sur 🙇🏽

Something worth mentioning for other people that run into this issue is that to get this update (as of the typing of this comment) you need to use the Development Release json file instead of the Stable Release json. You can find details in the install readme.

https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_dev_index.json

I was super confused at firwst b/c I only saw 1.0.4. A bit of searching/reading set me straight :)

@tenapier
Copy link

tenapier commented Dec 19, 2020 via email

@Lilanga
Copy link

Lilanga commented Jan 4, 2021

Thank you @chris-schmitz. Dev channel board worked. still having the 1.0.5-rc4 as the latest.

@chris-schmitz
Copy link

Thank you Christopher Schmitz! First time in weeks now, that I can once
again connect with my ESP32 and Mac, using Big Sur! I had tried a number of
code workaround solutions, and nothing was working, until you put out the
Board Manager Solution:
Thanks so much, it's great to get reconnected again!

https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_dev_index.json

On Wed, Dec 16, 2020 at 9:44 PM Christopher Schmitz <
notifications@github.com> wrote:

Thanks @me-no-dev https://github.com/me-no-dev, the 1.0.5-rc4 update
got this working for my mac running big sur 🙇🏽

Something worth mentioning for other people that run into this issue is
that to get this update (as of the typing of this comment) you need to use
the Development Release json file instead of the Stable Release json.
You can find details in the install readme
https://github.com/espressif/arduino-esp32/blob/master/docs/arduino-ide/boards_manager.md
.

https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_dev_index.json

I was super confused at firwst b/c I only saw 1.0.4. A bit of
searching/reading set me straight :)


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#540 (comment),
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJO4L5KPUVANJDNOK2J7U7DSVF5BZANCNFSM4OH3QGIA
.

No prob :) I didn't put out the release, just linked to it after I found it.

@philolo1
Copy link

For everyone still reading this issue, this worked for me

pip install --upgrade  pyserial 

@kurosh-z
Copy link

For everyone still reading this issue, this worked for me

pip install --upgrade  pyserial 

Hi
I have still problem although I have upgraded the pyserial. I'm still getting this error:

Traceback (most recent call last):
  File "esptool.py", line 57, in <module>
  File "/Library/Python/2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "serial/tools/list_ports.py", line 29, in <module>
  File "/Library/Python/2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "serial/tools/list_ports_posix.py", line 31, in <module>
  File "/Library/Python/2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "serial/tools/list_ports_osx.py", line 32, in <module>
ValueError: dlsym(RTLD_DEFAULT, kIOMasterPortDefault): symbol not found
Failed to execute script esptool
exit status 255
/Applications/Arduino.app/Contents/Java/arduino-builder returned 255
Error compiling for board Wireless Stick.

any idea what could be causing this?

@electronic-cosmos
Copy link

WEMos and M5 is working but for DOIT and a lot of other boards bring folloing Massage:

esptool.py v2.9-dev
Serial port /dev/cu.usbserial-0001
Connecting......................................____Beim Hochladen des Sketches ist ein Fehler aufgetreten
_

A fatal error occurred: Failed to connect to ESP32: Timed out waiting for packet header

does anybody can tell me what is going wrong?

@bfaliszek
Copy link

@electronic-cosmos it looks like a physical connection error. Check that everything is properly connected

@josmanperez
Copy link

For everyone still reading this issue, this worked for me

pip install --upgrade  pyserial 

Hi
I have still problem although I have upgraded the pyserial. I'm still getting this error:

Traceback (most recent call last):
  File "esptool.py", line 57, in <module>
  File "/Library/Python/2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "serial/tools/list_ports.py", line 29, in <module>
  File "/Library/Python/2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "serial/tools/list_ports_posix.py", line 31, in <module>
  File "/Library/Python/2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "serial/tools/list_ports_osx.py", line 32, in <module>
ValueError: dlsym(RTLD_DEFAULT, kIOMasterPortDefault): symbol not found
Failed to execute script esptool
exit status 255
/Applications/Arduino.app/Contents/Java/arduino-builder returned 255
Error compiling for board Wireless Stick.

any idea what could be causing this?

This is happening for me as well, I haven't solved it yet
@kurosh-z , have you managed to solve it?

Thanks

@Elijahg
Copy link

Elijahg commented Apr 8, 2021

For everyone still reading this issue, this worked for me

pip install --upgrade  pyserial 

Hi
I have still problem although I have upgraded the pyserial. I'm still getting this error:

Traceback (most recent call last):
  File "esptool.py", line 57, in <module>
  File "/Library/Python/2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "serial/tools/list_ports.py", line 29, in <module>
  File "/Library/Python/2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "serial/tools/list_ports_posix.py", line 31, in <module>
  File "/Library/Python/2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "serial/tools/list_ports_osx.py", line 32, in <module>
ValueError: dlsym(RTLD_DEFAULT, kIOMasterPortDefault): symbol not found
Failed to execute script esptool
exit status 255
/Applications/Arduino.app/Contents/Java/arduino-builder returned 255
Error compiling for board Wireless Stick.

any idea what could be causing this?

This is happening for me as well, I haven't solved it yet
@kurosh-z , have you managed to solve it?

Thanks

Try this. You can probably just follow the replace esptool part.

@josmanperez
Copy link

Thanks @Elijahg . It worked!

@Imikbel
Copy link

Imikbel commented Nov 12, 2021

https://docs.python-guide.org/starting/install3/osx/
After that: pip install --upgrade pyserial
Problem solved ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests