Mac OSX, Python: import klayout.dbcore, missing libpng16.16.dylib #1265
Comments
|
Hi @lukasc-ubc, I think (1) It looks like Homebrew's libpng16.16.dylib is being loaded. But your current environment is Anaconda3. In my Anaconda3 environment on Monterey...(2) My "pymod" is homemade, not from PyPI. However, I believe there is no difference. Let's see the site-packages. (3) This "libpng16.16.dylib" is Anaconda3's, which was installed by default. Regards, |
In the Homebrew environment... |
In the MacPorts environment... |
|
Oh, I understand! Again in the Anaconda3 environmentIt seems PyPI's "pymod" was built with Homebrew.
The above statement was incorrect. |
|
If you use "LW-klayout-0.28.3-macOS-Monterey-1-qt5Ana3-Rana3Pana3.dmg", reinstall the homemade "pymod" as follows. I recalled how I find the <png.h> and <libpng*> to build "pymod" in the three environments.
|
|
I have no real idea how to package additional files. The PyPI packages are built using github actions. The builds depend on pypa/cibuildwheel (https://github.com/pypa/cibuildwheel) which itself employs auditwheel or delocate which (to my understanding) should add all missing files to the package. So either the problem is that something is broken there or the github hosts already have libpng and delocate does not have a reason to add these files. Either way, I'm lost. Maybe someone else can help. Matthias |
|
Hi folks, Thanks for your thoughts. I realized that the HW and LW packages are for the GUI build, and pip import is a different beast all together that uses the local Python installation (rather than shipping with Python like the HW version). I was able to solve it by installing libpng using brew. In my case, it took more effort, including upgrading brew and installaing libpng: thank you |
|
Hi folks, same issue here. However my homebrew installs libpng here /opt/homebrew/lib/libpng16.16.dylibmeaning, that I cannot rectify the issue the way that @lukasc-ubc proposed |
|
Installing the homemade pymod works. It is however a major inconvenience to have to do that on every install e.g. in virtual environments. |
|
Bad news, but I can only emphasize, that I have no clue how to fix it. This needs someone with a deeper knowledge about packaging wheels. The details are hidden far too deep in the guts of these CI "helpers". I'm personally tired of digging into that stuff. I can do so, but that would put an end to KLayout feature development. No time left. |
|
This seems to be something related to where homebrew install libpng on m1. We are currently also running into this problem (I am running linux but colleagues are using M1 or newer). I requested an M1 from my IT and should get it soon:tm:. On suggestion that I was thinking about and that probably would make sense is to instead of relying on the dynamic library, compile libpng into the executable (probably makes it bigger, but might be less annoying). @Hsuan-Tung could you try this as long as we don't have a testing machine yet? |
|
When using conda environment to install, it works for me. I am using M1 mac. Here is what I did, But directly doing |
|
Hi folks, thank you for the help. I have completely removed Anaconda from my system, and installed Python using brew and pyenv, and the problem is fixed. |
|
That is interesting. Sorry for not being more constructive. Do you feel able to summarize the analysis for me? I am not really familiar with the ways HomeBrew, Anaconda and the system libraries interact. I'm simply a Linux guy. Thanks, Matthias |
|
From Hsuan-Tung's machine, I am pretty sure on m1 homebrew puts the libraries in a separate path (for the arm64 part, works fine if you use x86_64 brew and rely on rosetta doing work for you) which is not in PATH (into I tried yesterday with Hsuan-Tung to compile the library into klayout, but it doesn't seem straight forward. In conda it seems to work if you link against the library during compilation manually. |
|
cc |
|
Thanks for tagging me. I have a macOS M1 Ventura 13.0.1 and building it doesn't work for me because there is no libpng in my machine, so I can't include |
|
That's my understanding. libpng on M1 when installed is not installed to somewhere , where PATH (or ots equivalents) can find it. By manually linking as in Hsuan-Tung's description works (at least for conda). You can add it manually for other environments, but then it crashes while importing and not on compile |
|
So Lukas's first installation was on x64, whereas yours was on M1. the macOS environment in GitHub actions seems to have curl, expat and png libraries. Maybe no longer the case in the newest macOS. I just confirmed I could build klayout without png on my Mac. I think that python klayout without png would fail at loading png files. "tlPixelBuffer.cc". @klayoutmatthias was png always a dependency? I don't recall. I propose editing setup.py to search for homebrew libpng, and if it can't find it, install a version without libpng. Do you agree? |
|
Did some investigation. Matplotlib used to have libpng as a dependency as well. Here's how they checked and obtained flags for compilation: https://github.com/matplotlib/matplotlib/blob/73f6ac0bec7cb3845253adce82286530173f3a86/setupext.py#L653 However, in more recent versions they dropped the dependency and used PIL instead. Last version they used it was 3.2.x. So I downloaded their wheel from pypi and checked the linked libraries in their $ otool -L _png.cpython-38-darwin.so
_png.cpython-38-darwin.so:
@loader_path/.dylibs/libpng16.16.dylib (compatibility version 38.0.0, current version 38.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
$ ls .dylibs
libpng16.16.dylib* libz.1.2.10.dylib*As you can see, they simply shipped libpng and its dependency libz together with their macOS wheel. So it can be done but it may be a PITA to maintain. @klayoutmatthias we have two options: 1. disable libpng for the python package; or 2. ship libpng with the wheel AND edit the source install so it searches a homebrew version, fail if user does not have libpng installed (like in matplotlib's case). The libpng linking flags are |
|
Libpng is used for the new lay submodule afaik (as of 0.28). Though db now also fails without it. But I guess it is used for the screenshot and pixelbuffers. I'd like to keep it because it enables us to do awesome stuff directly in python, such as the klayout integration of gdsfactory into a jupyter notebooks |
|
@sebastian-goeldi , can you provide a script example of a use case of the screenshot/pixelbuffer? I would like to add it as a test in the suite of tests. |
|
@Hsuan-Tung @sebastian-goeldi I also fixed @lukasc-ubc 's original issue where the wheel was shipped with a hardcode link to home-brew's libpng. Now future macOS wheels will have libpng.dylib embedded. |
|
Thank you!
|
|
@sebastian-goeldi is this what you are referring to? https://github.com/gdsfactory/gdsfactory/blob/b1d857446e1918b6f3db5b1cce3ba4de165652a5/gdsfactory/widgets/layout_viewer.py#L14 Very cool. Would be nice to provide a minimal example so I can add to the test suite. |
|
@thomaslima sorry for the late reply. Had quite the busy week last week. Kinda. This is the move fleshed out way we are trying to use it https://github.com/gdsfactory/kweb . But yes, the widget is probably better. kweb is more based on Matthias' canvas2canvas project It's a prototype that can be fully loaded in jupyter. But yes, a widget is probably the better implementation. Let me make a minimal example that will only write a png, all the other stuff is added sugar that won't be necessary for the test |
|
Thanks @thomaslima ! I just tested it and it worked on my M1 Mac as well. |
|
@thomaslima minimal example: import klayout.db as kdb
import klayout.lay as lay
#### create waveguide gds
# this wouldn't need to be done by the test, this is just for show casing
# it is probably better to actually make this a separate file, so that it can be recreated,
# but the test doesn't create a false positive if e.g. the waveguide isn't drawn in the gds
layout = kdb.Layout()
cell = layout.create_cell("waveguide")
cell.shapes(layout.layer(1, 0)).insert(kdb.Box(0, -500, 10000, 500))
cell.write("wg.gds")
### load the gds through a layoutview object
lv = lay.LayoutView()
lv.load_layout("wg.gds")
lv.resize(400, 400)
lv.max_hier()
pb = lv.get_screenshot_pixels()
pb.write_png("wg.png") |
|
Many thanks for these contributions here! I really appreciate this. I'm sorry - I had some hardware+tapeout trouble, so I was not able to follow here. I merged the two PRs and the solution will go into the next 0.28.6 PyPI module/DMGs. I think that will something like in two weeks. Will that be good enough? Thanks and best regards, Matthias |
|
I believe this issue has been fully addressed by now. libpng is shipped with python wheels. the app uses Qt's PNG engine, so it doesn't depend on libpng. Installing klayout from source in linux and macOS will require libpng libraries to be installed (it uses the |
@Kazzz-S
The KLayout Python package is referencing a library "libpng16.16.dylib" that is not installed on my system. I understand that it can be installed by brew. Perhaps this library could be bundled as part of the "pip install" klayout package?
Thank you
The text was updated successfully, but these errors were encountered: