Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix linking to editable install on Windows #25

Merged
merged 3 commits into from
May 2, 2023
Merged

Fix linking to editable install on Windows #25

merged 3 commits into from
May 2, 2023

Conversation

navytux
Copy link
Contributor

@navytux navytux commented Apr 26, 2023

Please see individual patches for details.

@navytux
example: Make mylib.h to be installed alongside dsodemo
2572330 
In the next patch we will need to use C-API provided by dsodemo.lib.demo
from external package.
@navytux
Fix linking to editable install on Windows
fca6f29 
On Windows during compile time it is not only target .dll that is needed
to link to a DSO, but also corresponding .lib and .exp files. And if the
.lib file cannot be found linking fails. On regular install for a DSO X we are
currently installing all X.dll, X.lib and X.exp, but for in-place builds we
copy only X.dll into intree without X.lib and the rest.

This leads to build failures when an external project tries to link to a DSO
from `pip install -e` installed project.  Below is, for example, how it looks
for the link failure of added example/project2 to dsodemo.lib.demo without the fix:

```console
(1.wenv) Z:\home\kirr\src\tools\go\pygo-win\setuptools_dso\example\project2>python setup.py build_ext -i
running build_ext
building 'use_dsodemo.ext' extension
Z:\home\kirr\src\tools\go\pygo-win\BuildTools\vc\tools\msvc\14.35.32215\bin\Hostx64\x64\cl.exe /c /nologo /Od /W3 /GL /DNDEBUG /MD -I../src -IZ:\ho
me\kirr\src\tools\go\pygo-win\1.wenv\include "-IC:\Program Files\Python310\include" "-IC:\Program Files\Python310\Include" -Iz:\home\kirr\src\tools
\go\pygo-win\BuildTools\vc\tools\msvc\14.35.32215\include -Iz:\home\kirr\src\tools\go\pygo-win\BuildTools\kits\10\include\10.0.22000.0\shared -Iz:\
home\kirr\src\tools\go\pygo-win\BuildTools\kits\10\include\10.0.22000.0\ucrt -Iz:\home\kirr\src\tools\go\pygo-win\BuildTools\kits\10\include\10.0.2
2000.0\um -Iz:\home\kirr\src\tools\go\pygo-win\BuildTools\kits\10\include\10.0.22000.0\winrt /EHsc /Tpsrc/use_dsodemo/ext.cpp /Fobuild\temp.win-amd
64-cpython-310\Release\src/use_dsodemo/ext.obj
ext.cpp

creating Z:\home\kirr\src\tools\go\pygo-win\setuptools_dso\example\project2\build\lib.win-amd64-cpython-310
creating Z:\home\kirr\src\tools\go\pygo-win\setuptools_dso\example\project2\build\lib.win-amd64-cpython-310\use_dsodemo
Z:\home\kirr\src\tools\go\pygo-win\BuildTools\vc\tools\msvc\14.35.32215\bin\Hostx64\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED
,ID=2 /MANIFESTUAC:NO /LIBPATH:Z:\home\kirr\src\tools\go\pygo-win\setuptools_dso\example\src\dsodemo\lib "/LIBPATH:C:\Program Files\Python310\libs"
 "/LIBPATH:C:\Program Files\Python310" /LIBPATH:z:\home\kirr\src\tools\go\pygo-win\BuildTools\vc\tools\msvc\14.35.32215\lib\x64 /LIBPATH:z:\home\ki
rr\src\tools\go\pygo-win\BuildTools\kits\10\lib\10.0.22000.0\ucrt\x64 /LIBPATH:z:\home\kirr\src\tools\go\pygo-win\BuildTools\kits\10\lib\10.0.22000
.0\um\x64 demo.lib /EXPORT:PyInit_ext build\temp.win-amd64-cpython-310\Release\src/use_dsodemo/ext.obj /OUT:build\lib.win-amd64-cpython-310\use_dso
demo\ext.cp310-win_amd64.pyd /IMPLIB:build\temp.win-amd64-cpython-310\Release\src/use_dsodemo\ext.cp310-win_amd64.lib

LINK : fatal error LNK1181: cannot open input file 'demo.lib'
error: command 'Z:\\home\\kirr\\src\\tools\\go\\pygo-win\\BuildTools\\vc\\tools\\msvc\\14.35.32215\\bin\\Hostx64\\x64\\link.exe' failed with exit code 1181
```

This fix is simple: on inplace build copy into intree not only .dll but
also .lib and .exp files.
@navytux
.gitignore += *.lib *.exp *.pyd
4ce8d01 
Those files are autogenerated on Windows builds.
@navytux
Copy link
Contributor Author

navytux commented Apr 26, 2023

(amended the main patch to also update testme.sh + small fix)

navytux added a commit to navytux/pygolang that referenced this pull request Apr 27, 2023
@navytux
.gitignore += *.lib *.exp
0b97e0a 
On Windows this files are generated when linking alongside *.dll files
if there are exported symbols.

See also mdavidsaver/setuptools_dso#25.
@mdavidsaver mdavidsaver merged commit 4ce8d01 into mdavidsaver:master May 2, 2023
61 checks passed
@mdavidsaver
Copy link
Owner

Thanks!

@navytux
Copy link
Contributor Author

navytux commented May 2, 2023

You are welcome and thanks for merging.

@navytux navytux deleted the y/win-inplace branch May 3, 2023 07:24
navytux added a commit to navytux/pygolang that referenced this pull request May 3, 2023
@navytux
pyx.build: v↑ setuptools_dso (2.7 -> 2.8)
31fc695 
This fixes linking to editable pygolang install on Windows.
See mdavidsaver/setuptools_dso@fca6f29e and
mdavidsaver/setuptools_dso#25 for details.

Without this fix e.g. test_pyx_build was also failing:

    platform win32 -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0 -- Z:\home\kirr\src\tools\go\pygo-win\1.wenv\Scripts\python.exe
    cachedir: .pytest_cache
    rootdir: Z:\home\kirr\src\tools\go\pygo-win\pygolang
    collecting 3 items
    collected 3 items

    golang/pyx/build_test.py::test_pyx_build running build_ext
    skipping 'pyxuser\test.cpp' Cython extension (up-to-date)
    building 'pyxuser.test' extension
    Z:\home\kirr\src\tools\go\pygo-win\BuildTools\vc\tools\msvc\14.35.32215\bin\Hostx64\x64\cl.exe /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IZ:\home\kir
    r\src\tools\go\pygo-win\pygolang -IZ:\home\kirr\src\tools\go\pygo-win\pygolang\golang\_compat\windows -IZ:\home\kirr\src\tools\go\pygo-win\1.wen
    v\include\site\python3.10 -IZ:\home\kirr\src\tools\go\pygo-win\1.wenv\include "-IC:\Program Files\Python310\include" "-IC:\Program Files\Python3
    10\Include" -Iz:\home\kirr\src\tools\go\pygo-win\BuildTools\vc\tools\msvc\14.35.32215\include -Iz:\home\kirr\src\tools\go\pygo-win\BuildTools\ki
    ts\10\include\10.0.22000.0\shared -Iz:\home\kirr\src\tools\go\pygo-win\BuildTools\kits\10\include\10.0.22000.0\ucrt -Iz:\home\kirr\src\tools\go\
    pygo-win\BuildTools\kits\10\include\10.0.22000.0\um -Iz:\home\kirr\src\tools\go\pygo-win\BuildTools\kits\10\include\10.0.22000.0\winrt /EHsc /Tp
    pyxuser\test.cpp /Fobuild\temp.win-amd64-cpython-310\Release\pyxuser\test.obj /std:c++20 /EHc- /EHsr
    cl : Command line warning D9025 : overriding '/EHc' with '/EHc-'
    test.cpp
    ...
    Z:\home\kirr\src\tools\go\pygo-win\BuildTools\vc\tools\msvc\14.35.32215\bin\Hostx64\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EM
    BED,ID=2 /MANIFESTUAC:NO /LIBPATH:Z:\home\kirr\src\tools\go\pygo-win\pygolang\golang\runtime "/LIBPATH:C:\Program Files\Python310\libs" "/LIBPAT
    H:C:\Program Files\Python310" /LIBPATH:z:\home\kirr\src\tools\go\pygo-win\BuildTools\vc\tools\msvc\14.35.32215\lib\x64 /LIBPATH:z:\home\kirr\src
    \tools\go\pygo-win\BuildTools\kits\10\lib\10.0.22000.0\ucrt\x64 /LIBPATH:z:\home\kirr\src\tools\go\pygo-win\BuildTools\kits\10\lib\10.0.22000.0\
    um\x64 libgolang.lib /EXPORT:PyInit_test build\temp.win-amd64-cpython-310\Release\pyxuser\test.obj /OUT:build\lib.win-amd64-cpython-310\pyxuser\
    test.cp310-win_amd64.pyd /IMPLIB:build\temp.win-amd64-cpython-310\Release\pyxuser\test.cp310-win_amd64.lib
    LINK : fatal error LNK1181: cannot open input file 'libgolang.lib'		<-- NOTE
    error: command 'Z:\\home\\kirr\\src\\tools\\go\\pygo-win\\BuildTools\\vc\\tools\\msvc\\14.35.32215\\bin\\Hostx64\\x64\\link.exe' failed with exi
    t code 1181
    FAILED

    ========================== FAILURES ===========================
    _______________________ test_pyx_build ________________________

        def test_pyx_build():
            pyxuser = testprog + "/golang_pyx_user"
    >       pyrun(["setup.py", "build_ext", "-i"], cwd=pyxuser)

    golang\pyx\build_test.py:31:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    argv = ['setup.py', 'build_ext', '-i'], stdin = None, stdout = None, stderr = None
    kw = {'cwd': 'Z:\\home\\kirr\\src\\tools\\go\\pygo-win\\pygolang\\golang\\pyx/testprog/golang_pyx_user'}, retcode = 1

        def pyrun(argv, stdin=None, stdout=None, stderr=None, **kw):

            retcode, stdout, stderr = _pyrun(argv, stdin=stdin, stdout=stdout, stderr=stderr, **kw)
            if retcode:
    >           raise RuntimeError(' '.join(argv) + '\n' + (stderr and str(stderr) or '(failed)'))
    E           RuntimeError: setup.py build_ext -i
    E           (failed)

    golang\golang_test.py:1771: RuntimeError
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@navytux @mdavidsaver