ModuleNotFoundError during Kernel Initialization (Windows)

[mikkeyboi]

Traceback (most recent call last):
  File "C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\lib\site-packages\lldb\__init__.py", line 35, in <module>
    import _lldb
ModuleNotFoundError: No module named '_lldb'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\mleun\projects\swift-jupyter\swift_kernel.py", line 19, in <module>
    import lldb
  File "C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\lib\site-packages\lldb\__init__.py", line 38, in <module>
    from . import _lldb
ImportError: DLL load failed: The specified module could not be found.
[I 05:16:55.241 NotebookApp] KernelRestarter: restarting kernel (1/5)

Steps to reproduce:

1. Use conda setup instructions (in Anaconda Prompt, activate the conda environment), acquire toolchain from notes here (Fix paths for Windows #97)
2. To use changes for the fixed Windows paths in the above PR:
   • Clone this repository, and cd into the directory
   • git remote add -f mikkeyboi https://github.com/mikkeyboi/swift-jupyter.git
   • git checkout mikkeyboi/master
3. Run register.py while inside conda environment
   • ie: python register.py --sys-prefix --swift-python-use-conda --use-conda-shared-libs --swift-toolchain path/to/toolchain
4. Make a jupyter notebook session jupyter notebook while within conda environment
5. Create a new notebook with Swift kernel
6. While the kernel is starting, I get the error.

Alternatively..

1. cd into path/to/toolchain/usr/lib/site-packages and run interactive python ipython
2. import lldb

I think it's not loading the _lldb.pyd file that's inside the lldb module. Either that, or I'm missing DLLs (can confirm with Dependency Walker).

@compnerd Were you able to load the lldb module during your testing (specifically the _lldb.pyd file)?

[compnerd]

Yeah, I was able to get the Python bits loaded but ran into a few other issues that I’m working through in lldb itself (like module loading)

[compnerd]

Okay, so, one thing that I've found is that we need a few extra flags when starting the REPL:
lldb --repl="-sdk %SDKROOT% -I %SDKROOT%\usr\lib\swift -L %SDKROOT%\usr\lib\swift\windows" seems to ensure that the REPL starts up correctly.

I think that for the Jupyter notebook case, we can workaround this by modifying the toolchain in the Jupyter setup - take an additional parameter in the install script -sdk that provides the SDK path to use. We could then symlink the SDK into the toolchain which would allow the search to succeed without the flags.

[mikkeyboi]

Awesome, I'll give that a go. Is the SDK root this Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk? At the moment there's a reference to something similar in LD_LIBRARY_PATH (diverges to Library instead of SDKs):

kernel_env['LD_LIBRARY_PATH'] = os.path.join(os.path.dirname(os.path.dirname(args.swift_toolchain)),
   'Platforms','Windows.platform','Developer','Library','XCTest-development',
   'usr','lib','swift')

If it's all in relative paths I could add the SDKROOT as an env variable (a little messy with the dirname's and path joining), and we could symlink for readability.

[compnerd]

Yes, that’s the SDKROOT (%SystemDrive%\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk).

Wait, you’re saying that it’s possible to do this with environment variables? That would be really handy to switch between different SDK versions.

Hmm, I didn’t catch that before. Windows doesn’t have the concept of LD_LIBRARY_PATH. The path needs to be added to the PATH variable.

The path that you are spending which “diverges” is not the SDK. XCTest is not part of the SDK (you don’t ship the tests, you ship the application/library), it’s a developer library. That’s why it is separate.

Most things are relative to the SDKROOT. The only thing which is relative to the DEVELOPERROOT is XCTest.

[mikkeyboi]

Currently the register.py script assigns variables to tell the kernel where things are, the script says environment variables and I think it means within its own environment (rather than the Windows environment variables).
But it is possible to do this with Windows environment variables via os.environ['ENVIRONMENTVARIABLE']. If we do it this way it might require more work from the user unless your toolchain installer adds to the PATH and we can modify the script to read from this.

Oh I see, I think I might need your help with what each section in the folder tree does (and consequently what swift-jupyter needs). Which one contains the Swift toolchain libs, the XCTest or the SDK root?

[compnerd]

unless your toolchain installer adds to the PATH

Actually, it does adjust the path. The installer has been getting better over time, but, I think it comes down to finding the ways to improve it and finding others to help with it :) The current three interesting bits:

https://github.com/compnerd/swift-build/blob/master/wix/windows-runtime.wxs#L104-L106
https://github.com/compnerd/swift-build/blob/master/wix/windows-tensorflow.wxs#L47-L49
https://github.com/compnerd/swift-build/blob/master/wix/windows-toolchain.wxs#L363-L365

Basically, the toolchain, runtime, and tensorflow are injected into the path. Im totally open to patches to improve the installers BTW :)

If it would make things easier, having an advanced option to set SDKROOT and DEVELOPER_LIBRARY_DIR added as part of the installation sounds totally reasonable to me (preferably with the UI components).

I am absolutely happy to explain that. I'll actually see if I can draft up something as an explanation of the current layout in the swift-build project, and for whatever other pieces which are missing, we can extend it to cover that. That way we have an easy reference for the future.

[compnerd]

Initial pass at the documentation of the layout: https://github.com/compnerd/swift-build/blob/master/docs/Layout.md

For improvements to the toolchain layout/installation lets keep the discussion on the swift-build project so that others watching that project can possibly get involved as well.

[yogeshhk]

Still getting the error "ImportError: DLL load failed while importing _lldb: The specified module could not be found."

Any resolution?

[compnerd]

I think that your Path variable may not be setup properly, resulting in Python.dll not being found. Ensure that the python version matches what the library expects (you can use link -dump -imports or llvm-readobj -coff-imports to determine the version requirement).

[yogeshhk]

I think that your Path variable may not be setup properly, resulting in Python.dll not being found. Ensure that the python version matches what the library expects (you can use link -dump -imports or llvm-readobj -coff-imports to determine the version requirement).

Thanks