Python 3.6.2 include errors

[SecretShop]

Link to Reddit where problem started: https://www.reddit.com/r/Python/comments/6txk3x/pynsist_troubles_with_running/

The following is my installer.cfg file

name=LoLDataComp
version=1.0
entry_point=LoLDataComp:main

[Python]
version=3.6.1

[Include]
packages = requests
     urllib3
     chardet
     certifi
     idna
     tkinter
     _tkinter
     
files = README.txt
    tcl86t.dll
    tk86t.dll

So i have my desktop and a laptop, and my desktop has python and everything on it to work with python. So every time i want to test to see if the program runs i put it on my laptop since it doesn't have python and i wanna make sure it works in that environment.

The following is the error message i get whenever i use command line to run "python\python.exe LolDataComp.launch.pyw":

[Siecje]

To format your config wrap it with three backticks ( ``` ) before and after.

Do you have tcl86t.dll and tk86t.dll in the same directory as your installer.cfg?

Are your packages and files indented after the first line?

For example

packages = requests
    urllib3
    chardet
    certifi
    idna
    tkinter
    _tkinter

[Siecje]

Have you tried with different versions of Python?

[SecretShop]

I have not tried it with other versions of Python because i do not know how that would effect my code and all of my packages. I included tcl86t.dll and tk86t.dll in the same directory on my installer.cfg

[Siecje]

I think it is because your Desktop is 64bit and your laptop is 32bit?

Try setting the bitness to 32 bit

http://pynsist.readthedocs.io/en/latest/cfgfile.html#python-section

[SecretShop]

I have a first gen Surface Pro, i just checked the System specs and i do in fact have 64-bit operating system on both my Desktop and Laptop.

[takluyver]

Thanks @Siecje for helping debug this. :-)

That error message usually means that the DLLs are not all for the same bitness. You can run 32-bit applications on a 64-bit operating system, but all the DLLs have to have the same bitness as the exe (python.exe) you're running.

Where did you get the DLLs from? And can you check the bitness of the python.exe it installs in the application?

[SecretShop]

I got the DLL's from my pythyon36 DLL folder. On the reddit page i refered to this post thinking it could solve my issue:
https://www.reddit.com/r/learnpython/comments/5uvlu9/tkinter_could_not_be_found_when_running_project/

[takluyver]

BTW, you're using the 'bundled' Python format, which we made the default when you build with Python 3.6+ (and will be the only option at some point in the future). 'Bundled' Python can make much smaller installers than the alternative 'installer' format, but part of the reason for that is that it doesn't include Tkinter by default. So thanks for helping us figure out what's needed to use it with Tkinter!

[takluyver]

Here's some info on how to check if a Windows binary file is 32-bit or 64-bit: https://superuser.com/a/889267/209976

Can you try that on the python.exe and the two DLLs and report the results?

[SecretShop]

The .exe is x64
tcl86t.dll is x86
tk86t.dll is x86

If you need anything else from me please ask. Thank you very much for all the help.

[takluyver]

OK, that's the key. If the exe is x64, it needs x64 DLLs as well, but you've got x86 (32-bit) DLLs.

I think the easiest way for you to fix it is to build with an x86 Python, so you can keep using the same DLLs. You can do that by adding bitness=32to the Python section of your installer.cfg.

[SecretShop]

I changed the bitness=32 and double checked the python.exe and it is in fact x86, but i still get the same errors. I am currently double checking.

EDIT: should the DLL files be in the main directory or should i place them somewhere else? currently they are just included and that's it.

EDIT2: I Triple verified that python.exe and both DLL files are x86.

[takluyver]

The DLLs might need to go in the same directory as python.exe. I would have expected a different error message if it couldn't find them at all, but maybe I'm wrong.

After that, check the bitness of any other DLLs and .pyd files (which are also DLLs). If they all seem to be x86, the next thing I'd suggest is to investigate with dependency walker.

[SecretShop]

So i re-installed Python thinking maybe i screwed something up with that and i got some different errors this time around:

EDIT: I am currently trying to copy the TCL folder found in the python directory into my build to fix this.

EDIT2: OMG it works.

[takluyver]

Aha, I think that's progress - you're past one error and onto another.

I think what you're trying is the right thing. You might also need to set the TCL_LIBRARY environment variable in Python before importing tkinter.

[SecretShop]

It is now working, but now i'm trying not to include the 1000+ files of tcl folder.

EDIT: Alright so it needs more than just that one init.tcl file so because i don't wanna find out, i included the whole TCL folder found in the python directory and i placed it in a folder named "lib" which i include while building so that it can actually find the files.

EDIT2: ALRIGHT SO, with the fixes i made... i still have the original issue for some reason where i run the exe, i press the shortcut made by the installer, and nothing happens. Using the Cmd line works, but the shortcut doesn't and i really need that to work so anybody can run it easily. Any help in that area?

[takluyver]

Awesome! Could we get you to write up all the pieces you needed to distribute a tkinter application with bundled Python? Perhaps we can even make an example out of it. :-)

For the issue with the shortcut, if it fails to start it should write out a log file with details of any errors that occurred. Have a look at the info here:
http://pynsist.readthedocs.io/en/latest/installers.html#uncaught-exceptions

[SecretShop]

I will definitely write up what is needed when i get the shortcut working.

I found the log file and it gives the following error:

  File "C:\Users\SecretShop\Desktop\LoLDataComp\LoLDataComp.launch.pyw", line 30, in <module>
    from LoLDataComp import main
  File "C:\Users\SecretShop\Desktop\LoLDataComp\pkgs\LoLDataComp.py", line 5, in <module>
    import tkinter
  File "C:\Users\SecretShop\Desktop\LoLDataComp\pkgs\tkinter\__init__.py", line 36, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
ImportError: DLL load failed: The specified module could not be found.

To clarify, this does not pop up when i run from the Cmd line.

[takluyver]

Are you using the TCL_LIBRARY environment variable, and if so how are you calculating it? Where is the lib directory you created?

[takluyver]

I suspect that you might need to move either the lib folder, or the tcl/tk dlls, or both, to the directory where python.exe is.

[Siecje]

The only difference between the shortcut and running it from cmd is the current working directory.

[Siecje]

If you put the .dll files in the Python directory does it work?

[SecretShop]

putting the 2 .DLL files in the python directory does not fix the shortcut issue. I just tried that :/

EDIT: Although i do suspect that like my other issue, it is a matter of what directory they should be in.

[takluyver]

Is it still the same error in the log file, or a different one?

If we can't find a better way round it, one hackish workaround might be to use os.chdir() to change the working directory in Python before loading Tkinter. But I hope we can find a neater solution than that.

[SecretShop]

exact same error in the log file i am afraid.

[Siecje]

This seems like the same problem that I had by the of the long discussion

#90 (comment)

I thought we added the Python\ directory to PATH so that the .dll files would be found.

aea8231

[takluyver]

Yup, your PR #101 added that.

@SecretShop can you double check that by printing os.environ('PATH') before you attempt to load tkinter? Anything printed should go into the same log file as the error, so you can use it for debugging.

[takluyver]

Or maybe those DLLs need to go in the pkgs folder, alongside _tkinter.pyd.

[SecretShop]

EYYYYYYYYYYY in pkgs fixed it

EDIT: I'm rebuilding the whole thing to make sure it builds and works properly.

[takluyver]

🎉 ✨ 🍰

[SecretShop]

Okay new issue, how do i make the installer put those .dll files into pkgs?
im silly i know how

[takluyver]

Something like this should do the trick:

files = README.txt
    tcl86t.dll > $INSTDIR\pkgs
    tk86t.dll > $INSTDIR\pkgs

[SecretShop]

It works now!

1. I will create a small list here of what needs to be put in for tkinter to work.
2. Thank you both very much for holding my hand through this.
3. I hope this problem solving session was useful in some way to your project and its future and not some awful time sink just for me.

[takluyver]

Thanks! It's definitely useful to work out what's needed for a Tkinter app with bundled Python, especially as I'm planning to make bundled Python the only option.

[SecretShop]

Files or Folders needed and their location in ( )
These files need to be placed into the pynsis_pkgs folder:

• tcl86t.dll ("Python\DLLs")
• tk86t.dll ("Python\DLLs")

Copy all the contents of tcl found in the main directory of Python into a folder named lib in the main installation directory and include it as a file.

Be sure to include the following as packages :

• tkinter
• _tkinter

Finally, ensure bitness=32 is set since the DLL's are x86

[takluyver]

Thanks!

I think copying the tkinter folder to pynsist_pkgs manually should be equivalent to including tkinter in the list of packages in the config file. But maybe I've overlooked something.

Does the lib folder go in the main installation directory? So you end up with paths like $INSTDIR\lib\tcl86t.lib and $INSTDIR\lib\tcl8.6\init.tcl?

I'm pretty sure you do need _tkinter included - that refers to the file _tkinter.pyd, which is imported by the tkinter Python code.

[SecretShop]

You are correct about copying tkinter i just checked. I removed that and noted it just needs to be included.

Yes and Yes, i clarified up above because i was unclear.

Also yes, i double checked and this does grab that file.

[SnidhiSofpro]

With thanks to pynsist creator @takluyver & to @SecretShop for their inputs on this tkinter issue, here's a pynsist config file for ready-reference of future readers of this thread (might save a few iterations & hours!):

pynsist build config: for Filer app; w/ tkinter files

=== Application section ===

[Application]
name=myapp
version=0.1832
entry_point=myapppkg.mymodule:myfunction

=== Python section ===

[Python]
version=3.6.1
bitness=32

bitness is required for tkinter distribution; as of pynsist 2.1

=== Include section ===

[Include]

packages=mypkg1
mypkg2
mypkg3
tkinter
_tkinter

tkinter & _tkinter are required for tkinter distribution; as of pynsist 2.1 & python 361.

files = lib
tcl86t.dll > $INSTDIR\pkgs
tk86t.dll > $INSTDIR\pkgs

The above files are required for tkinter distribution; as of pynsist 2.1;

lib is a folder, with a copy of all contents of folder tcl found in the main directory of Python; must be named as lib!

Related: https://bitbucket.org/anthony_tuininga/cx_freeze/issues/155/required-environment-variables-tcl_library

[takluyver]

Thanks! Would anyone be interested in turning this into an example in the examples directory in the repo?

The ideal would be to include a script which can download and arrange the necessary files (like in the pywebview example), but it would also be a valuable contribution if it uses human-readable instructions on where to find the tcl/tk libraries.

[ntoll]

@takluyver I have exactly this requirement for Mu. Folks at the Raspberry Pi Foundation want turtle available with Mu (and it depends upon tkinter).

My question: where do I source the dlls needed (tcl86t.dll and tk86t.dll) and I assume they'll work for both 32 and 64bit versions of Python. I've found installers (see: https://www.activestate.com/activetcl/downloads) but all we need, AFAICT, are the dlls. Right..?

EDIT: Scratch that... we need the tcl directory too. :-/ Hmm... could we (pynsist) provide a zip for both 32 and 64 bit tcl dirs..?

[takluyver]

You should be able to get the DLLs from a regular Windows installation of Python. The problem for Pynsist is that the 'embeddable zip file' builds we're using don't include Tkinter.

I don't know how Python loads the tcl/tk DLLs, but I would assume they need to match the bitness of the Python interpreter.

[ntoll]

OK... I'm in London today (away from my sacrificial Windows laptop), but will have a go tomorrow and let you know how I get on.

[takluyver]

Thanks. I may try to investigate it as well.

[ntoll]

My current thinking (pending actual work on a Windows machine) is:

• Pynsist checks if tkinter and _tkinter are packages (EDIT: in the build config).
• If so, it downloads the correct bitness versions of the tcl directory and dlls.
• These are unzipped in the right place and creation of the installation asset proceeds as normal.

The requirement for downloads means that you could still build this on, say, a Linux box. We'd need to host them somewhere (as files associated with this project perhaps..?).

Thoughts..?

[takluyver]

Sounds good to me. We'll also need to make the _tkinter extension module available somewhere for each version of Python we want to support.

My strategy for hosting some files previously has been to put them in an orphan branch in this repository, make a tag, and then use RawGit for the download URLs. An alternative might be to pack them into wheels and use PyPI for hosting.

[ntoll]

Ugh... so this is proving problematic. I suspect part of this is because I'm doing this work on Linux (although pynsist should work on non-WIndows platforms).

Here's what I've done so far:

• I've grabbed both the 32 and 64 bit versions of the tcl directory and tcl86t.dll and tk86t.dll files.
• I've attempted to use these to make a 64-bit installable of Mu.
• What would be really helpful is knowing what tkinter stuff should go where on the users filesystem AFTER they've installed the package.
• When I include _tkinter and tkinter as packages in the config file for pynsist, it complains:

nsist.copymodules.ExtensionModuleMismatch: Found an extension module that will not be usable on Windows:
/home/ntoll/.virtualenvs/mufoo/lib/python3.6/lib-dynload/_tkinter.cpython-36m-x86_64-linux-gnu.so
Put Windows packages in pynsist_pkgs/ to avoid this.

• I guess the contents of tkinter and possibly turtle for Windows needs to be used rather than the native version (for me on Linux).

Ugh... "HELP"...!?!?!?

[ntoll]

tl;dr - once I know what needs to go where in the installed application so that it actually works, then I can work backwards from that to know what things need to be downloaded and copied into the correct locations.

[ntoll]

Also, when I try building the installer for Mu on Windows, well, you'll see here... ;-)

https://ci.appveyor.com/project/carlosperate/mu/build/1.0.788

388 Traceback (most recent call last):
389  File "c:\python36\lib\runpy.py", line 193, in _run_module_as_main
390    "__main__", mod_spec)
391  File "c:\python36\lib\runpy.py", line 85, in _run_code
392    exec(code, run_globals)
393  File "C:\Python36\scripts\pynsist.exe\__main__.py", line 9, in <module>
394  File "c:\python36\lib\site-packages\nsist\__init__.py", line 513, in main
395    InstallerBuilder(**args).run(makensis=(not options.no_makensis))
396  File "c:\python36\lib\site-packages\nsist\__init__.py", line 471, in run
397    self.copy_extra_files()
398  File "c:\python36\lib\site-packages\nsist\__init__.py", line 412, in copy_extra_files
399    shutil.copy2(file, self.build_dir)
400  File "c:\python36\lib\shutil.py", line 257, in copy2
401    copyfile(src, dst, follow_symlinks=follow_symlinks)
402  File "c:\python36\lib\shutil.py", line 120, in copyfile
403    with open(src, 'rb') as fsrc:
404FileNotFoundError: [Errno 2] No such file or directory: ''

:-/

Although I notice the lib directory containing the contents of tcl hasn't found its way onto the branch. :-/

[ntoll]

With the lib directory, I still get the same "No such file or directory: ''" error. :-(

I'm not sure how to proceed until I know what the geography of a known-to-be-working application should look like in terms of tkinter.

[ntoll]

Apologies for the noise... that final error is because I put the lib on its own line (thus creating an empty entry where the \n was after files=)

[takluyver]

Yeah, we should probably allow for empty lines in all lists in the config file.

From the config @SnidhiSofpro posted above, it looks like the tcl folder needs to be renamed lib and placed in the installation directory, while the DLLs go in the pkgs folder. The tkinter and _tkinter modules will also go in pkgs; the latter is a binary extension module, so it needs to come from a Windows computer too. It looks like the tkinter package is pure Python.

I'm going to try to play around with this tomorrow and see if I can get it working. May as well do something else useful while I'm on strike. ;-)

[ntoll]

OK... I've got it to work in that I've managed to build a 64bit installer for Mu that includes tkinter and (here's the ultimate goal) turtle modules.

Now that I know what file goes where, I'm left wondering if it might just be easier to address this problem as part of Mu's CI / build automation. I.e. I'm going to zip up the files needed for 32 and 64 bit tkinter so you just have to unzip them into the project's directory before running pynsist so everything will get into the right place. It's a far quicker / simpler solution (for me, YMMV) than hacking a fix into pynsist itself. Thoughts..? (I'll point you to the resulting work when done.)

In any case, I'll create a PR for an update to the docs to address tkinter in an easy to understand way.

[ntoll]

Assets to be included in the installer for 32 and 64bit Windows can be found here: https://github.com/mu-editor/mu_tkinter (subject to testing).

[takluyver]

Thanks @ntoll ! I might still look into some integration to make it easy to do this with Pynsist, but I don't know when I'll get round to that, so it would be good to have some documentation/example showing people how to do it with the machinery that's already there.

[ntoll]

OK... I've successfully completed this work for Mu. Documentation related PR coming just as soon as I've written it up. :-)

[sprocket1]

I just want to say thanks to takluyver for creating this tool, and also to ntoll for the work you've done in sorting this out and putting the resources up. This afternoon, I just successfully built my first ever windows installer for my first ever Python app using a GUI. It is a 32bit installer for a project using Python 3.6.4, matplotlib, numpy, scipy, and tkinter. I decided to make it 32bit so that I could run this on legacy machines as well as new ones my clients might have.

I would still be struggling if it weren't for this thread and these resources. After getting oriented on the structure of the installer.cfg, the only trouble I had was that some of the dependencies of matplotlib were not obvious. But I chased them down one at a time and added them as the execution showed the errors. In case anyone else comes across this same thing, they turned out to be, at least in my case:
six
pyparsing
cycler
python-dateutil
pytz

My working environment is Anaconda 3 and I guess you can say that this is a byproduct of it giving you almost everything you need, and casually shielding you from knowing exactly what that is precisely.

[takluyver]

Thanks @sprocket1 - it's always good to hear that the tool is working for someone.

I would still like to make the dependency handling more automatic at some point. But I don't know when that point will be.

[cc20101001]

thanks