ibm_boto3 cannot work along with pyinstaller?

[thetime1102]

Does there anyone use pyinstaller to generate execution file for ibm_boto3 ?
I always got error during "The 's3' resource does not exist."
please help me.

[paul-carron]

@thetime1102 could you provide more information? If you could provide a full error trace, the command you're running and anything else that might be relevant.

Also, does it work before you package it with pyinstaller?

[thetime1102]

@paul-carron

To reproduce:

1. Install library

    pip install ibm-cos-sdk
    pip install pyinstaller
   

2. Create test_cos.py:

   import ibm_boto3
   from ibm_botocore.client import Config
   api_key = 'API_KEY'
   service_instance_id = 'SERVICE_INSTANCE_ID'
   auth_endpoint = 'https://iam.bluemix.net/oidc/token'
   service_endpoint = 'SERVICE_ENDPOINT'
   
   new_bucket = 'testbucket99999'
   
   cos = ibm_boto3.resource('s3',
                        ibm_api_key_id=api_key,
                        ibm_service_instance_id=service_instance_id,
                        ibm_auth_endpoint=auth_endpoint,
                        config=Config(signature_version='oauth'),
                        endpoint_url=service_endpoint)
   
   cos.create_bucket(Bucket=new_bucket)
   
   for bucket in cos.buckets.all():
       print(bucket.name)

3. Run normally, it works:

   python test_cos.py
   

4. Run pyinstaller to generate execution test_cos.exe file:

   pyinstaller -F test_cos.py
   

5. Run test_cos.exe, it fails

   test_cos.exe
   

6. Bellow is trace back

   Traceback (most recent call last):
     File "test_cos.py", line 1, in <module>
     File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-    packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
       exec(bytecode, module.__dict__)
     File "site-packages\ibm_boto3\__init__.py", line 16, in <module>
     File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-    packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
       exec(bytecode, module.__dict__)
     File "site-packages\ibm_boto3\session.py", line 27, in <module>
     File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-    packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
       exec(bytecode, module.__dict__)
     File "site-packages\ibm_botocore\session.py", line 38, in <module>
     File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
       exec(bytecode, module.__dict__)
     File "site-packages\ibm_botocore\client.py", line 16, in <module>
     File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
       exec(bytecode, module.__dict__)
     File "site-packages\ibm_botocore\waiter.py", line 18, in <module>
     File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
       exec(bytecode, module.__dict__)
     File "site-packages\ibm_botocore\docs\__init__.py", line 15, in <module>
     File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
       exec(bytecode, module.__dict__)
     File "site-packages\ibm_botocore\docs\service.py", line 18, in <module>
     File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
       exec(bytecode, module.__dict__)
     File "site-packages\ibm_botocore\docs\bcdoc\restdoc.py", line 16, in <module>
     File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
       exec(bytecode, module.__dict__)
     File "site-packages\ibm_botocore\docs\bcdoc\docstringparser.py", line 16, in <module>
     File "site-packages\ibm_botocore\vendored\six.py", line 89, in __get__
     File "site-packages\ibm_botocore\vendored\six.py", line 108, in _resolve
     File "site-packages\ibm_botocore\vendored\six.py", line 79, in _import_module
   ModuleNotFoundError: No module named 'html.parser'
   [1564] Failed to execute script test_cos
   

7. Platform info:

   Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 03:37:03) [MSC v.1900 64 bit (AMD64)] on win32
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import platform
   >>> platform.machine()
   'AMD64'
   >>> platform.platform()
   'Windows-10-10.0.17134-SP0'
   >>> platform.processor()
   'Intel64 Family 6 Model 58 Stepping 9, GenuineIntel'
   >>> platform.system()
   'Windows'
   >>> platform.uname()
   uname_result(system='Windows', node='DESKTOP', release='10', version='10.0.17134',         machine='AMD64', processor='Intel64 Family 6 Model 58 Stepping 9, GenuineIntel')
   
   >>>pip show ibm-cos-sdk
   Name: ibm-cos-sdk
   Version: 2.3.1
   >>>pip show pyinstaller
   Name: PyInstaller
   Version: 3.5.dev0+0ffa574bf
   

[paul-carron]

The issue seems to be with pyinstaller rather than the cos-sdk.

Please try removing the docs and the tests folders from ibm-cos-sdk-python-core and ibm-cos-sdk-python before packaging. I believe these files should not be packaged.

Failing that, you may need to consult the pyinstalled documentation.

[thetime1102]

@paul-carron
i asked pyinstaller team.
can you link to that issue?
pyinstaller/pyinstaller#3822

[paul-carron]

Did you try try removing the docs and the tests folders from ibm-cos-sdk-python-core and ibm-cos-sdk-python before packaging?

[thetime1102]

@paul-carron

if removed the docs and the tests folders before packaging,
when run script bellow => it's failed

```
python test_cos.py
```

```
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    import ibm_boto3
  File "C:\Users\user_name\AppData\Local\Programs\Python\Python36\lib\site-        packages\ibm_boto3\__init__.py", line 16, in <module>
    from ibm_boto3.session import Session
  File "C:\Users\user_name\AppData\Local\Programs\Python\Python36\lib\site-    packages\ibm_boto3\session.py", line 27, in <module>
    import ibm_botocore.session
  File "C:\Users\user_name\AppData\Local\Programs\Python\Python36\lib\site-    packages\ibm_botocore\session.py", line 38, in <module>
    import ibm_botocore.client
  File "C:\Users\user_name\AppData\Local\Programs\Python\Python36\lib\site-packages\ibm_botocore\client.py", line 16, in <module>
    from ibm_botocore import waiter, xform_name
  File "C:\Users\user_name\AppData\Local\Programs\Python\Python36\lib\site-    packages\ibm_botocore\waiter.py", line 18, in <module>
    from ibm_botocore.docs.docstring import WaiterDocstring
ModuleNotFoundError: No module named 'ibm_botocore.docs'
```

[paul-carron]

This seems to be common with PyInstaller.

You could try passing the below as a hook:

from PyInstaller.utils.hooks import collect_data_files
from PyInstaller.compat import is_py2
from PyInstaller.utils.hooks import is_module_satisfies

if is_module_satisfies('ibm_botocore >= 2.0.0'):
    if is_py2:
        hiddenimports = ['HTMLParser']
    else:
        hiddenimports = ['html.parser']

datas = collect_data_files('ibm_botocore')

1. Create the hook file(hook-ibm_botocore.py)
2. Run this command to package the application pyinstaller -F --additional-hooks-dir=. --hidden-import=ibm_botocore test_cos.py
3. Run the application

Here's some info that may help:
https://pythonhosted.org/PyInstaller/hooks.html
pyinstaller/pyinstaller#2094
pyinstaller/pyinstaller@6f45521

[paul-carron]

@thetime1102 Did the above information help you resolve your issue?

[thetime1102]

@thetime1102 Did the above information help you resolve your issue?

@paul-carron
I tried but can not packaging.

To reproduce:

1. create hook-ibm_botocore.py file

2. add hook-ibm_botocore.py file to C:\Users\user_name\AppData\Local\Programs\Python\Python36\Lib\site-packages\PyInstaller\hooks\

3. run command to package the application

   pyinstaller -F --additional-hooks-dir=. --hidden-import=ibm_botocore test_cos.py
   

4. when packaging i got a error like bellow

   15556 INFO: Caching module hooks...
   15556 INFO: Analyzing C:\Tutorial\test_ibm_cos\test_cos.py
   15783 INFO: Processing pre-safe import module hook   six.moves
   25248 INFO: Processing pre-safe import module hook   urllib3.packages.six.moves
   30060 INFO: Loading module hooks...
   30060 INFO: Loading module hook "hook-certifi.py"...
   30074 INFO: Loading module hook "hook-cryptography.py"...
   31709 INFO: Loading module hook "hook-encodings.py"...
   32079 INFO: Loading module hook "hook-ibm_botocore.py"...
     File "<string>", line 2
       import ibm-botocore as m
                 ^
   SyntaxError: invalid syntax
   32377 INFO: Loading module hook "hook-ibm_botocore.py"...
     File "<string>", line 2
       import ibm-botocore as m
   SyntaxError: invalid syntax
   32645 INFO: Loading module hook "hook-pydoc.py"...
   32645 INFO: Loading module hook "hook-xml.etree.cElementTree.py"...
   32660 INFO: Loading module hook "hook-xml.py"...
   32932 INFO: Looking for ctypes DLLs
   32949 INFO: Analyzing run-time hooks ...
   32964 INFO: Looking for dynamic libraries
   

5. but .... when i try remove hook-ibm_botocore.py file and add bellow code to test_cos.py file

   import html.parser
   

6. and run this command to repackage the application pyinstaller -F test_cos.py

7. then when run test_cos.exe, i got error like the top comment in this topic ↑

   Traceback (most recent call last):
     File "site-packages\ibm_boto3\session.py", line 422, in resource
     File "site-packages\ibm_botocore\loaders.py", line 132, in _wrapper
     File "site-packages\ibm_botocore\loaders.py", line 378, in load_service_model
   ibm_botocore.exceptions.UnknownServiceError: Unknown service: 's3'. Valid service names are:
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
     File "test_cos.py", line 17, in <module>
     File "site-packages\ibm_boto3\__init__.py", line 92, in resource
     File "site-packages\ibm_boto3\session.py", line 428, in resource
   ibm_boto3.exceptions.ResourceNotExistsError: The 's3' resource does not exist.
   The available resources are:
     -
   
   [7424] Failed to execute script test_cos
   

[cpcunningham]

Create hook file --- hook-ibm_boto3.py (paste contents between lines)

from PyInstaller.utils.hooks import collect_data_files, collect_submodules

hiddenimports = (
collect_submodules('ibm_boto3.s3'))

datas = collect_data_files('ibm_boto3')

Create hook file --- hook-ibm_botocore.py (paste contents between lines)

from PyInstaller.utils.hooks import collect_data_files
from PyInstaller.compat import is_py2
from PyInstaller.utils.hooks import is_module_satisfies

hiddenimports = ['html.parser']

datas = collect_data_files('ibm_botocore')

Run pyinstaller with app
pyinstaller --additional-hooks-dir=. pytest.py

[cpcunningham]

I have tested the above and it works,
We will not be able to provide any more help on this as pyinstaller is not supported by ibmcos...