TreeliteError: Error occured in worker #0: (win 10)

[chjq201410695]

I am trying compile a Lightgbm model in jupyter lab. My code is as follows:

import treelite
model = treelite.Model.load(r'D:\1_Student\ChengJQ\data used by code\Result\model\lightgbm\201908\20190819_AX_Cla_total.txt',
                            'lightgbm')
toolchain = 'gcc'   # change this value as necessary
libpath = 'D:/1_Student/ChengJQ/data used by code/Result/model/lightgbm/201908/C++model/20190819_AX_Cla_compaired.dll'
model.export_lib(toolchain=toolchain, params={'parallel_comp': 256},
                 libpath=libpath, verbose=True)

But the compilation fails. And I get the following error:

[22:09:51] c:\projects\treelite-wheels\treelite\src\c_api\c_api.cc:314: Writing file tu254.c...
[22:09:51] C:\Users\TPM\AppData\Roaming\Python\Python36\site-packages\treelite\contrib\__init__.py:208: WARNING: some of the source files are long. Expect long compilation time. You may want to adjust the parameter parallel_comp.

[22:09:52] C:\Users\TPM\AppData\Roaming\Python\Python36\site-packages\treelite\contrib\util.py:98: Compiling sources files in directory C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6 into object files (*.o)...

TreeliteError                             Traceback (most recent call last)
<ipython-input-7-a6aa38f77224> in <module>
      2 libpath = 'D:/1_Student/ChengJQ/data used by code/Result/model/lightgbm/201908/C++model/20190819_AX_Cla_compaired.dll'
      3 model.export_lib(toolchain=toolchain, params={'parallel_comp': 256},
----> 4                  libpath=libpath, verbose=True)

~\AppData\Roaming\Python\Python36\site-packages\treelite\frontend.py in export_lib(self, toolchain, libpath, params, compiler, verbose, nthread, options)
     97       self.compile(temp_dir, params, compiler, verbose)
     98       temp_libpath = create_shared(toolchain, temp_dir, nthread,
---> 99                                    verbose, options)
    100       shutil.move(temp_libpath, libpath)
    101 

~\AppData\Roaming\Python\Python36\site-packages\treelite\contrib\__init__.py in create_shared(toolchain, dirpath, nthread, verbose, options)
    219     from .gcc import _create_shared, _openmp_supported
    220   libpath = \
--> 221     _create_shared(dirpath, toolchain, recipe, nthread, options, verbose)
    222   if verbose:
    223     log_info(__file__, lineno(),

~\AppData\Roaming\Python\Python36\site-packages\treelite\contrib\gcc.py in _create_shared(dirpath, toolchain, recipe, nthread, options, verbose)
     58   recipe['create_library_cmd'] = lib_cmd
     59   recipe['initial_cmd'] = ''
---> 60   return _create_shared_base(dirpath, recipe, nthread, verbose)
     61 
     62 def _check_ext(dllpath):

~\AppData\Roaming\Python\Python36\site-packages\treelite\contrib\util.py in _create_shared_base(dirpath, recipe, nthread, verbose)
    121         f.write(result[tid]['stdout'] + '\n')
    122       raise TreeliteError('Error occured in worker #{}: '.format(tid) +\
--> 123                           '{}'.format(result[tid]['stdout']))
    124 
    125   # 2. Package objects into a dynamic shared library

TreeliteError: Error occured in worker #0: Microsoft Windows [版本 10.0.18362.295]
(c) 2019 Microsoft Corporation。保留所有权利。

C:\Users\TPM\CHJQ_CODE>cd C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>type NUL > retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu221.o tu221.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu42.o tu42.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu31.o tu31.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu146.o tu146.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu35.o tu35.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu142.o tu142.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu49.o tu49.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu135.o tu135.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu131.o tu131.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu22.o tu22.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu50.o tu50.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu179.o tu179.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu176.o tu176.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu66.o tu66.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu74.o tu74.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu82.o tu82.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu90.o tu90.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu99.o tu99.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu107.o tu107.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu117.o tu117.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu123.o tu123.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu127.o tu127.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu148.o tu148.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu152.o tu152.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu156.o tu156.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu161.o tu161.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu165.o tu165.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu169.o tu169.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu173.o tu173.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu183.o tu183.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu191.o tu191.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>gcc -c -O3 -o tu199.o tu199.c -fPIC -std=c99 

cc1.exe: out of memory allocating 65536 bytes

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>echo %errorlevel% >> retcode_cpu0.txt

C:\Users\TPM\AppData\Local\Temp\tmpk_p4h0f6>

any advise on it?

[hcho3]

It looks like you ran out memory? How big is your model?

I'm planning to add a "fail-safe" compilation mode so that you can compile arbitrarily big models with reasonable amount of time and memory.

[chjq201410695]

It looks like you ran out memory? How big is your model?

I'm planning to add a "fail-safe" compilation mode so that you can compile arbitrarily big models with reasonable amount of time and memory.

I don't think it's an out memory error. my model is 250M, while my RAM is 48G. I compiled successfully in ubantu. it just failed in my win10 pc. Let me try set 'parallel_comp' to 512 and feedback you the result.

[chjq201410695]

It looks like you ran out memory? How big is your model?

I'm planning to add a "fail-safe" compilation mode so that you can compile arbitrarily big models with reasonable amount of time and memory.

Ha, you are right, it ran out memory. I set a larger number of parallel_comp to 512, and it works.