Android/Termux Lacks Semaphores (sem_open) - Fatal Error

[danieldjewell]

Hi!

Installing on Android/Termux seems to go just fine but when trying to convert JSON data to sqlite, there is a fatal error because Android doesn't have a functioning sem_open method for threading (semaphores)... see trace below and see Python Issue#3770. sqlite3 compiles and runs just fine as do most other python modules...

I haven't had a chance to really look through the trace yet to see where the multiprocessing module is getting pulled into the mix.

Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/data/com.termux/files/usr/lib/python3.8/lib-dynload/_multiprocessing.cpython-38.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/dataproperty/_extractor.py", line 556, in __to_dp_matrix_mt
    with futures.ProcessPoolExecutor(self.max_workers) as executor:
  File "/data/data/com.termux/files/usr/lib/python3.8/concurrent/futures/process.py", line 555, in __init__
    self._call_queue = _SafeQueue(
  File "/data/data/com.termux/files/usr/lib/python3.8/concurrent/futures/process.py", line 165, in __init__
    super().__init__(max_size, ctx=ctx)
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/queues.py", line 42, in __init__
    self._rlock = ctx.Lock()
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 67, in Lock
    from .synchronize import Lock
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/bin/sqlitebiter", line 8, in <module>
    sys.exit(cmd())
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/sqlitebiter/sqlitebiter.py", line 288, in file
    converter.convert(file_path)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/sqlitebiter/subcommand/_file.py", line 104, in convert
    self.__convert(file_path, source_info_record_base)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/sqlitebiter/subcommand/_file.py", line 143, in __convert
    self._table_creator.create(
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/sqlitebiter/_table_creator.py", line 27, in create
    con_mem.create_table_from_tabledata(
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/simplesqlite/core.py", line 1310, in create_table_from_tabledata
    self.__create_table_from_tabledata(
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/simplesqlite/core.py", line 1660, in __create_table_from_tabledata
    self.__extract_attr_descs_from_tabledata(
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/simplesqlite/core.py", line 1603, in __extract_attr_descs_from_tabledata
    six.iteritems(self.__extract_col_type_from_tabledata(table_data))
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/simplesqlite/core.py", line 1633, in __extract_col_type_from_tabledata
    for col_idx, col_dp in enumerate(table_data.column_dp_list)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/tabledata/_core.py", line 145, in column_dp_list
    return self.__dp_extractor.to_column_dp_list(self.value_dp_matrix)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/tabledata/_core.py", line 133, in value_dp_matrix
    self.__value_dp_matrix = self.__dp_extractor.to_dp_matrix(
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/dataproperty/_extractor.py", line 437, in to_dp_matrix
    return self.__to_dp_matrix_mt(value_matrix)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/dataproperty/_extractor.py", line 574, in __to_dp_matrix_mt
    executor.shutdown()
UnboundLocalError: local variable 'executor' referenced before assignment

[thombashi]

@danieldjewell
Thank you for your report.

The latest version of sqlitebiter (0.29.0) can execute properly at Termux.
Please try to upgrade the package and re-execute.