ValueError: Unrecognised argument(s): force

[kinzhong]

Describe the bug
ValueError when running Hello World Hunt using Python 3.6.9. Installed using pip install kestrel-lang.

Details of the bug

• What is the hunt flow/script you are executing?
  Hello World Hunt from readme
• What is the command that failed?

$ kestrel helloworld.hf

• What is the error message?

$ kestrel helloworld.hf
Traceback (most recent call last):
  File "/usr/local/bin/kestrel", line 8, in <module>
    runpy.run_module('kestrel', run_name='__main__')
  File "/usr/lib/python3.6/runpy.py", line 208, in run_module
    return _run_code(code, {}, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/kestrel/__main__.py", line 49, in <module>
    logging_setup(None, args.verbose, args.debug)
  File "/usr/local/lib/python3.6/dist-packages/kestrel/__main__.py", line 33, in logging_setup
    force=True,
  File "/usr/lib/python3.6/logging/__init__.py", line 1829, in basicConfig
    raise ValueError('Unrecognised argument(s): %s' % keys)
ValueError: Unrecognised argument(s): force

To Reproduce

1. pip install --upgrade pip setuptools wheel
2. pip install kestrel-lang
3. kestrel helloworld.hf

Expected behavior
Output from hunt flow.

Environment (please complete the following information):

• OS: Ubuntu 18.04
• Python version: Python 3.6.9
• Python install environment: Python virtual environment
• STIX-Shifter version: 3.5.0

[subbyte]

Possible cause: argument force is added to logging in version 3.8 according to basicConfig()

Would you want to have a try fixing it @kinzhong ? I think we just need to manually remove handlers attached to the root logger to avoid using the new parameter in basicConfig.

[kinzhong]

Yes sure. I removed the argument force which fixes the bug.

However, I got a sqlite3.OperationalError: near "ON": syntax error, logs can be seen below. This is similar to #71. Python 3.6 comes prebuilt with sqlite3 version 3.22.0 but firepit requires > 3.24.0. For anyone facing this issue, the fix can be found here.

Logs:

$ kestrel helloworld.hf 
Traceback (most recent call last):
  File "/home/kinzhong/.local/bin/kestrel", line 8, in <module>
    runpy.run_module('kestrel', run_name='__main__')
  File "/usr/lib/python3.6/runpy.py", line 208, in run_module
    return _run_code(code, {}, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/kinzhong/.local/lib/python3.6/site-packages/kestrel/__main__.py", line 55, in <module>
    outputs = session.execute(huntflow)
  File "/home/kinzhong/.local/lib/python3.6/site-packages/kestrel/session.py", line 266, in execute
    return self._execute_ast(ast)
  File "/home/kinzhong/.local/lib/python3.6/site-packages/kestrel/session.py", line 441, in _execute_ast
    output_var_struct, display = execute_cmd(stmt, self)
  File "/home/kinzhong/.local/lib/python3.6/site-packages/kestrel/codegen/commands.py", line 92, in wrapper
    return func(stmt, session)
  File "/home/kinzhong/.local/lib/python3.6/site-packages/kestrel/codegen/commands.py", line 60, in wrapper
    ret = func(stmt, session)
  File "/home/kinzhong/.local/lib/python3.6/site-packages/kestrel/codegen/commands.py", line 123, in new
    stmt["type"] = load_data(session.store, stmt["output"], stmt["data"], stmt["type"])
  File "/home/kinzhong/.local/lib/python3.6/site-packages/kestrel/codegen/data.py", line 30, in load_data
    store.load(output_entity_table, data, entity_type, query_id)
  File "/home/kinzhong/.local/lib/python3.6/site-packages/firepit/sqlstorage.py", line 334, in load
    splitter.close()
  File "/home/kinzhong/.local/lib/python3.6/site-packages/firepit/splitter.py", line 210, in close
    self.writer.write_records(obj_type, recs, self.schemas[obj_type], self.replace, self.query_id)
  File "/home/kinzhong/.local/lib/python3.6/site-packages/firepit/splitter.py", line 125, in write_records
    self.store.upsert_many(cursor, tablename, records, query_id)
  File "/home/kinzhong/.local/lib/python3.6/site-packages/firepit/sqlstorage.py", line 269, in upsert_many
    self.upsert(cursor, tablename, obj, query_id)
  File "/home/kinzhong/.local/lib/python3.6/site-packages/firepit/sqlstorage.py", line 260, in upsert
    cursor.execute(stmt, values)
sqlite3.OperationalError: near "ON": syntax error

[pcoccoli]

How are you installing python? I manage python versions with pyenv; it downloads and compiles the python version you specify, so it gets whatever version of sqlite3 you have on your system. On my RHEL8 box, I have sqlite-3.26.0-13.el8.x86_64 so:

$ python
Python 3.6.8 (default, Sep  1 2020, 08:39:30) 
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.26.0'

Kestrel and firepit work fine with this installation of python 3.6.

[kinzhong]

I was using the python version that came pre-installed by Ubuntu 18.04. I see, thank you for the recommendation.