Delete dataset not working?

[ajstewart]

When I tried to delete a dataset, which I had never tried before, I get the following error:

(trap) ada:~/ada2/CHILES/ada-trap$ trap-manage.py deldataset 8

Are you sure you want to delete dataset 8? [y/N]: y
/suphys/aste7152/.conda/envs/trap/lib/python2.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "/suphys/aste7152/.conda/envs/trap/bin/trap-manage.py", line 10, in <module>
    tkp.management.main()
  File "/suphys/aste7152/.conda/envs/trap/lib/python2.7/site-packages/tkp/management.py", line 361, in main
    args.func(args)
  File "/suphys/aste7152/.conda/envs/trap/lib/python2.7/site-packages/tkp/management.py", line 257, in deldataset
    dataset = db.session.query(Dataset).filter(Dataset.id==options.id).one()
  File "/suphys/aste7152/.conda/envs/trap/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, in one
    ret = self.one_or_none()
  File "/suphys/aste7152/.conda/envs/trap/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, in one_or_none
    ret = list(self)
  File "/suphys/aste7152/.conda/envs/trap/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, in __iter__
    return self._execute_and_instances(context)
  File "/suphys/aste7152/.conda/envs/trap/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2948, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/suphys/aste7152/.conda/envs/trap/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/suphys/aste7152/.conda/envs/trap/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/suphys/aste7152/.conda/envs/trap/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/suphys/aste7152/.conda/envs/trap/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/suphys/aste7152/.conda/envs/trap/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/suphys/aste7152/.conda/envs/trap/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/suphys/aste7152/.conda/envs/trap/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/suphys/aste7152/.conda/envs/trap/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation "dataset" does not exist
LINE 2: FROM dataset
             ^
 [SQL: 'SELECT dataset.id AS dataset_id, dataset.rerun AS dataset_rerun, dataset.type AS dataset_type, dataset.process_start_ts AS dataset_process_start_ts, dataset.process_end_ts AS dataset_process_end_ts, dataset.detection_threshold AS dataset_detection_threshold, dataset.analysis_threshold AS dataset_analysis_threshold, dataset.assoc_radius AS dataset_assoc_radius, dataset.backsize_x AS dataset_backsize_x, dataset.backsize_y AS dataset_backsize_y, dataset.margin_width AS dataset_margin_width, dataset.description AS dataset_description, dataset.node AS dataset_node, dataset.nodes AS dataset_nodes \nFROM dataset \nWHERE dataset.id = %(id_1)s'] [parameters: {'id_1': 8}] (Background on this error at: http://sqlalche.me/e/f405)

[ajstewart]

This error was caused by the pip install tkp not being up-to-date with the current master version here (specifically management.py which was missing the pipeline config load in the current pypi version).

Who can update the version on PyPi?

[gijzelaerr]

4.0 is the current release:

https://pypi.org/project/tkp/4.0/

We only upload released version of trap to pypi, so maybe a 4.1 or 4.0.1 should be made.

[ajstewart]

This confused me as according to the dates of everything it should be fine.

I tested it again earlier and simply installing via pip (pip install tkp) gives a version where the delete dataset is broken - there's a difference in the management.py delete function. When I looked at the code on here I noticed it was different to what I had, so re-installing from a git clone solved the problem. I tried pip install tkp in a new environment again to confirm it was different and it was. Is it possible that some commits got lost at some stage? As the deldataset commit was done in November 2016 so I'm not sure why it's broken with the pip version.

[AntoniaR]

Myself and the team in Amsterdam are planning a new 4.1 release later this year. We will add this as one of the things to resolve in that release.

[gijzelaerr]

there are some differences:

https://github.com/transientskp/tkp/compare/r4.0...master#diff-f1b1a9db1a592973e2ac19e7baa72220

basically, you are mixing the database/code of master release and 4.0, which is not expected to work.

EDIT: had to edit the URL since github formatting messes up the link somehow.

[AntoniaR]

I have set this as an issue to hopefully solve in the upcoming release 5.0

[AntoniaR]

This issue is likely also related to the following issues: #564 #556 So I have added them to the upcoming release 5.0

[AntoniaR]

This issue is resolved when using the matching tkp version for both TraP and the TraP database (i.e. if not sure, make a new database with current TraP version). But you must also not use the latest version of sqlalchemy (see issue #586 ). So, for now, run the following:

pip install sqlalchemy==1.3

As this solves this issue, I am closing the issue.