Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Critical error when trying to set memory limit #1137

Closed
tudurom opened this issue Oct 4, 2019 · 2 comments
Closed

Critical error when trying to set memory limit #1137

tudurom opened this issue Oct 4, 2019 · 2 comments

Comments

@tudurom
Copy link

tudurom commented Oct 4, 2019

Issue template

Description: when trying to set the memory limit, a critical error appears.

Additionally, the memory limit is a very high bogus number, such as 1073741824MB.

Steps to reproduce:

  1. Open task page
  2. Set memory limit
  3. Click update

Expected: The memory limit changes

Actual: Critical error shows, memory limit is bogus.

Logs

Traceback (most recent call last):
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
psycopg2.IntegrityError: new row for relation "datasets" violates check constraint "datasets_memory_limit_check1"
DETAIL:  Failing row contains (2, 2, Default, t, 3, 1, Communication, [1, "stub", "fifo_io"], GroupMul, [[15, "tiny*"], [35, "medium*"], [50, "big*"]]).


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/rmi/cms_env/lib/python3.6/site-packages/tornado/web.py", line 1510, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/tornado/web.py", line 2898, in wrapper
    return method(self, *args, **kwargs)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/cms-1.4rc1-py3.6.egg/cms/server/admin/handlers/base.py", line 188, in newfunc
    return func(self, *args, **kwargs)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/cms-1.4rc1-py3.6.egg/cms/server/admin/handlers/task.py", line 199, in post
    for testcase in itervalues(dataset.testcases):
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 237, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 584, in get
    value = self.callable_(state, passive)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 557, in _load_for_state
    return self._emit_lazyload(session, state, ident_key, passive)
  File "<string>", line 1, in <lambda>
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 635, in _emit_lazyload
    result = q.all()
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2703, in all
    return list(self)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2854, in __iter__
    self.session._autoflush()
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1428, in _autoflush
    util.raise_from_cause(e)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1418, in _autoflush
    self.flush()
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2209, in flush
    self._flush(objects)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2329, in _flush
  transaction.rollback(_capture_exception=True)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2293, in _flush
    flush_context.execute()
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
    rec.execute(self)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
    uow
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj
    mapper, table, update)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 737, in _emit_update_statements
    execute(statement, multiparams)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
    context)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
    exc_info
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (psycopg2.IntegrityError) new row for relation "datasets" violates check constraint "datasets_memory_limit_check1"
DETAIL:  Failing row contains (2, 2, Default, t, 3, 1, Communication, [1, "stub", "fifo_io"], GroupMul, [[15, "tiny*"], [35, "medium*"], [50, "big*"]]).
 [SQL: 'UPDATE datasets SET memory_limit=%(memory_limit)s WHERE datasets.id = %(datasets_id)s'] [parameters: {'memory_limit': 1, 'datasets_id': 2}]
2019-10-04 17:00:54,015 - ERROR [Admin,0 63 base::write_error] Uncaught exception (IntegrityError('(psycopg2.IntegrityError) new row for relation "datasets" violates check constraint "datasets_memory_limit_check1"\nDETAIL:  Failing row contains (2, 2, Default, t, 3, 1, Communication, [1, "stub", "fifo_io"], GroupMul, [[15, "tiny*"], [35, "medium*"], [50, "big*"]]).\n',)) while processing a request: Traceback (most recent call last):
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
psycopg2.IntegrityError: new row for relation "datasets" violates check constraint "datasets_memory_limit_check1"
DETAIL:  Failing row contains (2, 2, Default, t, 3, 1, Communication, [1, "stub", "fifo_io"], GroupMul, [[15, "tiny*"], [35, "medium*"], [50, "big*"]]).


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/rmi/cms_env/lib/python3.6/site-packages/tornado/web.py", line 1510, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/tornado/web.py", line 2898, in wrapper
    return method(self, *args, **kwargs)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/cms-1.4rc1-py3.6.egg/cms/server/admin/handlers/base.py", line 188, in newfunc
    return func(self, *args, **kwargs)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/cms-1.4rc1-py3.6.egg/cms/server/admin/handlers/task.py", line 199, in post
    for testcase in itervalues(dataset.testcases):
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 237, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 584, in get
    value = self.callable_(state, passive)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 557, in _load_for_state
    return self._emit_lazyload(session, state, ident_key, passive)
  File "<string>", line 1, in <lambda>
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 635, in _emit_lazyload
    result = q.all()
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2703, in all
    return list(self)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2854, in __iter__
    self.session._autoflush()
File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1428, in _autoflush
    util.raise_from_cause(e)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1418, in _autoflush
    self.flush()
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2209, in flush
    self._flush(objects)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2329, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2293, in _flush
    flush_context.execute()
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
    rec.execute(self)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
    uow
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj
    mapper, table, update)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 737, in _emit_update_statements
    execute(statement, multiparams)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
    context)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
    exc_info
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/home/rmi/cms_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (psycopg2.IntegrityError) new row for relation "datasets" violates check constraint "datasets_memory_limit_check1"
DETAIL:  Failing row contains (2, 2, Default, t, 3, 1, Communication, [1, "stub", "fifo_io"], GroupMul, [[15, "tiny*"], [35, "medium*"], [50, "big*"]]).
 [SQL: 'UPDATE datasets SET memory_limit=%(memory_limit)s WHERE datasets.id = %(datasets_id)s'] [parameters: {'memory_limit': 1, 'datasets_id': 2}]

System Information

CMS version: master
Was CMS installed: yes
Using a virtualenv: yes
Postgres version: 10+190
Distribution and version: Ubuntu 18.04
@tudurom
Copy link
Author

tudurom commented Oct 4, 2019

It might have happened because I didn't migrate the DB after updating.

@andreyv
Copy link
Member

andreyv commented Dec 7, 2021

This is most likely related to 7bdc1af. Observe that 1073741824 = 1024 × 1024².

While the description mentions master, the backtrace is from 1.4rc1. At the same time, constraint name datasets_memory_limit_check1 implies that there are at least two constraints on memory_limit in the database, while there was only one before 7bdc1af.

It is not fully clear, but something like this might have happened:

  1. Database was initialized by CMS master.
  2. Memory limit for the dataset was set to 1024 MiB.
  3. CMS 1.4rc1 was started without re-initializing the database.
  4. In the old AWS, which assumes memory limits are in MiB, the actual field would display as 1024 × 1024² = 1073741824.
  5. In the old AWS, attempting to set a sensible memory limit (looks like it is 1 MiB in the failing query) would violate the new 7bdc1af constraint that memory limit in bytes must be divisible by 1024².

The solution is to run both new cmsInitDB and new CMS, then it should work.

Closing. Feel free to reopen if you still observe this on master.

@andreyv andreyv closed this as completed Dec 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@andreyv @tudurom