Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable MYSQL scheduler on discovery api #1596

Closed
zolia opened this issue Jan 30, 2020 · 3 comments
Closed

Disable MYSQL scheduler on discovery api #1596

zolia opened this issue Jan 30, 2020 · 3 comments

Comments

@zolia
Copy link
Contributor

zolia commented Jan 30, 2020

Initial name: lock timeout on node availability calculation

2020-01-30 11:26:49 Failed to process node availabilities:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1226, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 148, in do_executemany
    rowcount = cursor.executemany(statement, parameters)
  File "/usr/local/lib/python3.6/site-packages/pymysql/cursors.py", line 197, in executemany
    self._get_db().encoding)
  File "/usr/local/lib/python3.6/site-packages/pymysql/cursors.py", line 234, in _do_execute_many
    rows += self.execute(sql + postfix)
  File "/usr/local/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute
    result = self._query(query)
  File "/usr/local/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query
    conn.query(q)
  File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 517, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 732, in _read_query_result
    result.read()
  File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 1075, in read
    first_packet = self.connection._read_packet()
  File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 684, in _read_packet
    packet.check_error()
  File "/usr/local/lib/python3.6/site-packages/pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/local/lib/python3.6/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.InternalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/code/api/node_availability_worker.py", line 24, in process_node_availabilities
    db_session.bulk_save_objects(batch)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2720, in bulk_save_objects
    False,
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2908, in _bulk_save_mappings
    transaction.rollback(_capture_exception=True)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2902, in _bulk_save_mappings
    render_nulls,
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 102, in _bulk_insert
    bookkeeping=return_defaults,
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1084, in _emit_insert_statements
    c = cached_connections[connection].execute(statement, multiparams)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 982, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
    distilled_params,
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1250, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1226, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 148, in do_executemany
    rowcount = cursor.executemany(statement, parameters)
  File "/usr/local/lib/python3.6/site-packages/pymysql/cursors.py", line 197, in executemany
    self._get_db().encoding)
  File "/usr/local/lib/python3.6/site-packages/pymysql/cursors.py", line 234, in _do_execute_many
    rows += self.execute(sql + postfix)
  File "/usr/local/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute
    result = self._query(query)
  File "/usr/local/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query
    conn.query(q)
  File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 517, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 732, in _read_query_result
    result.read()
  File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 1075, in read
    first_packet = self.connection._read_packet()
  File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 684, in _read_packet
    packet.check_error()
  File "/usr/local/lib/python3.6/site-packages/pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/local/lib/python3.6/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')
[SQL: INSERT INTO node_availability (node_key, date, service_type) VALUES (%(node_key)s, %(date)s, %(service_type)s)]
[parameters: ({'node_key': '0xdf705d92f91266378021dae323fa25685de3f25d', 'date': datetime.datetime(2020, 1, 30, 11, 25, 36, 961709), 'service_type': 'openvpn'}, {'node_key': '0xe2bc14230a48434682f2539b67dcb9c5a88d82c8', 'date': datetime.datetime(2020, 1, 30, 11, 25, 37, 286172), 'service_type': 'openvpn'}, {'node_key': '0x119ff75335726d4709ec55901a85590c6778b3ad', 'date': datetime.datetime(2020, 1, 30, 11, 25, 37, 551028), 'service_type': 'openvpn'}, {'node_key': '0x852f4d9cf0ecccfa4376d2480bea30377e23fd83', 'date': datetime.datetime(2020, 1, 30, 11, 25, 37, 728882), 'service_type': 'noop'}, {'node_key': '0x0b2ec464cbf568c03d8399296b1d55a5214f6446', 'date': datetime.datetime(2020, 1, 30, 11, 25, 38, 104969), 'service_type': 'openvpn'}, {'node_key': '0x9b60d1a6235b7eb9dc10efec2e0c2bf3059ebef7', 'date': datetime.datetime(2020, 1, 30, 11, 25, 38, 747505), 'service_type': 'openvpn'}, {'node_key': '0xd93fb0b84e8254181761a435c4d4b660961726f2', 'date': datetime.datetime(2020, 1, 30, 11, 25, 38, 895495), 'service_type': 'openvpn'}, {'node_key': '0x2a8dc4c943dbb98085eb21ef17f32eb4996b0a64', 'date': datetime.datetime(2020, 1, 30, 11, 25, 39, 278988), 'service_type': 'openvpn'}  ... displaying 10 of 22 total bound parameter sets ...  {'node_key': '0xc5fdb9e2f44a238e18f9d1e32ce255abe540d0fa', 'date': datetime.datetime(2020, 1, 30, 11, 25, 43, 252185), 'service_type': 'openvpn'}, {'node_key': '0x1c676012dec2dae44f5a37f19606c595841b8921', 'date': datetime.datetime(2020, 1, 30, 11, 25, 43, 294876), 'service_type': 'noop'})]
(Background on this error at: http://sqlalche.me/e/2j85)
@zolia zolia added this to Icebox in Engineering via automation Jan 30, 2020
@zolia zolia moved this from Icebox to Todo in Engineering Jan 30, 2020
@zolia
Copy link
Contributor Author

zolia commented Jan 30, 2020

We should remove this availability calculation function since availability will be calculated by mmn now and leaderboard will be removed from dashboard.

@chompomonim chompomonim removed this from Todo in Engineering Feb 20, 2020
@anjmao
Copy link
Contributor

anjmao commented Mar 2, 2020

@ignasbernotas Do we still need this page https://testnet.mysterium.network/leaderboard?

If https://my.mysterium.network/leaderboard doesn't use mysql for availability we can stop availability calculations.

@chompomonim
Copy link
Contributor

O disable it at all.

@chompomonim chompomonim changed the title lock timeout on node availability calculation Disable MYSQL scheduler on discovery api Mar 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants