Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

spanner: InvalidArgument: 400 Previously received a different request with this seqno. seqno=4 with no concurrency applied #11

Closed
odeke-em opened this issue Feb 4, 2020 · 2 comments · Fixed by #19
Closed

spanner: InvalidArgument: 400 Previously received a different request with this seqno. seqno=4 with no concurrency applied #11

odeke-em opened this issue Feb 4, 2020 · 2 comments · Fixed by #19
Assignees
@larkee
Labels
api: spanner Issues related to the googleapis/python-spanner API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@odeke-em
Copy link

odeke-em commented Feb 4, 2020

With spanner_v1 version 1.11, I am using this package without any concurrency and yet I am getting back an obscure error that looks to me like an issue with the underlying gRPC library or the coordination in this library

google.api_core.exceptions.InvalidArgument: 400 Previously received a different request with this seqno. seqno=4

and that finally results in #10 even after I've removed PingingPool to use the default pool.

Please find the full stack trace below in the details element

======================================================================
ERROR: test_xview_class (admin_docs.test_middleware.XViewMiddlewareTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/grpc/_channel.py", line 824, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/grpc/_channel.py", line 726, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Previously received a different request with this seqno. seqno=4"
	debug_error_string = "{"created":"@1580855981.982223904","description":"Error received from peer ipv4:108.177.13.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Previously received a different request with this seqno. seqno=4","grpc_status":3}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 89, in execute
    self.__handle_update(self.__get_txn(), sql, args or None)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 101, in __handle_update
    res = txn.execute_update(sql, params=params, param_types=get_param_types(params))
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/cloud/spanner_v1/transaction.py", line 202, in execute_update
    metadata=metadata,
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/cloud/spanner_v1/gapic/spanner_client.py", line 810, in execute_sql
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    on_error=on_error,
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 Previously received a different request with this seqno. seqno=4
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 93, in execute
    raise ProgrammingError(e.details if hasattr(e, 'details') else e)
spanner.dbapi.exceptions.ProgrammingError: 400 Previously received a different request with this seqno. seqno=4
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/test/testcases.py", line 267, in __call__
    self._pre_setup()
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/test/testcases.py", line 938, in _pre_setup
    self._fixture_setup()
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/test/testcases.py", line 1165, in _fixture_setup
    self.setUpTestData()
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/tests/admin_docs/tests.py", line 11, in setUpTestData
    cls.superuser = User.objects.create_superuser(username='super', password='secret', email='super@example.com')
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/contrib/auth/models.py", line 162, in create_superuser
    return self._create_user(username, email, password, **extra_fields)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/contrib/auth/models.py", line 145, in _create_user
    user.save(using=self._db)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/contrib/auth/base_user.py", line 66, in save
    super().save(*args, **kwargs)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/base.py", line 741, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/base.py", line 779, in save_base
    force_update, using, update_fields,
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/base.py", line 851, in _save_table
    forced_update)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/base.py", line 900, in _do_update
    return filtered._update(values) > 0
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/query.py", line 760, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/sql/compiler.py", line 1462, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/sql/compiler.py", line 1133, in execute_sql
    cursor.execute(sql, params)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 93, in execute
    raise ProgrammingError(e.details if hasattr(e, 'details') else e)
django.db.utils.ProgrammingError: 400 Previously received a different request with this seqno. seqno=4
======================================================================
ERROR: test_xview_func (admin_docs.test_middleware.XViewMiddlewareTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/grpc/_channel.py", line 824, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/grpc/_channel.py", line 726, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Previously received a different request with this seqno. seqno=4"
	debug_error_string = "{"created":"@1580855981.991133481","description":"Error received from peer ipv4:108.177.13.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Previously received a different request with this seqno. seqno=4","grpc_status":3}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 89, in execute
    self.__handle_update(self.__get_txn(), sql, args or None)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 101, in __handle_update
    res = txn.execute_update(sql, params=params, param_types=get_param_types(params))
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/cloud/spanner_v1/transaction.py", line 202, in execute_update
    metadata=metadata,
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/cloud/spanner_v1/gapic/spanner_client.py", line 810, in execute_sql
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    on_error=on_error,
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 Previously received a different request with this seqno. seqno=4
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 93, in execute
    raise ProgrammingError(e.details if hasattr(e, 'details') else e)
spanner.dbapi.exceptions.ProgrammingError: 400 Previously received a different request with this seqno. seqno=4
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/test/testcases.py", line 267, in __call__
    self._pre_setup()
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/test/testcases.py", line 938, in _pre_setup
    self._fixture_setup()
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/test/testcases.py", line 1165, in _fixture_setup
    self.setUpTestData()
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/tests/admin_docs/tests.py", line 11, in setUpTestData
    cls.superuser = User.objects.create_superuser(username='super', password='secret', email='super@example.com')
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/contrib/auth/models.py", line 162, in create_superuser
    return self._create_user(username, email, password, **extra_fields)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/contrib/auth/models.py", line 145, in _create_user
    user.save(using=self._db)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/contrib/auth/base_user.py", line 66, in save
    super().save(*args, **kwargs)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/base.py", line 741, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/base.py", line 779, in save_base
    force_update, using, update_fields,
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/base.py", line 851, in _save_table
    forced_update)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/base.py", line 900, in _do_update
    return filtered._update(values) > 0
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/query.py", line 760, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/sql/compiler.py", line 1462, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/sql/compiler.py", line 1133, in execute_sql
    cursor.execute(sql, params)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 93, in execute
    raise ProgrammingError(e.details if hasattr(e, 'details') else e)
django.db.utils.ProgrammingError: 400 Previously received a different request with this seqno. seqno=4
======================================================================
ERROR: test_bookmarklets (admin_docs.test_views.AdminDocViewTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/grpc/_channel.py", line 824, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/grpc/_channel.py", line 726, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Previously received a different request with this seqno. seqno=4"
	debug_error_string = "{"created":"@1580855982.000827256","description":"Error received from peer ipv4:108.177.13.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Previously received a different request with this seqno. seqno=4","grpc_status":3}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 89, in execute
    self.__handle_update(self.__get_txn(), sql, args or None)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 101, in __handle_update
    res = txn.execute_update(sql, params=params, param_types=get_param_types(params))
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/cloud/spanner_v1/transaction.py", line 202, in execute_update
    metadata=metadata,
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/cloud/spanner_v1/gapic/spanner_client.py", line 810, in execute_sql
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    on_error=on_error,
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 Previously received a different request with this seqno. seqno=4
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 93, in execute
    raise ProgrammingError(e.details if hasattr(e, 'details') else e)
spanner.dbapi.exceptions.ProgrammingError: 400 Previously received a different request with this seqno. seqno=4
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/test/testcases.py", line 267, in __call__
    self._pre_setup()
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/test/testcases.py", line 938, in _pre_setup
    self._fixture_setup()
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/test/testcases.py", line 1165, in _fixture_setup
    self.setUpTestData()
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/tests/admin_docs/tests.py", line 11, in setUpTestData
    cls.superuser = User.objects.create_superuser(username='super', password='secret', email='super@example.com')
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/contrib/auth/models.py", line 162, in create_superuser
    return self._create_user(username, email, password, **extra_fields)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/contrib/auth/models.py", line 145, in _create_user
    user.save(using=self._db)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/contrib/auth/base_user.py", line 66, in save
    super().save(*args, **kwargs)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/base.py", line 741, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/base.py", line 779, in save_base
    force_update, using, update_fields,
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/base.py", line 851, in _save_table
    forced_update)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/base.py", line 900, in _do_update
    return filtered._update(values) > 0
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/query.py", line 760, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/sql/compiler.py", line 1462, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/models/sql/compiler.py", line 1133, in execute_sql
    cursor.execute(sql, params)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 93, in execute
    raise ProgrammingError(e.details if hasattr(e, 'details') else e)
django.db.utils.ProgrammingError: 400 Previously received a different request with this seqno. seqno=4
======================================================================
ERROR: test_index (admin_docs.test_views.AdminDocViewTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
 ...
    raise ProgrammingError(e.details if hasattr(e, 'details') else e)
django.db.utils.ProgrammingError: 400 Previously received a different request with this seqno. seqno=4
======================================================================
ERROR: test_missing_docutils (admin_docs.test_views.AdminDocViewTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
...
django.db.utils.ProgrammingError: 400 Previously received a different request with this seqno. seqno=4
...
django.db.utils.ProgrammingError: 400 Previously received a different request with this seqno. seqno=4
======================================================================
ERROR: test_model_with_many_to_one (admin_docs.test_views.TestModelDetailView)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
...
django.db.utils.ProgrammingError: 400 Previously received a different request with this seqno. seqno=4
======================================================================
ERROR: test_model_with_no_backward_relations_render_only_relevant_fields (admin_docs.test_views.TestModelDetailView)
----------------------------------------------------------------------
...<SAME CONTENT>
django.db.utils.ProgrammingError: 400 Previously received a different request with this seqno. seqno=4
----------------------------------------------------------------------
Ran 13 tests in 0.927s
FAILED (errors=45)
Testing against Django installed in '/home/travis/build/orijtech/spanner-orm/django_tests/django/django' with up to 2 processes
Importing application admin_default_site
Importing application admin_docs
Skipping setup of unused database(s): other.
Operations to perform:
  Synchronize unmigrated apps: admin_default_site, admin_docs, auth, contenttypes, messages, sessions, staticfiles
  Apply all migrations: admin, sites
Synchronizing apps without migrations:
  Creating tables...
    Creating table django_content_type
    Creating table auth_permission
    Creating table auth_group
    Creating table auth_user
    Creating table django_session
    Creating table admin_docs_company
    Creating table admin_docs_group
    Creating table admin_docs_family
    Creating table admin_docs_person
    Running deferred SQL...
Running migrations:
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying sites.0001_initial... OK
  Applying sites.0002_alter_domain_unique... OK
System check identified no issues (0 silenced).
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/grpc/_channel.py", line 824, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/grpc/_channel.py", line 726, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.ABORTED
	details = "Transaction not found"
	debug_error_string = "{"created":"@1580855984.140864504","description":"Error received from peer ipv4:108.177.13.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Transaction not found","grpc_status":10}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "runtests.py", line 507, in <module>
    options.exclude_tags,
  File "runtests.py", line 294, in django_tests
    extra_tests=extra_tests,
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/test/runner.py", line 639, in run_tests
    self.teardown_databases(old_config)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/test/runner.py", line 583, in teardown_databases
    keepdb=self.keepdb,
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/test/utils.py", line 299, in teardown_databases
    connection.creation.destroy_test_db(old_name, verbosity, keepdb)
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/base/creation.py", line 241, in destroy_test_db
    self.connection.close()
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/base/base.py", line 288, in close
    self._close()
  File "/home/travis/build/orijtech/spanner-orm/django_tests/django/django/db/backends/base/base.py", line 250, in _close
    return self.connection.close()
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/connection.py", line 30, in close
    self.commit()
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/spanner/dbapi/connection.py", line 60, in commit
    res = self.__txn.commit()
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/cloud/spanner_v1/transaction.py", line 127, in commit
    metadata=metadata,
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/cloud/spanner_v1/gapic/spanner_client.py", line 1556, in commit
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    on_error=on_error,
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.Aborted: 409 Transaction not found
The command "bash django_test_suite.sh" exited with 1.
cache.2
store build cache
@product-auto-label product-auto-label bot added the api: spanner Issues related to the googleapis/python-spanner API. label Feb 4, 2020
@odeke-em
Copy link
Author

odeke-em commented Feb 5, 2020

@skuruppu @larkee I am getting this error quite a lot with the spanner-django project as per https://travis-ci.com/orijtech/spanner-orm/jobs/283644877#L611 which you have access to with your Github accounts.

@skuruppu skuruppu added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Feb 5, 2020
@larkee larkee added the status: investigating The issue is under investigation, which is determined to be non-trivial. label Feb 12, 2020
@larkee
Copy link
Contributor

larkee commented Feb 12, 2020 •
edited

I have managed to run into this same error without concurrency:

from google.cloud import spanner

client = spanner.Client()
instance = client.instance(INSTANCE)
database = instance.database(DATABASE)

sess = database.session()
sess.create()

txn = sess.transaction()

txn.begin()
try:
    txn.execute_update(
        "INSERT Singers (SingerId, FirstName, LastName) VALUES "
        "(10, 'Taylor', 'Swift'),"
    )
except Exception as e:
    print("error:")
    print(e)

txn.execute_update(
    "INSERT Singers (SingerId, FirstName, LastName) VALUES "
    "(10, 'Taylor', 'Swift')"
)

txn.commit()

The output is:

error:
400 Syntax error: Expected \"(\" but got end of statement [at 1:79]\nINSERT Singers (SingerId, FirstName, LastName) VALUES (10, \'Taylor\', \'Swift\'),\n

and raises:

google.api_core.exceptions.InvalidArgument: 400 Previously received a different request with this seqno. seqno=0

The problem seems to arise when a execute_sql, execute_batch_dml, or execute_streaming_sql call made by a Transaction or Snapshot raises an exception as it fails to increment _execute_sql_count. I am looking into a possible fix.

Thanks for finding this!

@larkee larkee removed the status: investigating The issue is under investigation, which is determined to be non-trivial. label Feb 12, 2020
@odeke-em odeke-em mentioned this issue Feb 13, 2020
Merged
@larkee larkee mentioned this issue Feb 18, 2020
Merged
@odeke-em odeke-em mentioned this issue Feb 28, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@larkee larkee

Labels
api: spanner Issues related to the googleapis/python-spanner API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: increment seqno before execute calls to prevent InvalidArgument … larkee/python-spanner
3 participants
@skuruppu @odeke-em @larkee