[Failing Test] ci-kubernetes-kubemark-gce-scale-scheduler | kubemark-5000-scheduler

[pacoxu]

Which jobs are failing?

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale-scheduler/1766766416834334720

Which tests are failing?

• kubetest.Kubemark
• kubetest.Kubemark Up

Since when has it been failing?

3 failures in a row since 03-06
lastpass was @ 03-04
before that 3 failures.

Testgrid link

https://testgrid.k8s.io/sig-scalability-kubemark#kubemark-5000-scheduler

Reason for failure (if possible)

WARNING: Could not store access token in cache: database is locked
ERROR: gcloud crashed (OperationalError): database is locked

If you would like to report this issue, please run the following command:
  gcloud feedback

To check gcloud for common problems, please run the following command:
  gcloud info --run-diagnostics
Traceback (most recent call last):
  File "/google-cloud-sdk/lib/gcloud.py", line 189, in <module>
    main()
  File "/google-cloud-sdk/lib/gcloud.py", line 185, in main
    sys.exit(gcloud_main.main())
             ^^^^^^^^^^^^^^^^^^
  File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/crash_handling.py", line 199, in Wrapper
    target_function(*args, **kwargs)
  File "/google-cloud-sdk/lib/googlecloudsdk/gcloud_main.py", line 186, in main
    gcloud_cli.Execute()
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 1024, in Execute
    self._HandleAllErrors(exc, command_path_string, specified_arg_names)
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 1061, in _HandleAllErrors
ERROR: gcloud crashed (OperationalError): database is locked
    exceptions.HandleError(exc, command_path_string, self.__known_error_handler)
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/exceptions.py", line 557, in HandleError
    core_exceptions.reraise(exc)
  File "/google-cloud-sdk/lib/googlecloudsdk/core/exceptions.py", line 146, in reraise
    six.reraise(type(exc_value), exc_value, tb)
  File "/google-cloud-sdk/lib/third_party/six/__init__.py", line 719, in reraise
...
sqlite3.OperationalError: database is locked
WARNING: Could not store access token in cache: database is locked
WARNING: Could not store access token in cache: database is locked

Anything else we need to know?

No response

Relevant SIG(s)

/sig scalibility scheduling

[k8s-ci-robot]

@pacoxu: The label(s) sig/scalibility cannot be applied, because the repository doesn't have them.

In response to this:

Which jobs are failing?

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale-scheduler/1766766416834334720

Which tests are failing?

• kubetest.Kubemark
• kubetest.Kubemark Up

Since when has it been failing?

3 failures in a row since 03-06
lastpass was @ 03-04
before that 3 failures.

Testgrid link

https://testgrid.k8s.io/sig-scalability-kubemark#kubemark-5000-scheduler

Reason for failure (if possible)

WARNING: Could not store access token in cache: database is locked
ERROR: gcloud crashed (OperationalError): database is locked

If you would like to report this issue, please run the following command:
 gcloud feedback

To check gcloud for common problems, please run the following command:
 gcloud info --run-diagnostics
Traceback (most recent call last):
 File "/google-cloud-sdk/lib/gcloud.py", line 189, in <module>
   main()
 File "/google-cloud-sdk/lib/gcloud.py", line 185, in main
   sys.exit(gcloud_main.main())
            ^^^^^^^^^^^^^^^^^^
 File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/crash_handling.py", line 199, in Wrapper
   target_function(*args, **kwargs)
 File "/google-cloud-sdk/lib/googlecloudsdk/gcloud_main.py", line 186, in main
   gcloud_cli.Execute()
 File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 1024, in Execute
   self._HandleAllErrors(exc, command_path_string, specified_arg_names)
 File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 1061, in _HandleAllErrors
ERROR: gcloud crashed (OperationalError): database is locked
   exceptions.HandleError(exc, command_path_string, self.__known_error_handler)
 File "/google-cloud-sdk/lib/googlecloudsdk/calliope/exceptions.py", line 557, in HandleError
   core_exceptions.reraise(exc)
 File "/google-cloud-sdk/lib/googlecloudsdk/core/exceptions.py", line 146, in reraise
   six.reraise(type(exc_value), exc_value, tb)
 File "/google-cloud-sdk/lib/third_party/six/__init__.py", line 719, in reraise
...
sqlite3.OperationalError: database is locked
WARNING: Could not store access token in cache: database is locked
WARNING: Could not store access token in cache: database is locked

Anything else we need to know?

No response

Relevant SIG(s)

/sig scalibility scheduling

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pacoxu]

/sig scalability

[kerthcet]

Is this related to kube-scheduler, didn't see related logs.

[pacoxu]

/cc @wojtek-t
Do you have any clue about this failure?