Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

K8s Helm release-2.0.0 CVAT not logging in #4438

Closed
2 tasks done
HarrisDePerceptron opened this issue Mar 10, 2022 · 6 comments
Closed
2 tasks done

K8s Helm release-2.0.0 CVAT not logging in #4438

HarrisDePerceptron opened this issue Mar 10, 2022 · 6 comments

Comments

@HarrisDePerceptron
Copy link

My actions before raising this issue

Testing out release on k8s using the helm chart from branch release-2.0.0. We already have a working deployment for release-1.7.0. This release deploys ok but once we login to the super user/user it throughs a strage error. perticularly :

GET https://cvat.snapthat.ai/api/users/self?org= 500
cvat-app.tsx:283 Error: Request failed with status code 500. "\n<!doctype html>\n<html lang=\"en\">\n<head>\n  <title>Server Error (500)</title>\n</head>\n<body>\n  <h1>Server Error (500)</h1><p></p>\n</body>\n</html>\n".

internal error 3

Expected Behaviour

should be able to log in to the super user without issue

Current Behaviour

Possible Solution

Looking at the git diff and the docker-compose file a seems like a new service was introduced openpolicyagent/opa. but i do not see this in the deployment charts or any mention of it any where. could this have been casuse due to this missing service. Also see the logs below

Steps to Reproduce (for bugs)

  • git checkout release-2.0.0
  • cd cvat/helm-chart
  • helm upgrade --install --create-namespace cvat -f values.yaml -f values.override.yaml --namespace cvat .
  • Login to frontend on the exposed endpoint

Context

Was trying to upgrade cvat from 1.7.0 to 2.0.0 on k8s.

Your Environment

  • Git hash commit (git log -1): commit 8aaa450 (HEAD -> release-2.0.0, origin/release-2.0.0)

  • Docker version docker version (e.g. Docker 17.0.05): not applicable

  • Are you using Docker Swarm or Kubernetes?: yes. using k8s

  • Operating System and version (e.g. Linux, Windows, MacOS): Linux, ubunut

  • Code example or link to GitHub repo or gist to reproduce problem: see above steps to reproduce

  • Other diagnostic information / logs:

2022-03-09 19:28:30,017 DEBG 'runserver' stderr output:
[Wed Mar 09 19:28:30.016914 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] [2022-03-09 19:28:30,016] ERROR django.request: Internal Server Error: /api/users/self
[Wed Mar 09 19:28:30.016931 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] Traceback (most recent call last):
[Wed Mar 09 19:28:30.016934 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
[Wed Mar 09 19:28:30.016938 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     conn = connection.create_connection(
[Wed Mar 09 19:28:30.016940 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 72, in create_connection
[Wed Mar 09 19:28:30.016943 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
[Wed Mar 09 19:28:30.016945 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/usr/lib/python3.8/socket.py", line 918, in getaddrinfo
[Wed Mar 09 19:28:30.016948 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
[Wed Mar 09 19:28:30.016950 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] socket.gaierror: [Errno -2] Name or service not known
[Wed Mar 09 19:28:30.016952 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] 
[Wed Mar 09 19:28:30.016955 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] During handling of the above exception, another exception occurred:
[Wed Mar 09 19:28:30.016957 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] 
[Wed Mar 09 19:28:30.016959 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] Traceback (most recent call last):
[Wed Mar 09 19:28:30.016962 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
[Wed Mar 09 19:28:30.016965 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     httplib_response = self._make_request(
[Wed Mar 09 19:28:30.016967 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
[Wed Mar 09 19:28:30.016970 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     conn.request(method, url, **httplib_request_kw)
[Wed Mar 09 19:28:30.016972 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
[Wed Mar 09 19:28:30.016975 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     super(HTTPConnection, self).request(method, url, body=body, headers=headers)

2022-03-09 19:28:30,017 DEBG 'runserver' stderr output:
[Wed Mar 09 19:28:30.016977 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/usr/lib/python3.8/http/client.py", line 1256, in request
[Wed Mar 09 19:28:30.016980 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     self._send_request(method, url, body, headers, encode_chunked)
[Wed Mar 09 19:28:30.016983 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
[Wed Mar 09 19:28:30.016986 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     self.endheaders(body, encode_chunked=encode_chunked)
[Wed Mar 09 19:28:30.016988 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
[Wed Mar 09 19:28:30.016991 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     self._send_output(message_body, encode_chunked=encode_chunked)
[Wed Mar 09 19:28:30.016993 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
[Wed Mar 09 19:28:30.016995 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     self.send(msg)
[Wed Mar 09 19:28:30.016998 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/usr/lib/python3.8/http/client.py", line 951, in send
[Wed Mar 09 19:28:30.017000 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     self.connect()
[Wed Mar 09 19:28:30.017002 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
[Wed Mar 09 19:28:30.017005 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     conn = self._new_conn()
[Wed Mar 09 19:28:30.017007 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
[Wed Mar 09 19:28:30.017010 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     raise NewConnectionError(
[Wed Mar 09 19:28:30.017012 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f13615fc460>: Failed to establish a new connection: [Errno -2] Name or service not known
[Wed Mar 09 19:28:30.017015 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] 
[Wed Mar 09 19:28:30.017017 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] During handling of the above exception, another exception occurred:
[Wed Mar 09 19:28:30.017020 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] 
[Wed Mar 09 19:28:30.017022 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] Traceback (most recent call last):
[Wed Mar 09 19:28:30.017024 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
[Wed Mar 09 19:28:30.017027 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     resp = conn.urlopen(
[Wed Mar 09 19:28:30.017029 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
[Wed Mar 09 19:28:30.017032 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     retries = retries.increment(
[Wed Mar 09 19:28:30.017034 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
[Wed Mar 09 19:28:30.017037 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     raise MaxRetryError(_pool, url, error or ResponseError(cause))
[Wed Mar 09 19:28:30.017039 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='opa', port=8181): Max retries exceeded with url: /v1/data/users/allow (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f13615fc460>: Failed to establish a new connection: [Errno -2] Name or service not known'))
[Wed Mar 09 19:28:30.017045 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] 
[Wed Mar 09 19:28:30.017048 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] During handling of the above exception, another exception occurred:
[Wed Mar 09 19:28:30.017050 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] 
[Wed Mar 09 19:28:30.017053 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] Traceback (most recent call last):
[Wed Mar 09 19:28:30.017055 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
[Wed Mar 09 19:28:30.017058 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     response = get_response(request)
[Wed Mar 09 19:28:30.017061 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
[Wed Mar 09 19:28:30.017063 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     response = wrapped_callback(request, *callback_args, **callback_kwargs)
[Wed Mar 09 19:28:30.017066 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
[Wed Mar 09 19:28:30.017068 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     return view_func(*args, **kwargs)
[Wed Mar 09 19:28:30.017071 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
[Wed Mar 09 19:28:30.017074 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     return self.dispatch(request, *args, **kwargs)
[Wed Mar 09 19:28:30.017076 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
[Wed Mar 09 19:28:30.017079 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     response = self.handle_exception(exc)
[Wed Mar 09 19:28:30.017081 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
[Wed Mar 09 19:28:30.017083 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     self.raise_uncaught_exception(exc)
[Wed Mar 09 19:28:30.017086 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
[Wed Mar 09 19:28:30.017088 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     raise exc
[Wed Mar 09 19:28:30.017091 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/views.py", line 497, in dispatch
[Wed Mar 09 19:28:30.017093 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     self.initial(request, *args, **kwargs)
[Wed Mar 09 19:28:30.017096 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/views.py", line 415, in initial
[Wed Mar 09 19:28:30.017098 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     self.check_permissions(request)
[Wed Mar 09 19:28:30.017101 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/views.py", line 332, in check_permissions
[Wed Mar 09 19:28:30.017105 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     if not permission.has_permission(request, self):
[Wed Mar 09 19:28:30.017108 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/home/django/cvat/apps/iam/permissions.py", line 1032, in has_permission
[Wed Mar 09 19:28:30.017111 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     return self.check_permission(request, view, None)
[Wed Mar 09 19:28:30.017113 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/home/django/cvat/apps/iam/permissions.py", line 1028, in check_permission
[Wed Mar 09 19:28:30.017116 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     return all(permissions)
[Wed Mar 09 19:28:30.017119 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/home/django/cvat/apps/iam/permissions.py", line 78, in __bool__
[Wed Mar 09 19:28:30.017121 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     r = requests.post(self.url, json=self.payload)
[Wed Mar 09 19:28:30.017124 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/requests/api.py", line 117, in post
[Wed Mar 09 19:28:30.017126 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     return request('post', url, data=data, json=json, **kwargs)
[Wed Mar 09 19:28:30.017129 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/requests/api.py", line 61, in request
[Wed Mar 09 19:28:30.017132 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     return session.request(method=method, url=url, **kwargs)
[Wed Mar 09 19:28:30.017135 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
[Wed Mar 09 19:28:30.017138 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     resp = self.send(prep, **send_kwargs)
[Wed Mar 09 19:28:30.017141 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
[Wed Mar 09 19:28:30.017143 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     r = adapter.send(request, **kwargs)
[Wed Mar 09 19:28:30.017146 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
[Wed Mar 09 19:28:30.017148 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]     raise ConnectionError(e, request=request)
[Wed Mar 09 19:28:30.017151 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] requests.exceptions.ConnectionError: HTTPConnectionPool(host='opa', port=8181): Max retries exceeded with url: /v1/data/users/allow (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f13615fc460>: Failed to establish a new connection: [Errno -2] Name or service not known'))
[Wed Mar 09 19:28:30.017315 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] ERROR - 2022-03-09 19:28:30,016 - log - Internal Server Error: /api/users/self
[Wed Mar 09 19:28:30.017328 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436] Traceback (most recent call last):
[Wed Mar 09 19:28:30.017331 2022] [wsgi:error] [pid 237:tid 139722954303232] [remote 10.42.0.8:59436]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn

Next steps

You may join our Gitter channel for community support.
@ningjunwei2
Copy link
Contributor

I also have this problem

@azhavoro
Copy link
Contributor

Duplicate of #4192

@azhavoro azhavoro marked this as a duplicate of #4192 Mar 10, 2022
@HarrisDePerceptron
Copy link
Author

Duplicate of #4192
@azhavoro this issue was very vague. doesn't mentions the version. this is much more detailed

@HarrisDePerceptron HarrisDePerceptron mentioned this issue Mar 10, 2022
Closed
2 tasks
@yaoguang97
Copy link

I deployed it from scratch;I have a similar problem.
docker deploy、develop branch
2022-03-10 16-12-50屏幕截图

@se-wo
Copy link
Contributor

se-wo commented Mar 11, 2022

As also mentioned in #4192 the current helm charts miss the Open Policy Agent container introduced in cvat 2.0. I am working on a updated helm chart. Maybe I can share it when it is finished. Main problem so far. It will need an additional custom container where opa and the rules are bundled. Mouting those from a local directory will not work with k8s.
Moreover as the charts depend on spec selectors upgrading won´t we be easy. If a new opa spec selector is introduced one will have to delete the current deployment as spec selectors are no longer mutable.

@se-wo se-wo mentioned this issue Mar 12, 2022
Merged
8 tasks
@bsekachev
Copy link
Member

Closed since 2.0 is a too old release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@se-wo @HarrisDePerceptron @bsekachev @azhavoro @yaoguang97 @ningjunwei2