Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Server error for /api/fields.json missing from nightly error_analyzer report #1295

Closed
rfrenchseti opened this issue Feb 14, 2023 · 4 comments
Closed

Server error for /api/fields.json missing from nightly error_analyzer report #1295

rfrenchseti opened this issue Feb 14, 2023 · 4 comments
Assignees
@fyellin
Labels
A-Bug B-Other Uncategorized Effort 2 Medium Priority 3 Important

Comments

@rfrenchseti
Copy link
Collaborator

The following error appears in the apache logs but is missing from the nightly error_analyzer report.

The relevant logs are in DropBox in the logs-2023-02-14 directory.

[Tue Feb 14 00:10:06.968157 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638] [14/Feb/2023 00:10:06] ERROR [django.request:241] Internal Server Error: /opus/api/fields.json
[Tue Feb 14 00:10:06.968180 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638] Traceback (most recent call last):
[Tue Feb 14 00:10:06.968184 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]   File "/opus/opus_venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
[Tue Feb 14 00:10:06.968186 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]     response = get_response(request)
[Tue Feb 14 00:10:06.968189 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]                ^^^^^^^^^^^^^^^^^^^^^
[Tue Feb 14 00:10:06.968192 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]   File "/opus/opus_venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
[Tue Feb 14 00:10:06.968195 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]     response = wrapped_callback(request, *callback_args, **callback_kwargs)
[Tue Feb 14 00:10:06.968198 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Tue Feb 14 00:10:06.968200 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]   File "/opus/opus_venv/lib/python3.11/site-packages/django/views/decorators/cache.py", line 62, in _wrapped_view_func
[Tue Feb 14 00:10:06.968203 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]     response = view_func(request, *args, **kwargs)
[Tue Feb 14 00:10:06.968205 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Tue Feb 14 00:10:06.968208 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]   File "/opus/src/pds-opus/opus/application/apps/metadata/views.py", line 656, in api_get_fields
[Tue Feb 14 00:10:06.968210 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]     ret = get_fields_info(fmt, request, api_code, slug=slug, collapse=collapse)
[Tue Feb 14 00:10:06.968213 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Tue Feb 14 00:10:06.968215 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]   File "/opus/src/pds-opus/opus/application/apps/metadata/views.py", line 837, in get_fields_info
[Tue Feb 14 00:10:06.968218 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]     cache.set(cache_key, return_obj)
[Tue Feb 14 00:10:06.968220 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]   File "/opus/opus_venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py", line 79, in set
[Tue Feb 14 00:10:06.968223 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]     if not self._cache.set(key, value, self.get_backend_timeout(timeout)):
[Tue Feb 14 00:10:06.968225 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Tue Feb 14 00:10:06.968227 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]   File "/opus/opus_venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 344, in set
[Tue Feb 14 00:10:06.968230 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]     return self._run_cmd("set", key, False, *args, **kwargs)
[Tue Feb 14 00:10:06.968232 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Tue Feb 14 00:10:06.968235 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]   File "/opus/opus_venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 322, in _run_cmd
[Tue Feb 14 00:10:06.968237 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]     return self._safely_run_func(client, func, default_val, *args, **kwargs)
[Tue Feb 14 00:10:06.968240 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Tue Feb 14 00:10:06.968242 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]   File "/opus/opus_venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 211, in _safely_run_func
[Tue Feb 14 00:10:06.968245 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]     result = func(*args, **kwargs)
[Tue Feb 14 00:10:06.968247 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]              ^^^^^^^^^^^^^^^^^^^^^
[Tue Feb 14 00:10:06.968250 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]   File "/opus/opus_venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 475, in set
[Tue Feb 14 00:10:06.968252 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]     return self._store_cmd(b"set", {key: value}, expire, noreply, flags=flags)[key]
[Tue Feb 14 00:10:06.968255 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Tue Feb 14 00:10:06.968257 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]   File "/opus/opus_venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 1247, in _store_cmd
[Tue Feb 14 00:10:06.968260 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]     self._raise_errors(line, name)
[Tue Feb 14 00:10:06.968262 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]   File "/opus/opus_venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 1042, in _raise_errors
[Tue Feb 14 00:10:06.968265 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638]     raise MemcacheServerError(error)
[Tue Feb 14 00:10:06.968267 2023] [wsgi:error] [pid 1583321:tid 139751262271232] [remote 10.1.10.1:44638] pymemcache.exceptions.MemcacheServerError: b'object too large for cache'
@fyellin
Copy link
Collaborator

fyellin commented Feb 14, 2023

Before I spend too much time on this. I don't have the command line you use to run the error_analyzer report. Is 10.1.10.1 one of the ignored IPs?

I would suspect that all requests for fields.json are generated internally.

@rfrenchseti
Copy link
Collaborator Author

There are no ignored IPs for errors. All occurrences of api/fields.json should be externally-generated, and probably only by the log analyzer.

python error_analyzer.py --ignore-errors-file /opus/src/pds-webserver/opus_servers/cron_files/error_analyzer_ignore_file.txt --cronjob -o "/tmp/error_analyzer_results_temp/%Y-%m/OPUS-error-analysis-%Y-%m-%d.txt" "/Volumes/opus/shared/apache_logs/tools_error_log-%Y-%m-%d" "/Volumes/opus/shared/apache_logs/tools2_error_log-%Y-%m-%d" "/Volumes/opus/shared/apache_logs/tools_access_log-%Y-%m-%d" "/Volumes/opus/shared/apache_logs/tools2_access_log-%Y-%m-%d"

error_analyzer_ignore_file.txt:

WARNING [django.request:228] Not Found
WARNING [django.request:228] Bad Request
AH01264: script not found or unable to stat
AH01276: Cannot serve directory
AH00126: Invalid URI in request
AH02042: rejecting client initiated renegotiation
to ALLOWED_HOSTS.

@fyellin
Copy link
Collaborator

fyellin commented Feb 14, 2023 via email

@rfrenchseti
Copy link
Collaborator Author

Oh, I see. We really don't do that kind of testing anymore. I'd much rather see all the errors.

fyellin added a commit to fyellin/pds-opus that referenced this issue Feb 14, 2023
@fyellin
Bug SETI#1295. We no longer want to filter out messages that occur on…
f32a94b 
… the local network.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fyellin fyellin

Labels
A-Bug B-Other Uncategorized Effort 2 Medium Priority 3 Important
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fyellin @rfrenchseti