You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The if sample.content is None line randomly triggers this error:
[
"Traceback (most recent call last):\n",
" File \"/usr/local/lib/python3.7/site-packages/urllib3/response.py\", line 441, in _error_catcher\n yield\n",
" File \"/usr/local/lib/python3.7/site-packages/urllib3/response.py\", line 518, in read\n data = self._fp.read() if not fp_closed else b\"\"\n",
" File \"/usr/local/lib/python3.7/http/client.py\", line 478, in read\n s = self._safe_read(self.length)\n",
" File \"/usr/local/lib/python3.7/http/client.py\", line 630, in _safe_read\n raise IncompleteRead(b''.join(s), amt)\n",
"http.client.IncompleteRead: IncompleteRead(2097152 bytes read, 15508683 more expected)\n",
"\nDuring handling of the above exception, another exception occurred:\n\n",
"Traceback (most recent call last):\n",
" File \"/usr/local/lib/python3.7/site-packages/karton/core/karton.py\", line 178, in internal_process\n self.process(self.current_task)\n",
" File \"/usr/local/lib/python3.7/site-packages/karton/rulematcher/rulematcher.py\", line 260, in process\n if sample.content is None:\n",
" File \"/usr/local/lib/python3.7/site-packages/karton/core/resource.py\", line 413, in content\n return self.download()\n",
" File \"/usr/local/lib/python3.7/site-packages/karton/core/resource.py\", line 467, in download\n self._content = self.backend.download_object(self.bucket, self.uid)\n",
" File \"/usr/local/lib/python3.7/site-packages/karton/core/backend.py\", line 600, in download_object\n return reader.read()\n",
" File \"/usr/local/lib/python3.7/site-packages/urllib3/response.py\", line 544, in read\n raise IncompleteRead(self._fp_bytes_read, self.length_remaining)\n",
" File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n",
" File \"/usr/local/lib/python3.7/site-packages/urllib3/response.py\", line 458, in _error_catcher\n raise ProtocolError(\"Connection broken: %r\" % e, e)\n",
"urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(2097152 bytes read, 15508683 more expected)', IncompleteRead(2097152 bytes read, 15508683 more expected))\n"
]
Actually it looks like connectivity issues. Sometimes we have them as well with MinIO cluster. As they're usually temporary, I think built-in download/upload retry in that method should be implemented
I'm running the following consumer on 20 replicas:
The
if sample.content is None
line randomly triggers this error:This seems to be the culprit
karton/backend.py
:I have about 1000 active tasks. The MinIO stack is deployed as follows:
The text was updated successfully, but these errors were encountered: