Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connection was closed before we received a valid response from endpoint URL: "https://runtime.sagemaker.us-east-2.amazonaws.com/endpoints/kmeans-2019-03-11-18-05-06-639/invocations". #3999

Closed
ZhenxinYu opened this issue Mar 11, 2019 · 10 comments
Assignees
Labels
closing-soon This issue will automatically close in 4 days unless further comments are made. guidance Question that needs advice or information. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.

Comments

@ZhenxinYu
Copy link

I am getting this error and don't know how to fix it. Could anyone give me some help here? Thanks.


ConnectionResetError Traceback (most recent call last)
~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
600 body=body, headers=headers,
--> 601 chunked=chunked)
602

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
356 else:
--> 357 conn.request(method, url, **httplib_request_kw)
358

~/anaconda3/envs/mxnet_p36/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
1238 """Send a complete request to the server."""
-> 1239 self._send_request(method, url, body, headers, encode_chunked)
1240

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/awsrequest.py in _send_request(self, method, url, body, headers, *args, **kwargs)
124 rval = super(AWSConnection, self)._send_request(
--> 125 method, url, body, headers, *args, **kwargs)
126 self._expect_header_set = False

~/anaconda3/envs/mxnet_p36/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
1284 body = _encode(body, 'body')
-> 1285 self.endheaders(body, encode_chunked=encode_chunked)
1286

~/anaconda3/envs/mxnet_p36/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
1233 raise CannotSendHeader()
-> 1234 self._send_output(message_body, encode_chunked=encode_chunked)
1235

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/awsrequest.py in _send_output(self, message_body, *args, **kwargs)
175 # we must run the risk of Nagle.
--> 176 self.send(message_body)
177

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/awsrequest.py in send(self, str)
235 return
--> 236 return super(AWSConnection, self).send(str)
237

~/anaconda3/envs/mxnet_p36/lib/python3.6/http/client.py in send(self, data)
982 datablock = datablock.encode("iso-8859-1")
--> 983 self.sock.sendall(datablock)
984 return

~/anaconda3/envs/mxnet_p36/lib/python3.6/ssl.py in sendall(self, data, flags)
971 while count < amount:
--> 972 v = self.send(byte_view[count:])
973 count += v

~/anaconda3/envs/mxnet_p36/lib/python3.6/ssl.py in send(self, data, flags)
940 self.class)
--> 941 return self._sslobj.write(data)
942 else:

~/anaconda3/envs/mxnet_p36/lib/python3.6/ssl.py in write(self, data)
641 """
--> 642 return self._sslobj.write(data)
643

ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

ProtocolError Traceback (most recent call last)
~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/httpsession.py in send(self, request)
257 preload_content=False,
--> 258 decode_content=False,
259 )

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
638 retries = retries.increment(method, url, error=e, _pool=self,
--> 639 _stacktrace=sys.exc_info()[2])
640 retries.sleep()

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
332 # Disabled, indicate to re-raise the error.
--> 333 raise six.reraise(type(error), error, _stacktrace)
334

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/urllib3/packages/six.py in reraise(tp, value, tb)
684 if value.traceback is not tb:
--> 685 raise value.with_traceback(tb)
686 raise value

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
600 body=body, headers=headers,
--> 601 chunked=chunked)
602

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
356 else:
--> 357 conn.request(method, url, **httplib_request_kw)
358

~/anaconda3/envs/mxnet_p36/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
1238 """Send a complete request to the server."""
-> 1239 self._send_request(method, url, body, headers, encode_chunked)
1240

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/awsrequest.py in _send_request(self, method, url, body, headers, *args, **kwargs)
124 rval = super(AWSConnection, self)._send_request(
--> 125 method, url, body, headers, *args, **kwargs)
126 self._expect_header_set = False

~/anaconda3/envs/mxnet_p36/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
1284 body = _encode(body, 'body')
-> 1285 self.endheaders(body, encode_chunked=encode_chunked)
1286

~/anaconda3/envs/mxnet_p36/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
1233 raise CannotSendHeader()
-> 1234 self._send_output(message_body, encode_chunked=encode_chunked)
1235

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/awsrequest.py in _send_output(self, message_body, *args, **kwargs)
175 # we must run the risk of Nagle.
--> 176 self.send(message_body)
177

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/awsrequest.py in send(self, str)
235 return
--> 236 return super(AWSConnection, self).send(str)
237

~/anaconda3/envs/mxnet_p36/lib/python3.6/http/client.py in send(self, data)
982 datablock = datablock.encode("iso-8859-1")
--> 983 self.sock.sendall(datablock)
984 return

~/anaconda3/envs/mxnet_p36/lib/python3.6/ssl.py in sendall(self, data, flags)
971 while count < amount:
--> 972 v = self.send(byte_view[count:])
973 count += v

~/anaconda3/envs/mxnet_p36/lib/python3.6/ssl.py in send(self, data, flags)
940 self.class)
--> 941 return self._sslobj.write(data)
942 else:

~/anaconda3/envs/mxnet_p36/lib/python3.6/ssl.py in write(self, data)
641 """
--> 642 return self._sslobj.write(data)
643

ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

ConnectionClosedError Traceback (most recent call last)
in ()
----> 1 train_result = kmeans_predictor.predict(train_data.values.astype('float32'))

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sagemaker/predictor.py in predict(self, data, initial_args)
76
77 request_args = self._create_request_args(data, initial_args)
---> 78 response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
79 return self._handle_response(response)
80

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
355 "%s() only accepts keyword arguments." % py_operation_name)
356 # The "self" in this scope is referring to the BaseClient.
--> 357 return self._make_api_call(operation_name, kwargs)
358
359 _api_call.name = str(py_operation_name)

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
646 else:
647 http, parsed_response = self._make_request(
--> 648 operation_model, request_dict, request_context)
649
650 self.meta.events.emit(

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/client.py in _make_request(self, operation_model, request_dict, request_context)
665 def _make_request(self, operation_model, request_dict, request_context):
666 try:
--> 667 return self._endpoint.make_request(operation_model, request_dict)
668 except Exception as e:
669 self.meta.events.emit(

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/endpoint.py in make_request(self, operation_model, request_dict)
100 logger.debug("Making request for %s with params: %s",
101 operation_model, request_dict)
--> 102 return self._send_request(request_dict, operation_model)
103
104 def create_request(self, params, operation_model=None):

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/endpoint.py in _send_request(self, request_dict, operation_model)
135 request, operation_model, context)
136 while self._needs_retry(attempts, operation_model, request_dict,
--> 137 success_response, exception):
138 attempts += 1
139 # If there is a stream associated with the request, we need

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/endpoint.py in _needs_retry(self, attempts, operation_model, request_dict, response, caught_exception)
229 event_name, response=response, endpoint=self,
230 operation=operation_model, attempts=attempts,
--> 231 caught_exception=caught_exception, request_dict=request_dict)
232 handler_response = first_non_none_response(responses)
233 if handler_response is None:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/hooks.py in emit(self, event_name, **kwargs)
354 def emit(self, event_name, **kwargs):
355 aliased_event_name = self._alias_event_name(event_name)
--> 356 return self._emitter.emit(aliased_event_name, **kwargs)
357
358 def emit_until_response(self, event_name, **kwargs):

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/hooks.py in emit(self, event_name, **kwargs)
226 handlers.
227 """
--> 228 return self._emit(event_name, kwargs)
229
230 def emit_until_response(self, event_name, **kwargs):

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/hooks.py in _emit(self, event_name, kwargs, stop_on_response)
209 for handler in handlers_to_call:
210 logger.debug('Event %s: calling handler %s', event_name, handler)
--> 211 response = handler(**kwargs)
212 responses.append((handler, response))
213 if stop_on_response and response is not None:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/retryhandler.py in call(self, attempts, response, caught_exception, **kwargs)
181
182 """
--> 183 if self._checker(attempts, response, caught_exception):
184 result = self._action(attempts=attempts)
185 logger.debug("Retry needed, action of: %s", result)

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/retryhandler.py in call(self, attempt_number, response, caught_exception)
249 def call(self, attempt_number, response, caught_exception):
250 should_retry = self._should_retry(attempt_number, response,
--> 251 caught_exception)
252 if should_retry:
253 if attempt_number >= self._max_attempts:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/retryhandler.py in _should_retry(self, attempt_number, response, caught_exception)
275 # If we've exceeded the max attempts we just let the exception
276 # propogate if one has occurred.
--> 277 return self._checker(attempt_number, response, caught_exception)
278
279

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/retryhandler.py in call(self, attempt_number, response, caught_exception)
315 for checker in self._checkers:
316 checker_response = checker(attempt_number, response,
--> 317 caught_exception)
318 if checker_response:
319 return checker_response

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/retryhandler.py in call(self, attempt_number, response, caught_exception)
221 elif caught_exception is not None:
222 return self._check_caught_exception(
--> 223 attempt_number, caught_exception)
224 else:
225 raise ValueError("Both response and caught_exception are None.")

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/retryhandler.py in _check_caught_exception(self, attempt_number, caught_exception)
357 # the MaxAttemptsDecorator is not interested in retrying the exception
358 # then this exception just propogates out past the retry code.
--> 359 raise caught_exception

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/endpoint.py in _do_get_response(self, request, operation_model)
198 http_response = first_non_none_response(responses)
199 if http_response is None:
--> 200 http_response = self._send(request)
201 except HTTPClientError as e:
202 return (None, e)

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/endpoint.py in _send(self, request)
242
243 def _send(self, request):
--> 244 return self.http_session.send(request)
245
246

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/httpsession.py in send(self, request)
287 error=e,
288 request=request,
--> 289 endpoint_url=request.url
290 )
291 except Exception as e:

ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "https://runtime.sagemaker.us-east-2.amazonaws.com/endpoints/kmeans-2019-03-11-18-05-06-639/invocations".

@jacoblazar
Copy link

I am getting the same error while uploading to S3 from CLI

@justnance
Copy link

@ZhenxinYu - Thank you for posting this issue. Please provide the CLI version (aws --version), the CLI command, and any steps I can use to reproduce this issue.

Initially, this appears to be an issue with the installation. How was the CLI installed and on what OS?

@justnance justnance self-assigned this May 8, 2019
@justnance justnance added guidance Question that needs advice or information. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. closing-soon This issue will automatically close in 4 days unless further comments are made. labels May 8, 2019
@no-response
Copy link

no-response bot commented May 18, 2019

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

@no-response no-response bot closed this as completed May 18, 2019
@abhilashshettigar
Copy link

Getting the same Error
Connection was closed before we received a valid response from endpoint URL

aws-cli/1.16.152 Python/3.6.0 Windows/10 botocore/1.12.142

@absrivastava
Copy link

I am also getting same error from last few days for Athena. (while it is also working for my other AWS accounts)
Connection was closed before we received a valid response from endpoint URL: "https://athena.us-west-2.amazonaws.com/ "".

cli-version : aws-cli/1.15.19 Python/2.7.16 Linux/4.14.138-89.102.amzn1.x86_64 botocore/1.10.19

@JavierLopezT
Copy link

I am getting the same error on a jupyter notebook sagemaker

@absrivastava
Copy link

I am also getting same error from last few days for Athena. (while it is also working for my other AWS accounts)
Connection was closed before we received a valid response from endpoint URL: "https://athena.us-west-2.amazonaws.com/ "".

cli-version : aws-cli/1.15.19 Python/2.7.16 Linux/4.14.138-89.102.amzn1.x86_64 botocore/1.10.19

This is resolved for me now, the endpoint was blocked at firewall.

@heberuriegas
Copy link

I am getting the same error on a jupyter notebook sagemaker

@phong-devpanel
Copy link

"Connection was closed before we received a valid response from endpoint URL" when I call invoke the endpoint from Jupyter Notebook Sagemaker

@dicksonj
Copy link

dicksonj commented May 19, 2020

I got the same error: ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL
Upon opening the URL I see - Missing Authentication Token

Not sure what this authentication is?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
closing-soon This issue will automatically close in 4 days unless further comments are made. guidance Question that needs advice or information. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.
Projects
None yet
Development

No branches or pull requests

9 participants