Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

samples/pubsub.py gives AWS_ERROR_HTTP_CALLBACK_FAILURE or AWS_ERROR_MQTT_UNEXPECTED_HANGUP #115

Closed
2 tasks done
EGrunzke opened this issue Oct 5, 2020 · 7 comments
Closed
2 tasks done
Assignees
Labels
bug This issue is a bug.

Comments

@EGrunzke
Copy link

EGrunzke commented Oct 5, 2020

Confirm by changing [ ] to [x] below to ensure that it's a bug:

Known Issue

  • I'm using ATS data type endpoint: the endpoint should look like <prefix>-ats.iot.<region>.amazonaws.com

Describe the bug
The pubsub sample gives errors:

  • With --websockets: AwsCrtError(name='AWS_ERROR_HTTP_CALLBACK_FAILURE', message='A callback has reported failure.', code=2064)
  • Without --websockets: AwsCrtError(name='AWS_ERROR_MQTT_UNEXPECTED_HANGUP', message='The connection was closed unexpectedly.', code=5134)

SDK version number
awsiotsdk-1.5.0
awscrt-0.9.0

Platform/OS/Device
Docker python:3.8-buster

To Reproduce (observed behavior)

  1. Follow the IoT quick connect guide to get a connection kit. Extract the kit to a directory.
  2. Run a docker image, mounting the connection kit as a volume: docker run -v `pwd`:/home/root/kit -it python:3.8-buster bash

Examine the start.sh file from the connection kit. It will have your endpoint. The cert and key are included in the connection kit as well.

While in bash in the docker container:

python3 -m pip install awsiotsdk
cd /home/root/kit
git clone https://github.com/aws/aws-iot-device-sdk-python-v2.git
curl https://www.amazontrust.com/repository/AmazonRootCA1.pem > root-CA.crt
python aws-iot-device-sdk-python-v2/samples/pubsub.py --endpoint YOUR_ENDPOINT_HERE-ats.iot.us-east-2.amazonaws.com --root-ca root-CA.crt --cert YOUR_CERT_NAME.cert.pem --key YOUR_CERT_NAME.private.key --use-websocket

This gives the errors listed above

Expected behavior
The pubsub example should publish and receive messages.

Logs/output

root@4818cae41c99:/home/root/kit# python aws-iot-device-sdk-python-v2/samples/pubsub.py --endpoint <SANITIZED>-ats.iot.us-east-2.amazonaws.com --root-ca root-CA.crt --cert eric-dev-docker.cert.pem --key eric-dev-docker.private.key --use-websocket --verbosity Debug
[INFO] [2020-10-05T05:45:36Z] [00007f2988c15740] [event-loop] - id=0x562da6ede5e0: Initializing edge-triggered epoll
[INFO] [2020-10-05T05:45:36Z] [00007f2988c15740] [event-loop] - id=0x562da6ede5e0: Using eventfd for cross-thread notifications.
[INFO] [2020-10-05T05:45:36Z] [00007f2988c15740] [event-loop] - id=0x562da6ede5e0: Starting event-loop thread.
[INFO] [2020-10-05T05:45:36Z] [00007f2988c15740] [dns] - id=0x562da6edd9a0: Initializing default host resolver with 16 max host entries.
[INFO] [2020-10-05T05:45:36Z] [00007f2988c15740] [channel-bootstrap] - id=0x562da6f23160: Initializing client bootstrap with event-loop group 0x562da6f22890
[DEBUG] [2020-10-05T05:45:36Z] [00007f2988c15740] [AuthProfile] - Creating profile collection from file at "/root/.aws/config"
[ERROR] [2020-10-05T05:45:36Z] [00007f2988c15740] [file-utils] - static: Failed to open file /root/.aws/config with errno 2
[WARN] [2020-10-05T05:45:36Z] [00007f2988c15740] [AuthProfile] - Failed to read file at "/root/.aws/config"
[DEBUG] [2020-10-05T05:45:36Z] [00007f2988c15740] [AuthProfile] - Creating profile collection from file at "/root/.aws/credentials"
[ERROR] [2020-10-05T05:45:36Z] [00007f2988c15740] [file-utils] - static: Failed to open file /root/.aws/credentials with errno 2
[WARN] [2020-10-05T05:45:36Z] [00007f2988c15740] [AuthProfile] - Failed to read file at "/root/.aws/credentials"
[ERROR] [2020-10-05T05:45:36Z] [00007f2988c15740] [AuthCredentialsProvider] - static: Profile credentials parser could not load or parse a credentials or config file.
[INFO] [2020-10-05T05:45:36Z] [00007f2988c15740] [connection-manager] - id=0x562da6f2b4f0: Successfully created
[INFO] [2020-10-05T05:45:36Z] [00007f2988c15740] [exp-backoff-strategy] - id=0x562da6f324e0: Initializing exponential backoff retry strategy with scale factor: 0 jitter mode: 0 and max retries 1
[INFO] [2020-10-05T05:45:36Z] [00007f2987679700] [event-loop] - id=0x562da6ede5e0: main loop started
[INFO] [2020-10-05T05:45:36Z] [00007f2987679700] [event-loop] - id=0x562da6ede5e0: default timeout 100000, and max events to process per tick 100
[DEBUG] [2020-10-05T05:45:36Z] [00007f2988c15740] [tls-handler] - ctx: Setting ALPN list http/1.1
Connecting to <SANITIZED>-ats.iot.us-east-2.amazonaws.com with client ID 'test-a8b55126-24ac-4731-85d1-2207e2da7b10'...
[DEBUG] [2020-10-05T05:45:36Z] [00007f2988c15740] [mqtt-client] - client=0x562da6faa760: Initalizing MQTT client
[DEBUG] [2020-10-05T05:45:36Z] [00007f2988c15740] [mqtt-client] - id=0x562da704ad50: Creating new connection
[DEBUG] [2020-10-05T05:45:36Z] [00007f2988c15740] [mqtt-topic-tree] - tree=0x562da704afa8: Creating new topic tree
[INFO] [2020-10-05T05:45:36Z] [00007f2988c15740] [mqtt-client] - id=0x562da704ad50: using ping timeout of 3000000000 ns
[DEBUG] [2020-10-05T05:45:36Z] [00007f2988c15740] [AuthCredentialsProvider] - (id=0x562da6ecebd0) Credentials provider chain get credentials dispatch
[INFO] [2020-10-05T05:45:36Z] [00007f2988c15740] [AuthCredentialsProvider] - (id=0x562da6db1d30) Cached credentials provider has expired credentials.  Requerying.
[DEBUG] [2020-10-05T05:45:36Z] [00007f2988c15740] [AuthCredentialsProvider] - (id=0x562da6f22830) Credentials provider chain get credentials dispatch
[DEBUG] [2020-10-05T05:45:36Z] [00007f2988c15740] [AuthCredentialsProvider] - (id=0x562da6f22830) Credentials provider chain callback 1 invoked with invalid credentials and error code 6152
[DEBUG] [2020-10-05T05:45:36Z] [00007f2988c15740] [AuthCredentialsProvider] - (id=0x562da6f22830) Credentials provider chain invoking chain member #1
[DEBUG] [2020-10-05T05:45:36Z] [00007f2988c15740] [exp-backoff-strategy] - id=0x562da6f324e0: Initializing retry token 0x562da704b0b0
[DEBUG] [2020-10-05T05:45:36Z] [00007f2988c15740] [Unknown] - (id=0x562da6f2a4e0) IMDS client's token is invalid and is now updating.
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [task-scheduler] - id=0x562da704b100: Scheduling aws_exponential_backoff_retry_task task for immediate execution
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [task-scheduler] - id=0x562da704b100: Running aws_exponential_backoff_retry_task task with <Running> status
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [exp-backoff-strategy] - id=0x562da6f324e0: Vending retry_token 0x562da704b0b0
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [Unknown] - id=0x562da6f2a4e0: IMDS Client successfully acquired retry token.
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [connection-manager] - id=0x562da6f2b4f0: Acquire connection
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [connection-manager] - id=0x562da6f2b4f0: snapshot - state=1, idle_connection_count=0, pending_acquire_count=1, pending_connect_count=1, vended_connection_count=0, open_connection_count=0, ref_count=1
[INFO] [2020-10-05T05:45:36Z] [00007f2987679700] [connection-manager] - id=0x562da6f2b4f0: Requesting 1 new connections from http
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [dns] - id=0x562da6edd9a0: Host resolution requested for 169.254.169.254
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [dns] - id=0x562da6edd9a0: No cached entries found for 169.254.169.254 starting new resolver thread.
[DEBUG] [2020-10-05T05:45:36Z] [00007f2986327700] [dns] - static: resolving host 169.254.169.254
[DEBUG] [2020-10-05T05:45:36Z] [00007f2986327700] [dns] - static: resolved record: 169.254.169.254
[DEBUG] [2020-10-05T05:45:36Z] [00007f2986327700] [dns] - static: new address resolved 169.254.169.254 for host 169.254.169.254 caching
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [task-scheduler] - id=0x7f297c005110: Scheduling attempt_connection task for immediate execution
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [task-scheduler] - id=0x7f297c005110: Running attempt_connection task with <Running> status
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [socket] - id=0x7f2978006b20 fd=7: initializing with domain 0 and type 0
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [socket] - id=0x7f2978006b20 fd=7: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2020-10-05T05:45:36Z] [00007f2987679700] [socket] - id=0x7f2978006b20 fd=7: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [socket] - id=0x7f2978006b20 fd=7: beginning connect.
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [socket] - id=0x7f2978006b20 fd=7: connecting to endpoint 169.254.169.254:80.
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [task-scheduler] - id=0x7f2978007070: Scheduling (null) task for future execution at time 1642261114754200
[ERROR] [2020-10-05T05:45:36Z] [00007f2987679700] [socket] - id=0x7f2978006b20 fd=7: connection failure
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [channel-bootstrap] - id=0x562da6f23160: client connection on socket 0x7f2978006b20 completed with error 1047.
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [channel-bootstrap] - id=0x562da6f23160: recording bad address 169.254.169.254.
[INFO] [2020-10-05T05:45:36Z] [00007f2987679700] [dns] - id=0x562da6edd9a0: recording failure for record 169.254.169.254 for 169.254.169.254, moving to bad list
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [dns] - static: purging address 169.254.169.254 for host 169.254.169.254 from the cache due to cache eviction or shutdown
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [socket] - id=0x7f2978006b20 fd=7: closing
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [task-scheduler] - id=0x7f2978006ce0: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[ERROR] [2020-10-05T05:45:36Z] [00007f2987679700] [channel-bootstrap] - id=0x562da6f23160: Connection failed with error_code 1047.
[ERROR] [2020-10-05T05:45:36Z] [00007f2987679700] [http-connection] - static: Client connection failed with error 1047 (AWS_IO_SOCKET_CONNECTION_REFUSED).
[WARN] [2020-10-05T05:45:36Z] [00007f2987679700] [connection-manager] - id=0x562da6f2b4f0: Failed to obtain new connection from http layer, error 1047(socket connection refused.)
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [connection-manager] - id=0x562da6f2b4f0: Failing excess connection acquisition with error code 1047
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [connection-manager] - id=0x562da6f2b4f0: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=0, ref_count=1
[WARN] [2020-10-05T05:45:36Z] [00007f2987679700] [connection-manager] - id=0x562da6f2b4f0: Failed to complete connection acquisition with error_code 1047(socket connection refused.)
[WARN] [2020-10-05T05:45:36Z] [00007f2987679700] [Unknown] - id=0x562da6f2a4e0: IMDS Client failed to acquire a connection, error code 1047(socket connection refused.)
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [exp-backoff-strategy] - id=0x562da6f324e0: Initializing retry token 0x7f2978006a10
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [task-scheduler] - id=0x7f2978006a60: Scheduling aws_exponential_backoff_retry_task task for immediate execution
[ERROR] [2020-10-05T05:45:36Z] [00007f2987679700] [Unknown] - (id=0x562da6f2a4e0) IMDS client failed to update the token from IMDS.
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [channel-bootstrap] - id=0x562da6f23160: releasing bootstrap reference
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [task-scheduler] - id=0x7f2978006ce0: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [task-scheduler] - id=0x7f2978006a60: Running aws_exponential_backoff_retry_task task with <Running> status
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [exp-backoff-strategy] - id=0x562da6f324e0: Vending retry_token 0x7f2978006a10
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [Unknown] - id=0x562da6f2a4e0: IMDS Client successfully acquired retry token.
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [connection-manager] - id=0x562da6f2b4f0: Acquire connection
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [connection-manager] - id=0x562da6f2b4f0: snapshot - state=1, idle_connection_count=0, pending_acquire_count=1, pending_connect_count=1, vended_connection_count=0, open_connection_count=0, ref_count=1
[INFO] [2020-10-05T05:45:36Z] [00007f2987679700] [connection-manager] - id=0x562da6f2b4f0: Requesting 1 new connections from http
[DEBUG] [2020-10-05T05:45:36Z] [00007f2987679700] [dns] - id=0x562da6edd9a0: Host resolution requested for 169.254.169.254
[DEBUG] [2020-10-05T05:45:37Z] [00007f2986327700] [dns] - static: resolving host 169.254.169.254
[DEBUG] [2020-10-05T05:45:37Z] [00007f2986327700] [dns] - static: resolved record: 169.254.169.254
[INFO] [2020-10-05T05:45:37Z] [00007f2986327700] [dns] - static: promoting spotty record 169.254.169.254 for 169.254.169.254 back to good list
[DEBUG] [2020-10-05T05:45:37Z] [00007f2986327700] [dns] - static: purging address 169.254.169.254 for host 169.254.169.254 from the cache due to cache eviction or shutdown
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [task-scheduler] - id=0x7f297c005630: Scheduling attempt_connection task for immediate execution
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [task-scheduler] - id=0x7f297c005630: Running attempt_connection task with <Running> status
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [socket] - id=0x7f2978006b20 fd=7: initializing with domain 0 and type 0
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [socket] - id=0x7f2978006b20 fd=7: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2020-10-05T05:45:37Z] [00007f2987679700] [socket] - id=0x7f2978006b20 fd=7: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [socket] - id=0x7f2978006b20 fd=7: beginning connect.
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [socket] - id=0x7f2978006b20 fd=7: connecting to endpoint 169.254.169.254:80.
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [task-scheduler] - id=0x7f29780081f0: Scheduling (null) task for future execution at time 1642262115665200
[ERROR] [2020-10-05T05:45:37Z] [00007f2987679700] [socket] - id=0x7f2978006b20 fd=7: connection failure
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [channel-bootstrap] - id=0x562da6f23160: client connection on socket 0x7f2978006b20 completed with error 1047.
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [channel-bootstrap] - id=0x562da6f23160: recording bad address 169.254.169.254.
[INFO] [2020-10-05T05:45:37Z] [00007f2987679700] [dns] - id=0x562da6edd9a0: recording failure for record 169.254.169.254 for 169.254.169.254, moving to bad list
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [dns] - static: purging address 169.254.169.254 for host 169.254.169.254 from the cache due to cache eviction or shutdown
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [socket] - id=0x7f2978006b20 fd=7: closing
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [task-scheduler] - id=0x7f29780078e0: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[ERROR] [2020-10-05T05:45:37Z] [00007f2987679700] [channel-bootstrap] - id=0x562da6f23160: Connection failed with error_code 1047.
[ERROR] [2020-10-05T05:45:37Z] [00007f2987679700] [http-connection] - static: Client connection failed with error 1047 (AWS_IO_SOCKET_CONNECTION_REFUSED).
[WARN] [2020-10-05T05:45:37Z] [00007f2987679700] [connection-manager] - id=0x562da6f2b4f0: Failed to obtain new connection from http layer, error 1047(socket connection refused.)
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [connection-manager] - id=0x562da6f2b4f0: Failing excess connection acquisition with error code 1047
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [connection-manager] - id=0x562da6f2b4f0: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=0, ref_count=1
[WARN] [2020-10-05T05:45:37Z] [00007f2987679700] [connection-manager] - id=0x562da6f2b4f0: Failed to complete connection acquisition with error_code 1047(socket connection refused.)
[WARN] [2020-10-05T05:45:37Z] [00007f2987679700] [Unknown] - id=0x562da6f2a4e0: IMDS Client failed to acquire a connection, error code 1047(socket connection refused.)
[INFO] [2020-10-05T05:45:37Z] [00007f2987679700] [AuthCredentialsProvider] - (id=0x562da6f22830) Credentials provider chain callback terminating on index 2, with invalid credentials and error code 6154
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [AuthCredentialsProvider] - (id=0x562da6db1d30) Cached credentials provider next refresh time set to 1643160120369800
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [AuthCredentialsProvider] - (id=0x562da6db1d30) Cached credentials provider was unable to source credentials on refresh
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [AuthCredentialsProvider] - (id=0x562da6db1d30) Cached credentials provider notifying pending queries of new credentials
[ERROR] [2020-10-05T05:45:37Z] [00007f2987679700] [AuthCredentialsProvider] - (id=0x562da6ecebd0) Default chain credentials provider failed to source credentials with error 6154(aws-c-auth: AWS_AUTH_CREDENTIALS_PROVIDER_IMDS_SOURCE_FAILURE, Valid credentials could not be sourced by the IMDS provider)
[ERROR] [2020-10-05T05:45:37Z] [00007f2987679700] [AuthSigning] - (id=0x562da70be2b0) Credentials Provider failed to source credentials with error 6154(aws-c-auth: AWS_AUTH_CREDENTIALS_PROVIDER_IMDS_SOURCE_FAILURE, Valid credentials could not be sourced by the IMDS provider)
[ERROR] [2020-10-05T05:45:37Z] [00007f2987679700] [mqtt-client] - id=0x562da704ad50: Failure reported by websocket handshake transform callback.
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [channel-bootstrap] - id=0x562da6f23160: releasing bootstrap reference
[DEBUG] [2020-10-05T05:45:37Z] [00007f2987679700] [task-scheduler] - id=0x7f29780078e0: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
Traceback (most recent call last):
  File "aws-iot-device-sdk-python-v2/samples/pubsub.py", line 130, in <module>
    connect_future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
awscrt.exceptions.AwsCrtError: AwsCrtError(name='AWS_ERROR_HTTP_CALLBACK_FAILURE', message='A callback has reported failure.', code=2064)
[DEBUG] [2020-10-05T05:45:37Z] [00007f2988c15740] [mqtt-client] - id=0x562da704ad50: user called disconnect.
[ERROR] [2020-10-05T05:45:37Z] [00007f2988c15740] [mqtt-client] - id=0x562da704ad50: Connection is not open, and may not be closed
[DEBUG] [2020-10-05T05:45:37Z] [00007f2988c15740] [mqtt-client] - id=0x562da704ad50: Destroying connection
[DEBUG] [2020-10-05T05:45:37Z] [00007f2988c15740] [mqtt-topic-tree] - tree=0x562da704afa8: Cleaning up topic tree
[DEBUG] [2020-10-05T05:45:37Z] [00007f2988c15740] [mqtt-client] - client=0x562da6faa760: Cleaning up MQTT client
[DEBUG] [2020-10-05T05:45:37Z] [00007f2988c15740] [channel-bootstrap] - id=0x562da6f23160: releasing bootstrap reference
[INFO] [2020-10-05T05:45:37Z] [00007f2988c15740] [connection-manager] - id=0x562da6f2b4f0: release
[INFO] [2020-10-05T05:45:37Z] [00007f2988c15740] [connection-manager] - id=0x562da6f2b4f0: ref count now zero, starting shut down process
[INFO] [2020-10-05T05:45:37Z] [00007f2988c15740] [connection-manager] - id=0x562da6f2b4f0: manager release, failing 0 pending acquisitions
[DEBUG] [2020-10-05T05:45:37Z] [00007f2988c15740] [connection-manager] - id=0x562da6f2b4f0: snapshot - state=2, idle_connection_count=0, pending_acquire_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=0, ref_count=0
[INFO] [2020-10-05T05:45:37Z] [00007f2988c15740] [connection-manager] - id=0x562da6f2b4f0: Destroying self
[DEBUG] [2020-10-05T05:45:37Z] [00007f2988c15740] [channel-bootstrap] - id=0x562da6f23160: releasing bootstrap reference
[DEBUG] [2020-10-05T05:45:37Z] [00007f2988c15740] [channel-bootstrap] - id=0x562da6f23160: releasing bootstrap reference
[DEBUG] [2020-10-05T05:45:37Z] [00007f2988c15740] [channel-bootstrap] - id=0x562da6f23160: destroying
[INFO] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [event-loop] - id=0x55b935967140: Initializing edge-triggered epoll
[INFO] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [event-loop] - id=0x55b935967140: Using eventfd for cross-thread notifications.
[INFO] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [event-loop] - id=0x55b935967140: Starting event-loop thread.
[INFO] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [dns] - id=0x55b93591e360: Initializing default host resolver with 16 max host entries.
[INFO] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [event-loop] - id=0x55b935967140: main loop started
[INFO] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [event-loop] - id=0x55b935967140: default timeout 100000, and max events to process per tick 100
[INFO] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [channel-bootstrap] - id=0x55b9359818b0: Initializing client bootstrap with event-loop group 0x55b935980c90
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [tls-handler] - ctx: Certificate and key have been set, setting them up now.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [tls-handler] - ctx: Setting ALPN list x-amzn-mqtt-ca
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [mqtt-client] - client=0x55b935adcb80: Initalizing MQTT client
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [mqtt-client] - id=0x55b935adb660: Creating new connection
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [mqtt-topic-tree] - tree=0x55b935adb8b8: Creating new topic tree
Connecting to <SANITIZED>-ats.iot.us-east-2.amazonaws.com with client ID 'test-37169f71-bf7c-4ce8-b74c-da182203deed'...
[INFO] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [mqtt-client] - id=0x55b935adb660: using ping timeout of 3000000000 ns
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [dns] - id=0x55b93591e360: Host resolution requested for <SANITIZED>-ats.iot.us-east-2.amazonaws.com
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [dns] - id=0x55b93591e360: No cached entries found for <SANITIZED>-ats.iot.us-east-2.amazonaws.com starting new resolver thread.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf0c7e700] [dns] - static: resolving host <SANITIZED>-ats.iot.us-east-2.amazonaws.com
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf0c7e700] [dns] - static: resolved record: 18.219.65.120
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf0c7e700] [dns] - static: resolved record: 3.18.85.137
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf0c7e700] [dns] - static: resolved record: 3.131.154.213
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf0c7e700] [dns] - static: resolved record: 2600:1f00:6000::12db:63c0
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf0c7e700] [dns] - static: resolved record: 2600:1f00:6000::12df:f5f8
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf0c7e700] [dns] - static: resolved record: 2600:1f00:6000::382:4e48
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf0c7e700] [dns] - static: new address resolved 18.219.65.120 for host <SANITIZED>-ats.iot.us-east-2.amazonaws.com caching
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf0c7e700] [dns] - static: new address resolved 3.18.85.137 for host <SANITIZED>-ats.iot.us-east-2.amazonaws.com caching
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf0c7e700] [dns] - static: new address resolved 3.131.154.213 for host <SANITIZED>-ats.iot.us-east-2.amazonaws.com caching
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf0c7e700] [dns] - static: new address resolved 2600:1f00:6000::12db:63c0 for host <SANITIZED>-ats.iot.us-east-2.amazonaws.com caching
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf0c7e700] [dns] - static: new address resolved 2600:1f00:6000::12df:f5f8 for host <SANITIZED>-ats.iot.us-east-2.amazonaws.com caching
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf0c7e700] [dns] - static: new address resolved 2600:1f00:6000::382:4e48 for host <SANITIZED>-ats.iot.us-east-2.amazonaws.com caching
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce8008210: Scheduling attempt_connection task for immediate execution
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce80083c0: Scheduling attempt_connection task for immediate execution
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce8008210: Running attempt_connection task with <Running> status
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: initializing with domain 1 and type 0
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: beginning connect.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: connecting to endpoint 2600:1f00:6000::12db:63c0:443.
[ERROR] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: connect failed with error code 99.
[INFO] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [dns] - id=0x55b93591e360: recording failure for record 2600:1f00:6000::12db:63c0 for <SANITIZED>-ats.iot.us-east-2.amazonaws.com, moving to bad list
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [dns] - static: purging address 2600:1f00:6000::12db:63c0 for host <SANITIZED>-ats.iot.us-east-2.amazonaws.com from the cache due to cache eviction or shutdown
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: is still open, closing...
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: closing
[ERROR] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel-bootstrap] - id=0x55b9359818b0: failed to create socket with error 1055
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce80083c0: Running attempt_connection task with <Running> status
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: initializing with domain 0 and type 0
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: beginning connect.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: connecting to endpoint 18.219.65.120:443.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce4006850: Scheduling (null) task for future execution at time 1642285485124700
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce40056f0: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[INFO] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: connection success
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: local endpoint 172.17.0.3:50754
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: assigning to event loop 0x55b935967140
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel-bootstrap] - id=0x55b9359818b0: client connection on socket 0x7f3ce4005180 completed with error 0.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel] - id=0x7f3ce4006af0: Beginning creation and setup of new channel.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce4006f10: Scheduling on_channel_setup_complete task for immediate execution
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce40056f0: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce4006f10: Running on_channel_setup_complete task with <Running> status
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel] - id=0x7f3ce4006af0: setup complete, notifying caller.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel] - id=0x7f3ce4006af0: no message pool is currently stored in the event-loop local storage, adding 0x7f3ce40072b0 with max message size 16384, message count 4, with 4 small blocks of 128 bytes.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel-bootstrap] - id=0x55b9359818b0: channel 0x7f3ce4006af0 setup succeeded: bootstrapping.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket-handler] - id=0x7f3ce4017860: Socket handler created with max_read_size of 16384
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce40179c0: Scheduling tls_timeout task for future execution at time 1642284545017500
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [tls-handler] - id=0x7f3ce4017990: Alpn protocol negotiated as x-amzn-mqtt-ca
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [tls-handler] - id=0x7f3ce4017990: Remote server name is <SANITIZED>-ats.iot.us-east-2.amazonaws.com
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel-bootstrap] - id=0x55b9359818b0: tls negotiation result 0 on channel 0x7f3ce4006af0
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [mqtt-client] - id=0x55b935adb660: Connection successfully opened, sending CONNECT packet
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce4030d00: Scheduling mqtt_connack_timeout task for future execution at time 1642283671704000
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [mqtt-client] - id=0x55b935adb660: Adding username ?SDK=PythonV2&Version=0.9.0 to connection
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [tls-handler] - id=0x7f3ce4017990: Alert code 0
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce4006cf0: Scheduling channel_shutdown task for immediate execution
[INFO] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: zero read, socket is closed
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel] - id=0x7f3ce4006af0: Channel shutdown is already pending, not scheduling another.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce4006cf0: Running channel_shutdown task with <Running> status
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel] - id=0x7f3ce4006af0: beginning shutdown process
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel] - id=0x7f3ce4006af0: handler 0x7f3ce4017860 shutdown in read dir completed.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [tls-handler] - id=0x7f3ce4017990: Shutting down read direction with error code 0
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel] - id=0x7f3ce4006af0: handler 0x7f3ce4017990 shutdown in read dir completed.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel] - id=0x7f3ce4006af0: handler 0x55b935adb688 shutdown in read dir completed.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce4006b18: Scheduling (null) task for immediate execution
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce4006b18: Running (null) task with <Running> status
[INFO] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [mqtt-client] - id=0x55b935adb660: sending disconnect message as part of graceful shutdown.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [mqtt-client] - id=0x55b935adb660: failed to send courteous disconnect io message
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel] - id=0x7f3ce4006af0: handler 0x55b935adb688 shutdown in write dir completed.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [tls-handler] - id=0x7f3ce4017990: Shutting down write direction
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel] - id=0x7f3ce4006af0: handler 0x7f3ce4017990 shutdown in write dir completed.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [socket] - id=0x7f3ce4005180 fd=6: closing
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce40067a0: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce4017900: Scheduling socket_handler_close task for immediate execution
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce40067a0: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce4017900: Running socket_handler_close task with <Running> status
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel] - id=0x7f3ce4006af0: handler 0x7f3ce4017860 shutdown in write dir completed.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce4006b18: Scheduling (null) task for immediate execution
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce4006b18: Running (null) task with <Running> status
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel] - id=0x7f3ce4006af0: during shutdown, canceling task 0x7f3ce40179c0
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce40179c0: Running tls_timeout task with <Canceled> status
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel] - id=0x7f3ce4006af0: during shutdown, canceling task 0x7f3ce4030d00
Traceback (most recent call last):
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [task-scheduler] - id=0x7f3ce4030d00: Running mqtt_connack_timeout task with <Canceled> status
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel-bootstrap] - id=0x55b9359818b0: channel 0x7f3ce4006af0 shutdown with error 0.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel] - id=0x7f3ce4006af0: destroying channel.
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf18e9700] [channel-bootstrap] - id=0x55b9359818b0: releasing bootstrap reference
  File "aws-iot-device-sdk-python-v2/samples/pubsub.py", line 130, in <module>
    connect_future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
awscrt.exceptions.AwsCrtError: AwsCrtError(name='AWS_ERROR_MQTT_UNEXPECTED_HANGUP', message='The connection was closed unexpectedly.', code=5134)
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [mqtt-client] - id=0x55b935adb660: user called disconnect.
[ERROR] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [mqtt-client] - id=0x55b935adb660: Connection is not open, and may not be closed
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [mqtt-client] - id=0x55b935adb660: Destroying connection
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [mqtt-topic-tree] - tree=0x55b935adb8b8: Cleaning up topic tree
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [mqtt-client] - client=0x55b935adcb80: Cleaning up MQTT client
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [channel-bootstrap] - id=0x55b9359818b0: releasing bootstrap reference
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [channel-bootstrap] - id=0x55b9359818b0: releasing bootstrap reference
[DEBUG] [2020-10-05T05:45:57Z] [00007f3cf2e45740] [channel-bootstrap] - id=0x55b9359818b0: destroying

Additional context
The start.sh that comes with the connection kit appears to work, but it uses the old V1 sdk. You can see it work by following the guide:

chmod +x start.sh
./start.sh
@EGrunzke EGrunzke added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Oct 5, 2020
@jmklix
Copy link
Member

jmklix commented Oct 5, 2020

Can you try installing from source and see if that gives you the same error?

git clone https://github.com/aws/aws-iot-device-sdk-python-v2.git
python3 -m pip install ./aws-iot-device-sdk-python-v2

@jmklix jmklix self-assigned this Oct 5, 2020
@jmklix jmklix added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 2 days. and removed needs-triage This issue or PR still needs to be triaged. labels Oct 5, 2020
@EGrunzke
Copy link
Author

EGrunzke commented Oct 5, 2020

Good news! Doing a fresh checkout and installing from source, python aws-iot-device-sdk-python-v2/samples/pubsub.py --endpoint <SANITIZED>-ats.iot.us-east-2.amazonaws.com --root-ca root-CA.crt --cert eric-dev-docker.cert.pem --key eric-dev-docker.private.key did work!

However, if I add --use-websocket I still get AwsCrtError(name='AWS_ERROR_HTTP_CALLBACK_FAILURE', message='A callback has reported failure.', code=2064) I'm running in docker; are there ports I need to expose for websockets to work?

@EGrunzke
Copy link
Author

EGrunzke commented Oct 5, 2020

Also, I modified the auto-generated policy to be extremely lenient. That might have had an impact? I know that samples/README.md says the policy needs to provide some privileges. I will try a more specific policy (right now it's just "*") and see.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 2 days. label Oct 5, 2020
@bretambrose
Copy link
Contributor

A Websockets connection requires aws credentials. The default credentials provider chain used by the samples can retrieve credentials from the environment, an aws profile file, or imds/ecs (which isn't applicable to you). The IoT SDK supports most of the credentials providers you can find in the standard AWS SDKs and their documentation can be a good starting point (https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html for example). As with certificate-based authentication, the IAM policy associated with the aws credentials will need to allow all of the IoT operations you want to perform.

@jmklix
Copy link
Member

jmklix commented Nov 18, 2020

@EGrunzke have you been able to successfully connect with websockets?

@jmklix jmklix added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 2 days. label Nov 18, 2020
@EGrunzke
Copy link
Author

EGrunzke commented Nov 27, 2020

I have not tried with websockets since I got it working with normal mqtt. I suggest closing this as can't reproduce and I'll comment again if I have any trouble.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 2 days. label Nov 27, 2020
@jmklix jmklix closed this as completed Nov 30, 2020
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug.
Projects
None yet
Development

No branches or pull requests

3 participants