UnicodeDecodeError with aioredis tracing enabled #3088
Description
Which version of dd-trace-py are you using?
0.57.0
Which version of pip are you using?
21.3.1
Which version of the libraries are you using?
aiohttp==3.8.1
aioredis==1.3.1
aiosignal==1.2.0
amqp==5.0.9
analytics-python==1.4.0
asgiref==3.3.4
async-timeout==4.0.2
attrs==21.3.0
autobahn==21.3.1
Automat==20.2.0
aws-encryption-sdk==3.0.0
backcall==0.2.0
backoff==1.10.0
billiard==3.6.4.0
boto3==1.18.53
botocore==1.21.65
celery==5.1.2
Celery-OpenTracing==0.0.1
certifi==2021.10.8
cffi==1.15.0
channels==3.0.4
channels-redis==3.3.0
charset-normalizer==2.0.9
click==7.1.2
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
constantly==15.1.0
cryptography==36.0.1
daphne==3.0.2
ddtrace==0.53.3
decorator==5.1.0
Django==3.2.7
django-cors-headers==3.9.0
django-countries==7.2.1
django-enumfields==2.1.1
django-ipware==4.0.0
django-netfields==1.2.4
django-otp==1.1.1
django-phonenumber-field==5.2.0
django-polymorphic==3.0.0
django-ratelimit==3.0.1
django-redis==5.0.0
django-rest-auth==0.9.5
django-rest-knox==4.1.0
django-rest-polymorphic==0.1.9
django-storages==1.11.1
djangorestframework==3.12.4
drf-spectacular==0.19.0
exponent-server-sdk==2.0.0
fireblocks-sdk==1.6.4
frozenlist==1.2.0
geoip2==4.4.0
googlemaps==4.5.3
hiredis==2.0.0
hyperlink==21.0.0
idna==3.3
incremental==21.3.0
inflection==0.5.1
ipython==7.28.0
jedi==0.18.1
Jinja2==3.0.1
jmespath==0.10.0
jsonschema==4.3.2
kombu==5.2.2
MarkupSafe==2.0.1
matplotlib-inline==0.1.3
maxminddb==2.2.0
monotonic==1.6
more-itertools==8.10.0
msgpack==1.0.3
multidict==5.2.0
netaddr==0.8.0
opentracing==2.4.0
packaging==21.3
parso==0.8.3
pdfkit==0.6.1
pexpect==4.8.0
phonenumbers==8.12.40
pickleshare==0.7.5
prompt-toolkit==3.0.24
protobuf==3.19.1
psycopg2-binary==2.9.1
ptyprocess==0.7.0
pyaes==1.6.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycoingecko==2.2.0
pycparser==2.21
pydantic==1.8.2
Pygments==2.10.0
PyJWT==1.7.1
pyOpenSSL==21.0.0
pyparsing==3.0.6
pyrsistent==0.18.0
python-dateutil==2.8.2
python-http-client==3.3.4
python-json-logger==2.0.2
pytz==2021.3
PyYAML==6.0
redis==3.5.3
requests==2.26.0
rsa==4.8
s3transfer==0.5.0
sendgrid==6.8.2
sentry-sdk==1.3.1
service-identity==21.1.0
six==1.16.0
slackclient==2.9.3
sqlparse==0.4.2
starkbank-ecdsa==2.0.3
Telethon==1.23.0
tenacity==8.0.1
traitlets==5.1.1
Twisted==21.7.0
txaio==21.2.1
typing_extensions==4.0.1
ua-parser==0.10.0
uritemplate==4.1.1
urllib3==1.26.7
user-agents==2.2.0
uWSGI==2.0.19.1
vine==5.0.0
wcwidth==0.2.5
wrapt==1.13.3
xmltodict==0.12.0
yarl==1.7.2
zope.interface==5.4.0
How can we reproduce your problem?
This appears to be an issue with the args channels-redis sends. I don't have exact reproduction steps, but it's happening on all group_send calls. The issue is traced to this line.
From Sentry, we can see that args is:
[b'EVAL', "\n local over_capacity = 0\n local current_time = ARGV[#ARGV - 1]\n local expiry = ARGV[#ARGV]\n for i=1,#KEYS do\n if redis.call('ZCOUNT', KEYS[i], '-inf', '+inf') < tonumber(ARGV[i + #KEYS]) then\n redis.call('ZADD', KEYS[i], current_time, ARGV[i])\n redis.call('EXPIRE', KEYS[i], expiry)\n else\n over_capacity = over_capacity + 1\n ..., 82, 'asgispecific.01b14e3967944a38bf1a6d77860b0d9e!', 'asgispecific.1eb9eaed1ed5481992af051b3571ab0c!', 'asgispecific.eaad34e0007a4d1d94410e1ba4badf18!', 'asgispecific.f281fa62a0bc45ffaf67578bc762b519!', 'asgispecific.cf716363a9844c9ea24e19693d78eb4f!', 'asgispecific.1c41b7e2918c4a7aa9020f01639c2ceb!', 'asgispecific.60c6d5c8d88e4663bef2accc5664cfda!']
What is the result that you get?
An exception is thrown:
UnicodeDecodeError
'utf-8' codec can't decode byte 0xb1 in position 1: invalid start byte
We also see,
UnicodeDecodeError
'utf-8' codec can't decode byte 0xee in position 1: invalid continuation byte
What is the result that you expected?
No exception is thrown and the call is correctly traced.