Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

on-demand content results in unclosed, idle, database connections #2816

Closed
ggainey opened this issue Jun 9, 2022 · 15 comments
Closed

on-demand content results in unclosed, idle, database connections #2816

ggainey opened this issue Jun 9, 2022 · 15 comments
Labels

Comments

@ggainey
Copy link
Contributor

ggainey commented Jun 9, 2022

Version

  • core/main, pulp_rpm/main
  • core/3.16, pulp_rpm/3.17
  • pulp3-source-centos7

Describe the bug
A client asking for content from an "on_demand" remote results in a database connection remaining open for each piece of RemoteArtifact content that was streamed back to the client. This can run a pulp instance out of database connections, if (for example) a downstream Pulp attempts to sync 'immediate' from an on_demand upstream.

The idle connections all show an identical last-query, where they asked for the remote of the not-yet-streamed content. Here's an example:

SELECT "core_remote"."pulp_id", "core_remote"."pulp_created", "core_remote"."pulp_last_updated", "core_remote"."pulp_type", "core_remote"."name", "core_remote"."url", "core
_remote"."ca_cert", "core_remote"."client_cert", "core_remote"."client_key", "core_remote"."tls_validation", "core_remote"."username", "core_remote"."password", "core_remote
"."proxy_url", "core_remote"."proxy_username", "core_remote"."proxy_password", "core_remote"."download_concurrency", "core_remote"."max_retries", "core_remote"."policy", "co
re_remote"."total_timeout", "core_remote"."connect_timeout", "core_remote"."sock_connect_timeout", "core_remote"."sock_read_timeout", "core_remote"."headers", "core_remote".
"rate_limit" FROM "core_remote" WHERE "core_remote"."pulp_id" = '949036ee-cb03-4bfd-aaf8-739e2af1a0fc'::uuid LIMIT 21

If multiple clients ask for the same content at the same time, you can end up with more idle connections than content-units, due to the occasional concurrency-collisions.

To Reproduce

The following bash script will result in one connection to the database for each piece of content in the /rpm-signed fixture:

#!/usr/bin/bash
pulp rpm remote create --name test --policy on_demand --url https://fixtures.pulpproject.org/rpm-signed/
pulp rpm repository create --name test --remote test --autopublish
pulp rpm repository sync --name test
pulp rpm distribution create --name test --base-path test --repository test

echo "Connections at start:"
psql -U pulp -d pulp --host 127.0.0.1 -c "SELECT count(pid) FROM pg_stat_activity WHERE datname = 'pulp' and query like '%remote%';"

FILES=(\
b/bear-4.1-1.noarch.rpm \
c/camel-0.1-1.noarch.rpm \
c/cat-1.0-1.noarch.rpm \
c/cheetah-1.25.3-5.noarch.rpm \
c/chimpanzee-0.21-1.noarch.rpm \
c/cockateel-3.1-1.noarch.rpm \
c/cow-2.2-3.noarch.rpm \
c/crow-0.8-1.noarch.rpm \
d/dog-4.23-1.noarch.rpm \
d/dolphin-3.10.232-1.noarch.rpm \
d/duck-0.6-1.noarch.rpm \
d/duck-0.7-1.noarch.rpm \
d/duck-0.8-1.noarch.rpm \
e/elephant-8.3-1.noarch.rpm \
f/fox-1.1-2.noarch.rpm \
f/frog-0.1-1.noarch.rpm \
g/giraffe-0.67-2.noarch.rpm \
g/gorilla-0.62-1.noarch.rpm \
h/horse-0.22-2.noarch.rpm \
k/kangaroo-0.2-1.noarch.rpm \
k/kangaroo-0.3-1.noarch.rpm \
l/lion-0.4-1.noarch.rpm \
m/mouse-0.1.12-1.noarch.rpm \
p/penguin-0.9.1-1.noarch.rpm \
p/pike-2.2-1.noarch.rpm \
s/shark-0.1-1.noarch.rpm \
s/squirrel-0.1-1.noarch.rpm \
s/stork-0.12-2.noarch.rpm \
t/tiger-1.0-4.noarch.rpm \
t/trout-0.12-1.noarch.rpm \
w/walrus-0.71-1.noarch.rpm \
w/walrus-5.21-1.noarch.rpm \
w/whale-0.2-1.noarch.rpm \
w/wolf-9.4-2.noarch.rpm \
z/zebra-0.1-2.noarch.rpm \
)

for f in ${FILES[@]}; do
  echo "FILE ${f}"
  FPATH="test/Packages/${f}"
  ( http GET :/pulp/content/${FPATH} ) &
done

sleep 10
echo "Connections at end:"
psql -U pulp -d pulp --host 127.0.0.1 -c "SELECT count(pid) FROM pg_stat_activity WHERE datname = 'pulp' and query like '%remote%';"

Notes

  • Restarting pulpcore-content releases the idle connections.
  • Issuing the GET calls in the script above sequentially results in fewer stale connections (e.g., 22)

Expected behavior
Database connection must be closed as soon as the streamed Artifact has been successfully saved.

Additional context

See the discussion starting at https://bugzilla.redhat.com/show_bug.cgi?id=2062526#c19 for the investigation that leads us here. The BZ says "deadlock", this problem interferes with verifying that fix, and will have its own BZ "soon". Some of the discussion on '2526 contains machines and access-info, and is alas private. I have made as much of it public as I reasonably can.

@fao89
Copy link
Member

fao89 commented Jun 13, 2022

@ggainey
Copy link
Contributor Author

ggainey commented Jun 13, 2022

Workin in a core/3.16 env, I have narrowed the problem down to the commit where we moved from loop.run_in_executor() to sync_to_async(): 5dc7314

Reverting this commit gets the reproducer back to 1 connection post-run. Investigation continues.

@dralley
Copy link
Contributor

dralley commented Jun 13, 2022

Workin in a core/3.16 env, I have narrowed the probglem down to the commit where we moved from loop.run_in_executor() to sync_to_async(): 5dc7314

Ooof.

@ggainey
Copy link
Contributor Author

ggainey commented Jun 13, 2022

@dralley no worries - Something Magic was happening in run_in_executor(), that we are (apparently) losing with sync_to_async(). Calling django.db.connection.close() at the End Of Things gets us 1 cnx back; I am unclear on where the others are hiding. Investigation continues.

@dralley
Copy link
Contributor

dralley commented Jun 13, 2022

How many stale connections are we looking at? Prior to this patch we used a threadpool of size 2, afterwards we use the default-sized threadpool, which is something like 2x $num_cores. I think there should be 1 database connection per thread.

5dc7314#diff-453fdf3a6e81f92ff5391cf65f9c6154852133a656d82082021e3beaa887776fL50

@dralley
Copy link
Contributor

dralley commented Jun 13, 2022

A good place to start might also be to just throw thread_sensitive=True on all the callsites. There's this issue which implies that it might be necessary? https://code.djangoproject.com/ticket/31905#comment:14

@ggainey
Copy link
Contributor Author

ggainey commented Jun 13, 2022

How many stale connections are we looking at? Prior to this patch we used a threadpool of size 2, afterwards we use the default-sized threadpool, which is something like 2x $num_cores. I think there should be 1 database connection per thread.

5dc7314#diff-453fdf3a6e81f92ff5391cf65f9c6154852133a656d82082021e3beaa887776fL50

One per-RPM-requested. See 2062526#c21, that's where we started trying to debug this issue. (apologies for this and a number of the ensuing comments being private, they have non-public info in them)

@ggainey
Copy link
Contributor Author

ggainey commented Jun 13, 2022

A good place to start might also be to just throw thread_sensitive=True on all the callsites. There's this issue which implies that it might be necessary? https://code.djangoproject.com/ticket/31905#comment:14

That's the default, so we shouldn't have to make it explicit.

@ggainey
Copy link
Contributor Author

ggainey commented Jun 13, 2022

One per-RPM-requested. See 2062526#c21, that's where we started trying to debug this issue. (apologies for this and a number of the ensuing comments being private, they have non-public info in them)

And just to be explicit - when I run on core/3.14 , I end the reproducer-run above with one database connection open - the one I'm using to query the number-of-open-connections with. On 3.16+, it's one-per-rpm.

@dkliban
Copy link
Member

dkliban commented Jun 13, 2022

I am not able to reproduce on Fedora 36.

Here is what I did:

(pulp) [vagrant@pulp3-source-fedora36 pulp-openapi-generator]$ ls -la /var/lib/pulp/media/
total 0
drwxrwxr-x. 1 vagrant pulp   0 Jun 13 20:07 .
drwxr-xr-x. 1 vagrant pulp 104 Jun 13 12:50 ..

Then I ran the script above and it ended with:

+-----------------------------------------+
| NOTE: binary data not shown in terminal |
+-----------------------------------------+

Connections at end:
 count 
-------
     1
(1 row)

And I can see that the artifacts got created:

(pulp) [vagrant@pulp3-source-fedora36 pulp-openapi-generator]$ ls -la /var/lib/pulp/media/artifact/
total 0
drwxr-xr-x. 1 vagrant pulp 156 Jun 13 20:10 .
drwxrwxr-x. 1 vagrant pulp  16 Jun 13 20:10 ..
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 05
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 06
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 0c
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 0d
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 28
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 2c
drwxr-xr-x. 1 vagrant pulp 248 Jun 13 20:10 2d
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 2f
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 41
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 50
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 54
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 58
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 59
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 62
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 65
drwxr-xr-x. 1 vagrant pulp 248 Jun 13 20:10 68
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 6c
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 71
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 72
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 73
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 7a
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 7d
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 88
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 98
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 9a
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 9e
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 a0
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 a7
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 b4
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 b5
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 b8
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 c2
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 c4
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 cd
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 d6
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 e0
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 f4
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 fa
drwxr-xr-x. 1 vagrant pulp 124 Jun 13 20:10 fd

I have following packages in my virtualenv:

aiodns==3.0.0
aiofiles==0.8.0
aiohttp==3.8.1
aiohttp-xmlrpc==1.5.0
aioredis==2.0.1
aiosignal==1.2.0
alabaster==0.7.12
asgiref==3.5.2
asttokens==2.0.5
async-timeout==4.0.2
asyncio-throttle==1.0.2
asynctest==0.13.0
attrs==21.4.0
azure-core==1.24.1
azure-storage-blob==12.12.0
Babel==2.10.1
backcall==0.2.0
backoff==2.1.2
black==22.3.0
boto3==1.24.7
botocore==1.27.7
build==0.8.0
certifi==2022.5.18.1
cffi==1.15.0
charset-normalizer==2.0.12
check-manifest==0.48
click==8.1.3
click-default-group==1.2.2
click-shell==2.1
codecov==2.1.12
commonmark==0.9.1
contextlib2==21.6.0
coverage==6.4.1
createrepo-c==0.20.0
cryptography==37.0.2
decorator==5.1.1
defusedxml==0.7.1
Deprecated==1.2.13
diff-match-patch==20200713
Django==3.2.13
django-crispy-forms==1.14.0
django-currentuser==0.5.3
django-extensions==3.1.5
django-filter==21.1
django-guid==3.3.0
django-import-export==2.8.0
django-lifecycle==1.0.0
django-readonly-field==1.1.1
django-storages==1.12.3
djangorestframework==3.13.1
djangorestframework-queryfields==1.0.0
docutils==0.17.1
drf-access-policy==1.1.1
drf-nested-routers==0.93.4
drf-spectacular==0.22.1
dynaconf==3.1.9
ecdsa==0.17.0
et-xmlfile==1.1.0
execnet==1.9.0
executing==0.8.3
flake8==4.0.1
flake8-black==0.3.3
frozenlist==1.3.0
future==0.18.2
gunicorn==20.1.0
httpie==3.2.1
httpie-jwt-auth==0.4.0
httplib2==0.20.4
idna==3.3
imagesize==1.3.0
incremental==21.3.0
inflection==0.5.1
iniconfig==1.1.1
ipython==8.4.0
isodate==0.6.1
jedi==0.18.1
Jinja2==3.1.2
jmespath==1.0.0
jsonschema==4.6.0
libcomps==0.1.15.post1
lxml==4.9.0
m2r==0.2.1
MarkupPy==1.14
MarkupSafe==2.1.1
matplotlib-inline==0.1.3
mccabe==0.6.1
mistune==0.8.4
mock==4.0.3
msrest==0.7.0
multidict==6.0.2
mypy-extensions==0.4.3
naya==1.1.1
oauthlib==3.2.0
odfpy==1.4.1
openpyxl==3.0.10
packaging==21.3
parso==0.8.3
pathspec==0.9.0
pep517==0.12.0
pexpect==4.8.0
pickleshare==0.7.5
plantuml==0.3.0
platformdirs==2.5.2
pluggy==1.0.0
plumbum==1.7.2
productmd==1.33
prompt-toolkit==3.0.29
proxy.py==2.4.3
psycopg2==2.9.3
ptyprocess==0.7.0
pulp-certguard-client==1.5.2
# Editable Git install (pulp-cli==0.15.0.dev0) with either a deleted local remote or invalid URI:
# 'git@github.com:/dkliban/pulp-cli'
-e /home/vagrant/devel/pulp-cli
# Editable Git install (pulp-container==2.13.0.dev0) with either a deleted local remote or invalid URI:
# 'git@github.com:/dkliban/pulp_container'
-e /home/vagrant/devel/pulp_container
pulp-container-client==0.0.0
# Editable Git install (pulp-file==1.11.0.dev0) with either a deleted local remote or invalid URI:
# 'git@github.com:/dkliban/pulp_file'
-e /home/vagrant/devel/pulp_file
pulp-file-client==1.11.0.dev0
# Editable Git install (pulp-rpm==3.18.0.dev0) with either a deleted local remote or invalid URI:
# 'git@github.com:/dkliban/pulp_rpm'
-e /home/vagrant/devel/pulp_rpm
pulp-smash @ git+https://github.com/pulp/pulp-smash.git@ca6c35d1455f39e5e8a618bc7833624f9d7d5983
# Editable Git install (pulpcore==3.20.0.dev0) with either a deleted local remote or invalid URI:
# 'git@github.com:/dkliban/pulpcore'
-e /home/vagrant/devel/pulpcore
pulpcore-client==3.20.0.dev0
pure-eval==0.2.2
py==1.11.0
pycairo==1.21.0
pycares==4.1.2
pycodestyle==2.8.0
pycparser==2.21
pycryptodomex==3.14.1
pyflakes==2.4.0
Pygments==2.12.0
PyGObject==3.42.1
pygtrie==2.4.2
pyjwkest==1.4.2
PyJWT==2.4.0
pyparsing==3.0.9
pyrsistent==0.18.1
PySocks==1.7.1
pytest==7.1.2
pytest-custom-exit-code==0.3.0
pytest-django==4.5.2
pytest-forked==1.4.0
pytest-xdist==2.5.0
python-dateutil==2.8.2
python-gnupg==0.4.9
pytz==2022.1
pyxdg==0.28
PyYAML==5.4.1
redis==4.3.3
requests==2.28.0
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
rich==12.4.4
s3transfer==0.6.0
schema==0.7.5
six==1.16.0
snowballstemmer==2.2.0
solv==0.7.22
Sphinx==5.0.1
sphinx-rtd-theme==1.0.0
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-httpdomain==1.8.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-openapi==0.7.0
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
sqlparse==0.4.2
stack-data==0.2.0
tablib==3.2.1
toml==0.10.2
tomli==2.0.1
towncrier==21.9.0
traitlets==5.2.2.post1
trustme==0.9.0
typing_extensions==4.2.0
uritemplate==4.1.1
url-normalize==1.4.3
urllib3==1.26.9
urlman==2.0.1
wcwidth==0.2.5
whitenoise==6.1.0
wrapt==1.14.1
xlrd==2.0.1
xlwt==1.3.0
yarl==1.7.2

I am running postgresql 14.1.

@ggainey
Copy link
Contributor Author

ggainey commented Jun 14, 2022

On pulpcore/3.16, on pulp3-source-f36, with postgresql-14.1-3.fc36.x86_64 , I still end up with 36 open connections. Diff between the versions in the prev comment and the output of pip freeze on my test setup:

(pulp) [vagrant@pulp3-source-fedora36 hold]$ diff dkliban_f36 ggainey_f36 
2c2
< aiofiles==0.8.0
---
> aiofiles==0.7.0
16c16
< Babel==2.10.1
---
> Babel==2.10.2
18c18
< backoff==2.1.2
---
> backoff==1.11.1
20,21c20,21
< boto3==1.24.7
< botocore==1.27.7
---
> boto3==1.24.8
> botocore==1.27.8
27c27
< click==8.1.3
---
> click==8.0.4
29d28
< click-shell==2.1
35c34
< cryptography==37.0.2
---
> cryptography==35.0.0
38a38
> dictdiffer==0.9.0
45,47c45,48
< django-guid==3.3.0
< django-import-export==2.8.0
< django-lifecycle==1.0.0
---
> django-guardian==2.4.0
> django-guid==3.2.2
> django-import-export==2.6.1
> django-lifecycle==0.9.6
50c51
< djangorestframework==3.13.1
---
> djangorestframework==3.12.4
54,55c55,56
< drf-nested-routers==0.93.4
< drf-spectacular==0.22.1
---
> drf-nested-routers==0.93.3
> drf-spectacular==0.20.1
57d57
< ecdsa==0.17.0
64d63
< future==0.18.2
77c76
< Jinja2==3.1.2
---
> Jinja2==3.0.3
89c88
< msrest==0.7.0
---
> msrest==0.7.1
111,125c110,114
< pulp-certguard-client==1.5.2
< # Editable Git install (pulp-cli==0.15.0.dev0) with either a deleted local remote or invalid URI:
< # 'git@github.com:/dkliban/pulp-cli'
< -e /home/vagrant/devel/pulp-cli
< # Editable Git install (pulp-container==2.13.0.dev0) with either a deleted local remote or invalid URI:
< # 'git@github.com:/dkliban/pulp_container'
< -e /home/vagrant/devel/pulp_container
< pulp-container-client==0.0.0
< # Editable Git install (pulp-file==1.11.0.dev0) with either a deleted local remote or invalid URI:
< # 'git@github.com:/dkliban/pulp_file'
< -e /home/vagrant/devel/pulp_file
< pulp-file-client==1.11.0.dev0
< # Editable Git install (pulp-rpm==3.18.0.dev0) with either a deleted local remote or invalid URI:
< # 'git@github.com:/dkliban/pulp_rpm'
< -e /home/vagrant/devel/pulp_rpm
---
> -e git+ssh://git@github.com/ggainey/pulp-certguard.git@fe3ad7b1b97c04a5188f2cde3c3e46bb21135585#egg=pulp_certguard
> -e git+ssh://git@github.com/ggainey/pulp-cli.git@13ba7dbf69c3068c07cc0262dcd55f2fd7ea432f#egg=pulp_cli
> -e git+ssh://git@github.com/ggainey/pulp_file.git@e8ff7b09e95de6a677b707eecccd3d9e79fbaea8#egg=pulp_file
> pulp-file-client==1.10.2
> -e git+ssh://git@github.com/ggainey/pulp_rpm.git@22ece9f5e820fe7c4084d9c46b6ec99c7848e6a0#egg=pulp_rpm
127,130c116,117
< # Editable Git install (pulpcore==3.20.0.dev0) with either a deleted local remote or invalid URI:
< # 'git@github.com:/dkliban/pulpcore'
< -e /home/vagrant/devel/pulpcore
< pulpcore-client==3.20.0.dev0
---
> -e git+ssh://git@github.com/ggainey/pulpcore.git@a82ffa7259d506499a2bbad9b2ceecee74ebbd39#egg=pulpcore
> pulpcore-client==3.19.0
137d123
< pycryptodomex==3.14.1
142,143c128
< pyjwkest==1.4.2
< PyJWT==2.4.0
---
> pyOpenSSL==22.0.0
157c142
< redis==4.3.3
---
> redis==4.1.4
167c152
< Sphinx==5.0.1
---
> Sphinx==4.3.2
187d171
< url-normalize==1.4.3
191c175
< whitenoise==6.1.0
---
> whitenoise==5.3.0
194a179
> xmltodict==0.13.0

@ggainey
Copy link
Contributor Author

ggainey commented Jun 14, 2022

Running pulp3-source-fedora36, and pulpcore/main, I was not able to reproduce the problem.

The difference between f36/core-3.16 and f36/main is the following:

(pulp) [vagrant@pulp3-source-fedora36 hold]$ diff ggainey_f36 f36_main
2c2
< aiofiles==0.7.0
---
> aiofiles==0.8.0
18c18
< backoff==1.11.1
---
> backoff==2.1.2
34c34
< cryptography==35.0.0
---
> cryptography==37.0.2
45,48c45,47
< django-guardian==2.4.0
< django-guid==3.2.2
< django-import-export==2.6.1
< django-lifecycle==0.9.6
---
> django-guid==3.3.0
> django-import-export==2.8.0
> django-lifecycle==1.0.0
51c50
< djangorestframework==3.12.4
---
> djangorestframework==3.13.1
55,56c54,55
< drf-nested-routers==0.93.3
< drf-spectacular==0.20.1
---
> drf-nested-routers==0.93.4
> drf-spectacular==0.22.1
76c75
< Jinja2==3.0.3
---
> Jinja2==3.1.2
110c109,110
< -e git+ssh://git@github.com/ggainey/pulp-certguard.git@fe3ad7b1b97c04a5188f2cde3c3e46bb21135585#egg=pulp_certguard
---
> -e git+ssh://git@github.com/ggainey/pulp-certguard.git@1d4110215607af56433789d7afd9e9b1b2afe7ec#egg=pulp_certguard
> pulp-certguard-client==1.5.2
112c112
< -e git+ssh://git@github.com/ggainey/pulp_file.git@e8ff7b09e95de6a677b707eecccd3d9e79fbaea8#egg=pulp_file
---
> -e git+ssh://git@github.com/ggainey/pulp_file.git@ad9870ff96494834ef393b8b17d7ec67cb73ba51#egg=pulp_file
114c114
< -e git+ssh://git@github.com/ggainey/pulp_rpm.git@22ece9f5e820fe7c4084d9c46b6ec99c7848e6a0#egg=pulp_rpm
---
> -e git+ssh://git@github.com/ggainey/pulp_rpm.git@5244cfa7c6dec099cebef2c6df7307f7fa39141f#egg=pulp_rpm
116c116
< -e git+ssh://git@github.com/ggainey/pulpcore.git@a82ffa7259d506499a2bbad9b2ceecee74ebbd39#egg=pulpcore
---
> -e git+ssh://git@github.com/ggainey/pulpcore.git@9e63e8d1c5a4a97b85a5a16b3230a8c9cdb71c6a#egg=pulpcore
142c142
< redis==4.1.4
---
> redis==4.3.3
152c152
< Sphinx==4.3.2
---
> Sphinx==5.0.1
171a172
> url-normalize==1.4.3
175c176
< whitenoise==5.3.0
---
> whitenoise==6.2.0

Investigation continues.

@mdellweg
Copy link
Member

And i thought Django made thread_sensitive=True default in a bugfix commit somewhere.

@dkliban
Copy link
Member

dkliban commented Jun 14, 2022

3.16.8 is missing 2 commits that are needed to resolve this issue: #1684 and #1679

@ggainey
Copy link
Contributor Author

ggainey commented Jun 14, 2022

Independently confirmed - cherry-picking the commits from the above PRs into core/3.16 fixes the problem.

dralley added a commit to dralley/pulpcore that referenced this issue Jun 14, 2022
closes pulp#2816
(cherry picked from commit 1a8c7c5)
(cherry picked from commit d492eac)
dralley added a commit to dralley/pulpcore that referenced this issue Jun 14, 2022
closes pulp#2816
(cherry picked from commit 1a8c7c5)
(cherry picked from commit d492eac)
dralley added a commit to dralley/pulpcore that referenced this issue Jun 14, 2022
closes pulp#2816
(cherry picked from commit 1a8c7c5)
(cherry picked from commit d492eac)
dkliban pushed a commit that referenced this issue Jun 15, 2022
closes #2816
(cherry picked from commit 1a8c7c5)
(cherry picked from commit d492eac)
@dralley dralley closed this as completed Jul 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants