Uploading large files fails on the "Too many open files" error

[fao89]

Author: @lubosmj (lmjachky)

Redmine Issue: 9634, https://pulp.plan.io/issues/9634

---

Steps to reproduce the behaviour:

(pulp) [vagrant@pulp3-source-fedora34 pulp_ostree]$  truncate -s 3G gentoo_root.img

(pulp) [vagrant@pulp3-source-fedora34 pulp_ostree]$ pulp artifact upload --file gentoo_root.img
Uploading file gentoo_root.img
................[truncated the number of dots]...................Upload complete. Creating artifact.
Started background task /pulp/api/v3/tasks/2d3cf569-2d5c-449d-9ac6-7a53ab5f3a30/
........Error: Task /pulp/api/v3/tasks/2d3cf569-2d5c-449d-9ac6-7a53ab5f3a30/ failed: '[Errno 24] Too many open files: '/var/lib/pulp/media/upload/ec2d5c92-57e9-4191-ae95-ff4b9ec94353''

(pulp) [vagrant@pulp3-source-fedora34 pulp_ostree]$ ls -la gentoo_root.img
-rw-r--r--. 1 vagrant vagrant 3221225472 Dec 13 11:32 gentoo_root.img

Traceback:

Task 2d3cf569-2d5c-449d-9ac6-7a53ab5f3a30 failed ([Errno 24] Too many open files: '/var/lib/pulp/media/upload/ec2d5c92-57e9-4191-ae95-ff4b9ec94353')
pulp [3a3a9ea662994f609eea7d43ac8f30aa]: pulpcore.tasking.pulpcore_worker:INFO:   File "/home/vagrant/devel/pulpcore/pulpcore/tasking/pulpcore_worker.py", line 362, in _perform_task
    result = func(*args, **kwargs)
  File "/home/vagrant/devel/pulpcore/pulpcore/app/tasks/upload.py", line 31, in commit
    temp_file.write(chunk.file.read())
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/files/utils.py", line 42, in <lambda>
    read = property(lambda self: self.file.read)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/fields/files.py", line 45, in _get_file
    self._file = self.storage.open(self.name, 'rb')
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/files/storage.py", line 38, in open
    return self._open(name, mode)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/files/storage.py", line 238, in _open
    return File(open(self.path(name), mode))

[fao89]

From: rverdile (rverdile)
Date: 2021-12-20T16:31:30Z

---

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2034317

[fao89]

From: pulpbot (pulpbot)
Date: 2022-01-03T17:34:19Z

---

PR: #1790

[fao89]

From: @ipanova (ipanova@redhat.com)
Date: 2022-01-04T11:46:19Z

---

Applied in changeset commit:pulpcore|dcbf9357c20480bf8d94b9d8740e491be1fa661f.