Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Disk space leak due to empty folders creation #484

Closed
EgorSemenov opened this issue Aug 22, 2023 · 1 comment · Fixed by #500
Closed

Disk space leak due to empty folders creation #484

EgorSemenov opened this issue Aug 22, 2023 · 1 comment · Fixed by #500
Labels
good first issue Good for newcomers
Milestone
1.1.0

Comments

@EgorSemenov
Copy link
Contributor

EgorSemenov commented Aug 22, 2023 •
edited
Loading

astronomer-cosmos 1.0.5:
Scheduler Error(dags weren't executed):
FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/usr/local/airflow']

It was thrown because disk space was over by reason of the creation of empty temp folders.

This line should be updated:

astronomer-cosmos/cosmos/operators/virtualenv.py

Line 53 in c238a0b

self._venv_tmp_dir = TemporaryDirectory()

@EgorSemenov
Copy link
Contributor Author

@tatiana

@tatiana tatiana added this to the 1.1.0 milestone Aug 22, 2023
@tatiana tatiana added the good first issue Good for newcomers label Aug 22, 2023
@tatiana tatiana mentioned this issue Aug 30, 2023
Merged
tatiana added a commit that referenced this issue Aug 30, 2023
@tatiana
Fix incorrect temporary directory creation in VirtualenvOperator init (…
9387c45 
…#500)

As part of adding MyPy checks, we introduced an issue of creating
temporary directories during the VirtualenvOperator initialization. This
leads to Cosmos creating unnecessary empty temporary directories in the
user scheduler during parse time.

Fixes: #484
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
good first issue Good for newcomers
Projects
None yet
Milestone
1.1.0
Development

Successfully merging a pull request may close this issue.

Fix incorrect temporary directory creation in VirtualenvOperator init astronomer/astronomer-cosmos
2 participants
@tatiana @EgorSemenov