Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pre-commit hooks to delete the containers they create #37269

Closed
2 tasks done
rawwar opened this issue Feb 9, 2024 · 6 comments · Fixed by #37277
Closed
2 tasks done

pre-commit hooks to delete the containers they create #37269

rawwar opened this issue Feb 9, 2024 · 6 comments · Fixed by #37277
Labels
area:dev-tools kind:feature Feature Requests pending-response

Comments

@rawwar
Copy link
Contributor

rawwar commented Feb 9, 2024 •
edited

Description

For every single pre-commit run, it seems we use docker to run several checks. But, these containers are not removed by default. Hence, I can see something as below in my docker desktop:

image

Use case/motivation

Would be nice if we can just use --rm when running the container or some other possible mechanism so that they are removed right after they are stopped.

Related issues

Could not find anything related.

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@rawwar rawwar added kind:feature Feature Requests needs-triage label for new issues that we didn't triage yet labels Feb 9, 2024
@boring-cyborg boring-cyborg
Copy link

boring-cyborg bot commented Feb 9, 2024

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

@Taragolis Taragolis added area:dev-tools and removed needs-triage label for new issues that we didn't triage yet labels Feb 9, 2024
@potiuk
Copy link
Member

potiuk commented Feb 9, 2024

  1. Are you on latest main ?

  2. Can you look at the containers and see what commands they used / logs ?

  3. Is it likely that you simply Ctrl-C'd those pre-commits while running ?

We do remove all containers (and that has been improved some 2 months ago where I switched all pre-commits to use docker-compose under the hood and remove all of the remaining containers withdocker-compose down after they do the job (one of the reasons to unify that was precisely to not forget about removing the containers).

But maybe there is some remaining one in some specific circumstances?

Can you post some details on what you did and about those containers that are remaining ?

@potiuk
Copy link
Member

potiuk commented Feb 9, 2024

BTW. I just run pre-commit run --all-files and I see no containers remaining after that.

@rawwar
Copy link
Contributor Author

rawwar commented Feb 9, 2024 •
edited

I also have pre-commits' for pre-push enabled. I think containers are created when running them. I will re-run and confirm.

Are you on latest main ?

I am using main branch.

Can you look at the containers and see what commands they used / logs ?

I will share them in sometime.

@rawwar
Copy link
Contributor Author

rawwar commented Feb 9, 2024 •
edited

Update:

While I was working on pinecone provider update, I tried to add few work in progress files and commit. But, they were failing pre-commit checks. That's when 4 containers were left behind.

Command used for all of the left-behind containers is "/opt/airflow/scripts/in_container/run_fix_ownership.py"

potiuk added a commit to potiuk/airflow that referenced this issue Feb 9, 2024
@potiuk
The fix-ownership command missed --rm flag and left dangling containers
a9602e3 
Fixes: apache#37269
@potiuk potiuk mentioned this issue Feb 9, 2024
Merged
potiuk added a commit to potiuk/airflow that referenced this issue Feb 9, 2024
@potiuk
The fix-ownership command missed --rm flag and left dangling containers
eda5576 
Fixes: apache#37269
@potiuk
Copy link
Member

potiuk commented Feb 9, 2024

Ah cool. It's not pre-commit.

It's breeze trying to make sure that the files created are owned by host user. Only happens on Linux when docker is not rootless - because then files are created with root owner and we want to make sure that you can still own them as host user.

Fix in #37277

potiuk added a commit that referenced this issue Feb 9, 2024
@potiuk
The fix-ownership command missed --rm flag and left dangling containe…
7835fd2 
…rs (#37277)

Fixes: #37269
satish-chinthanippu added a commit to Teradata/airflow that referenced this issue Feb 9, 2024
@satish-chinthanippu
Squashed commit of the following:
db35783 
commit 58279b8
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Sat Feb 10 00:12:42 2024 +0530

    Update teradata.py

commit ed75e3e
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Sat Feb 10 00:09:24 2024 +0530

    D401 support in Teradata Provider

commit f56bede
Merge: e859a1d 00ed467
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Fri Feb 9 23:58:15 2024 +0530

    Merge remote-tracking branch 'upstream/main' into pr_teradata_release_1.0.0

commit 00ed467
Author: Vincent <97131062+vincbeck@users.noreply.github.com>
Date:   Fri Feb 9 13:13:36 2024 -0500

    D401 support in fab provider (apache#37283)

commit 48bfb1a
Author: Niko Oliveira <onikolas@amazon.com>
Date:   Fri Feb 9 08:43:32 2024 -0800

    Merge all ECS executor configs following recursive python dict update (apache#37137)

    Also document the behaviour and interaction between exec_config and
    run_task_kwargs config

commit 8317ed9
Author: Amogh Desai <amoghrajesh1999@gmail.com>
Date:   Fri Feb 9 20:17:38 2024 +0530

    Updating the README and visuals for breeze build-docs (apache#37276)

commit 17945fc
Author: Kalyan <kalyan.ben10@live.com>
Date:   Fri Feb 9 20:16:33 2024 +0530

    D401 fixes in Pinecone provider (apache#37270)

commit ab9e2e1
Author: Kalyan <kalyan.ben10@live.com>
Date:   Fri Feb 9 20:15:31 2024 +0530

    fix: D401 lint issues in airflow core (apache#37274)

commit 7835fd2
Author: Jarek Potiuk <jarek@potiuk.com>
Date:   Fri Feb 9 14:59:33 2024 +0100

    The fix-ownership command missed --rm flag and left dangling containers (apache#37277)

    Fixes: apache#37269

commit e859a1d
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Fri Feb 9 19:21:28 2024 +0530

    Update teradata.py

commit d1c08e1
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Fri Feb 9 19:18:37 2024 +0530

    Update teradata.py

commit e5ac0ec
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Fri Feb 9 05:41:42 2024 -0800

    static check issue is fixed

commit ce490f7
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Fri Feb 9 19:10:36 2024 +0530

    static format issue fixed

commit f9498c5
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Fri Feb 9 05:36:36 2024 -0800

    static check issue is fixed

commit 77bddae
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Fri Feb 9 18:23:18 2024 +0530

    common sql unit test failure fixed

commit 9f4f208
Author: Aleksey Kirilishin <54231417+avkirilishin@users.noreply.github.com>
Date:   Fri Feb 9 15:53:04 2024 +0300

    Fix the bug that affected the DAG end date. (apache#36144)

commit 0f8dfeb
Author: Sudipto Baral <sudiptobaral.me@gmail.com>
Date:   Fri Feb 9 06:54:23 2024 -0500

    fix: update  hyperlink to the new documentation section for local virtualenv setup. (apache#37272)

    Signed-off-by: sudipto baral <sudiptobaral.me@gmail.com>
sunank200 pushed a commit to astronomer/airflow that referenced this issue Feb 21, 2024
@potiuk @sunank200
The fix-ownership command missed --rm flag and left dangling containe…
78f45f7 
…rs (apache#37277)

Fixes: apache#37269
abhishekbhakat pushed a commit to abhishekbhakat/my_airflow that referenced this issue Mar 5, 2024
@potiuk @abhishekbhakat
The fix-ownership command missed --rm flag and left dangling containe…
c65095c 
…rs (apache#37277)

Fixes: apache#37269
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:dev-tools kind:feature Feature Requests pending-response
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

The fix-ownership command missed --rm flag and left dangling containers potiuk/airflow
3 participants
@potiuk @Taragolis @rawwar