Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

astro dev pytest generated artifacts not persistant #1387

Closed
2 of 3 tasks
SebOpo opened this issue Sep 13, 2023 · 4 comments
Closed
2 of 3 tasks

astro dev pytest generated artifacts not persistant #1387

SebOpo opened this issue Sep 13, 2023 · 4 comments

Comments

@SebOpo
Copy link

SebOpo commented Sep 13, 2023

Describe the bug

Acorrindg to docs https://docs.astronomer.io/astro/cli/astro-dev-pytest all files saved in include folder should be accessible outside contianer. In v1.19.0 this feature was dissabled: 046705c
Without volume mounts all generated files i.e code coverage or test reports are gone along with the pytest container .

astro dev pytest --args "--cov --cov-report xml:include/coverage.xml"

What CLI Version did you experience this bug?

1.19.0+

This CLI bug is related to which Astronomer Platform?

  • Astro
  • Software
  • None/Unknown

What Operating System is the above CLI installed on?

🪜 Steps To Reproduce

Run

astro dev pytest --args "--cov --cov-report xml:include/coverage.xml"

Check if "coverage.xml" file exists in include folder outside docker container after test has finished.

@sunkickr
Copy link
Contributor

We removed the volumes because they were causing tests and dags to be missing in certain CI/CD tools we will have to think of a way to fix this issue with this in mind

@sunkickr
Copy link
Contributor

@neel-astro idea is to simulate the old behavior by coping the include folder from the docker container to the local include folder after the pytest runs

@zach-nicoll-wcq
Copy link
Contributor

zach-nicoll-wcq commented Nov 27, 2023

Would love to see this fixed. In fact, would love to fix it and open a PR.

I've found this line that implies the test report is being copied out of the container using docker cp.

However, it doesn't look like htmlReport is being set to true at any point, so this isn't running.
Additionally, this doesn't take into account .xml coverage outputs.

Could we change the command to copy all *.html and *.xml files every time, regardless of htmlReport flag state?

Not sure who to @ here, so @sunkickr @neel-astro thoughts?


For anyone who needs a quick solution to this, we are using this yucky command to achieve the desired outcome:

astro dev start
docker exec -u 1000 airflow-scheduler pytest --cov --cov-config=include/.coveragerc --cov-report=xml:include/coverage.xml --cov-report=html:include/html/

@zach-nicoll-wcq
Copy link
Contributor

Have created, which adheres to the docs: #1459

zach-nicoll-wcq added a commit to zach-nicoll-wcq/astro-cli that referenced this issue Dec 1, 2023
zach-nicoll-wcq added a commit to zach-nicoll-wcq/astro-cli that referenced this issue Dec 1, 2023
zach-nicoll-wcq added a commit to zach-nicoll-wcq/astro-cli that referenced this issue Dec 5, 2023
zach-nicoll-wcq added a commit to zach-nicoll-wcq/astro-cli that referenced this issue Dec 7, 2023
zach-nicoll-wcq added a commit to zach-nicoll-wcq/astro-cli that referenced this issue Dec 7, 2023
sunkickr added a commit that referenced this issue Jan 29, 2024
* fix astro deploy when dag deploy is off

* fix deploy flow

* add description

* fix dags

* fix dags

* fix dags

* fix dags

* fix lint

* fix lint

* fix dags

* fix lint

* fix lint

* list and selection process to core

* list/selection to core

* deployment list & selection to CORE

* broken but working on it

* update mocks

* fix log

* fix lint

* fixing worker queues

* worker-queue

* worker-queue work

* able to compile

* get create command working

* manually tested commands

* fix deploy

* fix lint

* remove commented out code

* fix some tests

* fix some deployment tests

* fix unit tests

* fix unit tests

* fix unit tests

* fix tests

* fix tests

* fix tests

* fix update tests

* fix test

* fix test

* fix test

* fix inspect and worker queue tests

* fix inspect

* fix inspect tests

* fix update tests

* fix variable modify

* fix setup tests

* fix root

* remove some comments

* remove comments

* fix some lint

* fix lint

* fix lint

* fix lint

* fix lint

* fix lint

* fix lint

* fix lint

* fix lint

* fix lint

* fix lint

* fix lint

* fix broken tests

* fix broken tests

* fix org switch

* fix org switch

* fix deploy tests

* fix setup tests

* Fixing tests

* Fixing tests

* test cloud to local

* add tests

* add tests

* add tests

* add tests

* fix test

* add test

* add test

* update deployment update tests

* add test

* fix lint

* fix test

* fix test

* add inspect test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix lint

* fix dag deploy enabled

* add tests

* fix test

* fix test

* add test

* add test

* fix lint

* clean code

* fix mock

* fix test

* fix test

* update audit logs command

* migrate audit logs

* remove astrohub api

* list and selection process to core

* list/selection to core

* deployment list & selection to CORE

* broken but working on it

* fixing worker queues

* worker-queue

* worker-queue work

* able to compile

* get create command working

* manually tested commands

* fix deploy

* remove commented out code

* fix some tests

* fix some deployment tests

* fix unit tests

* fix unit tests

* fix unit tests

* fix tests

* fix tests

* fix tests

* fix update tests

* fix test

* fix test

* fix test

* fix inspect and worker queue tests

* fix inspect

* fix inspect tests

* fix update tests

* fix variable modify

* fix setup tests

* fix root

* remove some comments

* remove comments

* fix some lint

* Fixing tests

* fix lint

* fix lint

* fix lint

* fix lint

* fix lint

* fix lint

* fix lint

* fix lint

* fix lint

* fix lint

* migrate deploy to core (#1411)

* migrate deploy to core

* fix lint

* fix lint

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix astro deploy when dag deploy is off

* fix deploy flow

* add description

* fix dags

* fix dags

* fix dags

* fix dags

* fix lint

* fix lint

* fix dags

* fix lint

* fix lint

* update mocks

* fix log

* fix lint

* remove pull (#1431)

* fix dag bundle copy (#1433)

* pinning version 1.20.0 (#1434)

* Revert "pinning version 1.20.0 (#1434)" (#1435)

This reverts commit cc04467.

* Releasing 1.20.0 (#1436)

* pinning version back to 1.19.3 (#1439)

* Set default version back to v1.20.0 (#1441)

Now that the issue has been fixed
(https://status.astronomer.io/incidents/4z4r646s6mkx) we can make this
version the default again

* catch API error (#1440)

* Fix deployment connections and airflow variable commands (#1442)

* Fix deployment connections and airflow variable commands

* fixing lint

* Fix variable name

* Fix url in test

* Fix url in test

* pin 1.20.1 (#1443)

* allow ASTRO_DOMAIN env var to override context (#1451)

* Updating error message (#1456)

* [pre-commit.ci] pre-commit autoupdate (#1447)

updates:
- [github.com/psf/black: 23.10.1 → 23.11.0](psf/black@23.10.1...23.11.0)
- [github.com/adrienverge/yamllint.git: v1.32.0 → v1.33.0](https://github.com/adrienverge/yamllint.git/compare/v1.32.0...v1.33.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix deployment file create for kube deployment (#1454)

* fix dpeloyment file create for kube deployment

* undo comment

* undelete file

* fix test

* fix test

* add --image flag for image only deploy (#1446)

* add --image flag for image only deploy

* don't send tarball on image only

* image only deploy works

* image only deploy works

* fix lint

* add errors

* Fix issues from bug bash (#1460)

* Fix issues from bug bash

* refactor code

* Update cloud/deploy/deploy.go

Co-authored-by: kushalmalani <kushal@astronomer.io>

* fix test

* fix test

---------

Co-authored-by: kushalmalani <kushal@astronomer.io>

* override the domain in more places (#1463)

* FIx astro deploy <deployment-id> --image (#1464)

* astro deploy <deployment-id> --image does not work

* update tests

* update tests

* pin 1.21.0 (#1465)

* Add v1beta1 API Clients (#1444)

* add v1beta1 api

* fix GetPublicRESTAPIURL

* fix GetPublicRESTAPIURL

* fix GetPublicRESTAPIURL

* made changes from code review

* fix error

* empty commit

* empty commit

* Update astro-client-platform-core/client.test.go

Co-authored-by: Vandy Liu <33995460+vandyliu@users.noreply.github.com>

* Update astro-client-iam-core/client.test.go

Co-authored-by: Vandy Liu <33995460+vandyliu@users.noreply.github.com>

---------

Co-authored-by: Vandy Liu <33995460+vandyliu@users.noreply.github.com>

* Set CLI environment management flag default to true (#1467)

* fix: #1387 persist include dir after running pytest (#1459)

* Migrate organization to use core (#1450)

* add v1beta1 api

* fix GetPublicRESTAPIURL

* fix GetPublicRESTAPIURL

* fix GetPublicRESTAPIURL

* migrate organization to v1beta1

* made changes from code review

* fix error

* empty commit

* Update root.go

* gofumpt

* Update setup.go

* Update setup.go

* Update setup_test.go

* fix pytest

---------

Co-authored-by: David Koenitzer <davidkoenit@gmail.com>

* [pre-commit.ci] pre-commit autoupdate (#1470)

updates:
- [github.com/psf/black: 23.11.0 → 23.12.0](psf/black@23.11.0...23.12.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fixing unit tests to use LocalPlatform Config (#1471)

* Fixing unit tests to use LocalPlatform Config

* Fixing tests

* Fixing tests

* Fixing tests

* fix lint

* fix broken tests

* fix broken tests

* fix org switch

* fix org switch

* fix deploy tests

* fix setup tests

* Fixing tests

* test cloud to local

* add tests

* add tests

* add tests

* add tests

* fix test

* add test

* add test

* update deployment update tests

* add test

* fix lint

* fix test

* fix test

* add inspect test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix lint

* Bump github.com/containerd/containerd from 1.5.18 to 1.6.26 (#1472)

Bumps [github.com/containerd/containerd](https://github.com/containerd/containerd) from 1.5.18 to 1.6.26.
- [Release notes](https://github.com/containerd/containerd/releases)
- [Changelog](https://github.com/containerd/containerd/blob/main/RELEASES.md)
- [Commits](containerd/containerd@v1.5.18...v1.6.26)

---
updated-dependencies:
- dependency-name: github.com/containerd/containerd
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#1474)

updates:
- [github.com/psf/black: 23.12.0 → 23.12.1](psf/black@23.12.0...23.12.1)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Bump golang.org/x/crypto from 0.14.0 to 0.17.0 (#1473)

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.14.0 to 0.17.0.
- [Commits](golang/crypto@v0.14.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix dag deploy enabled

* add tests

* fix test

* fix test

* add test

* add test

* fix lint

* Updating code owners (#1480)

* Updating code owners

* adding individual members

* Updating code owners (#1482)

* Fix login and organization list/switch (#1476)

* fix not finshed

* fix login, organization list/switch

* fix version

* clean code

* Build Secrets Flag (#1478)

* add build secrets flag

* fix tests

* fix lint

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* add test

* add test

* remove debugging messages

* Update cmd/airflow.go

Co-authored-by: Jake Witz <74574233+jwitz@users.noreply.github.com>

---------

Co-authored-by: Jake Witz <74574233+jwitz@users.noreply.github.com>

* fix mock

* fix test

* update audit logs command

* migrate audit logs

* remove astrohub api

* fix errors

* fix lint

* fix lint

* fix test

* fix lint and test

* audit logs test

* fix test

* fix lint

* fix merge issue

* fix lint

* remove debug comments

* remove debug comment

* remove audit logs files

* Update cloud/organization/organization.go

Co-authored-by: Vandy Liu <33995460+vandyliu@users.noreply.github.com>

* update tests

* fix lint

* fix tests

* fix lint

* add assertions

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Kushal Malani <kushal@astronomer.io>
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: Mehul Goyal <mehul.goyal@astronomer.io>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Vandy Liu <33995460+vandyliu@users.noreply.github.com>
Co-authored-by: Julian LaNeve <lanevejulian@gmail.com>
Co-authored-by: Zach Nicoll <151692348+zach-nicoll-wcq@users.noreply.github.com>
Co-authored-by: Shalin Patel <42554035+PatelShalin@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jake Witz <74574233+jwitz@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants