Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: clean install & share pkg build #15986

Merged
merged 26 commits into from
Dec 13, 2022
Merged

CI: clean install & share pkg build #15986

merged 26 commits into from
Dec 13, 2022

Conversation

Borda
Copy link
Member

@Borda Borda commented Dec 9, 2022

What does this PR do?

  • share building stage
  • test package at completely clean env

Before submitting

  • Was this discussed/approved via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you list all the breaking changes introduced by this pull request?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

cc @Borda @carmocca @akihironitta

@Borda Borda changed the title abstract pkg build abstract pkg build [wip] Dec 9, 2022
@Borda Borda marked this pull request as ready for review December 9, 2022 13:58
@github-actions
Copy link
Contributor

github-actions bot commented Dec 9, 2022

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow
Check ID Status
pl-cpu (macOS-11, pytorch, 3.8, 1.11) success
pl-cpu (macOS-11, pytorch, 3.9, 1.12) success
pl-cpu (macOS-11, pytorch, 3.10, 1.13) success
pl-cpu (macOS-11, pytorch, 3.8, 1.10, oldest) success
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.10) success
pl-cpu (ubuntu-20.04, pytorch, 3.9, 1.11) success
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.12) success
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.13) success
pl-cpu (ubuntu-20.04, pytorch, 3.7, 1.10, oldest) success
pl-cpu (windows-2022, pytorch, 3.9, 1.11) success
pl-cpu (windows-2022, pytorch, 3.10, 1.12) success
pl-cpu (windows-2022, pytorch, 3.10, 1.13) success
pl-cpu (windows-2022, pytorch, 3.7, 1.10, oldest) success
pl-cpu (slow, macOS-11, pytorch, 3.7, 1.11) success
pl-cpu (slow, ubuntu-20.04, pytorch, 3.7, 1.11) success
pl-cpu (slow, windows-2022, pytorch, 3.7, 1.11) success
pl-cpu (macOS-11, lightning, 3.8, 1.13) success
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.13) success
pl-cpu (windows-2022, lightning, 3.8, 1.13) success

These checks are required after the changes to .github/workflows/ci-pytorch-tests.yml, src/pytorch_lightning/core/mixins/hparams_mixin.py, src/pytorch_lightning/utilities/parsing.py.

🟢 pytorch_lightning: Azure GPU
Check ID Status
pytorch-lightning (GPUs) success

These checks are required after the changes to src/pytorch_lightning/core/mixins/hparams_mixin.py, src/pytorch_lightning/utilities/parsing.py.

🟢 pytorch_lightning: Azure HPU
Check ID Status
pytorch-lightning (HPUs) success

These checks are required after the changes to src/pytorch_lightning/core/mixins/hparams_mixin.py, src/pytorch_lightning/utilities/parsing.py.

🟢 pytorch_lightning: Azure IPU
Check ID Status
pytorch-lightning (IPUs) success

These checks are required after the changes to src/pytorch_lightning/core/mixins/hparams_mixin.py, src/pytorch_lightning/utilities/parsing.py.

🟢 pytorch_lightning: Docs
Check ID Status
make-doctest (pytorch) success
make-html (pytorch) success

These checks are required after the changes to src/pytorch_lightning/core/mixins/hparams_mixin.py, src/pytorch_lightning/utilities/parsing.py, .github/workflows/docs-checks.yml.

🟢 lightning_lite: CPU workflow
Check ID Status
lite-cpu (macOS-11, lite, 3.8, 1.11) success
lite-cpu (macOS-11, lite, 3.9, 1.12) success
lite-cpu (macOS-11, lite, 3.10, 1.13) success
lite-cpu (macOS-11, lite, 3.7, 1.10, oldest) success
lite-cpu (ubuntu-20.04, lite, 3.8, 1.10) success
lite-cpu (ubuntu-20.04, lite, 3.9, 1.11) success
lite-cpu (ubuntu-20.04, lite, 3.10, 1.12) success
lite-cpu (ubuntu-20.04, lite, 3.10, 1.13) success
lite-cpu (ubuntu-20.04, lite, 3.7, 1.10, oldest) success
lite-cpu (windows-2022, lite, 3.9, 1.11) success
lite-cpu (windows-2022, lite, 3.10, 1.12) success
lite-cpu (windows-2022, lite, 3.10, 1.13) success
lite-cpu (windows-2022, lite, 3.7, 1.10, oldest) success
lite-cpu (macOS-11, lightning, 3.8, 1.13) success
lite-cpu (ubuntu-20.04, lightning, 3.8, 1.13) success
lite-cpu (windows-2022, lightning, 3.8, 1.13) success

These checks are required after the changes to .github/workflows/ci-lite-tests.yml.

🟢 lightning_app: Tests workflow
Check ID Status
app-pytest (macOS-11, app, 3.8, latest) success
app-pytest (macOS-11, app, 3.8, oldest) success
app-pytest (macOS-11, lightning, 3.9, latest) success
app-pytest (ubuntu-20.04, app, 3.8, latest) success
app-pytest (ubuntu-20.04, app, 3.8, oldest) success
app-pytest (ubuntu-20.04, lightning, 3.9, latest) success
app-pytest (windows-2022, app, 3.8, latest) success
app-pytest (windows-2022, app, 3.8, oldest) success
app-pytest (windows-2022, lightning, 3.8, latest) success

These checks are required after the changes to .github/workflows/ci-app-tests.yml, src/lightning_app/components/python/popen.py, src/lightning_app/components/python/tracer.py, src/lightning_app/core/app.py, src/lightning_app/core/flow.py, src/lightning_app/structures/dict.py, src/lightning_app/structures/list.py, src/lightning_app/utilities/packaging/build_config.py.

🟢 lightning_app: Examples
Check ID Status
app-examples (macOS-11, app, 3.9, latest) success
app-examples (macOS-11, app, 3.9, oldest) success
app-examples (macOS-11, lightning, 3.9, latest) success
app-examples (ubuntu-20.04, app, 3.9, latest) success
app-examples (ubuntu-20.04, app, 3.9, oldest) success
app-examples (ubuntu-20.04, lightning, 3.9, latest) success
app-examples (windows-2022, app, 3.9, latest) success
app-examples (windows-2022, app, 3.9, oldest) success
app-examples (windows-2022, lightning, 3.9, latest) success

These checks are required after the changes to .github/workflows/ci-app-examples.yml, src/lightning_app/components/python/popen.py, src/lightning_app/components/python/tracer.py, src/lightning_app/core/app.py, src/lightning_app/core/flow.py, src/lightning_app/structures/dict.py, src/lightning_app/structures/list.py, src/lightning_app/utilities/packaging/build_config.py.

🟢 lightning_app: Azure
Check ID Status
App.cloud-e2e success

These checks are required after the changes to src/lightning_app/components/python/popen.py, src/lightning_app/components/python/tracer.py, src/lightning_app/core/app.py, src/lightning_app/core/flow.py, src/lightning_app/structures/dict.py, src/lightning_app/structures/list.py, src/lightning_app/utilities/packaging/build_config.py.

🟢 lightning_app: Docs
Check ID Status
make-doctest (app) success
make-html (app) success

These checks are required after the changes to src/lightning_app/components/python/popen.py, src/lightning_app/components/python/tracer.py, src/lightning_app/core/app.py, src/lightning_app/core/flow.py, src/lightning_app/structures/dict.py, src/lightning_app/structures/list.py, src/lightning_app/utilities/packaging/build_config.py, .github/workflows/docs-checks.yml.

🟢 mypy
Check ID Status
mypy success

These checks are required after the changes to src/lightning_app/components/python/popen.py, src/lightning_app/components/python/tracer.py, src/lightning_app/core/app.py, src/lightning_app/core/flow.py, src/lightning_app/structures/dict.py, src/lightning_app/structures/list.py, src/lightning_app/utilities/packaging/build_config.py, src/pytorch_lightning/core/mixins/hparams_mixin.py, src/pytorch_lightning/utilities/parsing.py.

🟢 install
Check ID Status
install-pkg (ubuntu-22.04, app, 3.7) success
install-pkg (ubuntu-22.04, app, 3.10) success
install-pkg (ubuntu-22.04, lite, 3.7) success
install-pkg (ubuntu-22.04, lite, 3.10) success
install-pkg (ubuntu-22.04, pytorch, 3.7) success
install-pkg (ubuntu-22.04, pytorch, 3.10) success
install-pkg (ubuntu-22.04, lightning, 3.7) success
install-pkg (ubuntu-22.04, lightning, 3.10) success
install-pkg (ubuntu-22.04, notset, 3.7) success
install-pkg (ubuntu-22.04, notset, 3.10) success
install-pkg (macOS-12, app, 3.7) success
install-pkg (macOS-12, app, 3.10) success
install-pkg (macOS-12, lite, 3.7) success
install-pkg (macOS-12, lite, 3.10) success
install-pkg (macOS-12, pytorch, 3.7) success
install-pkg (macOS-12, pytorch, 3.10) success
install-pkg (macOS-12, lightning, 3.7) success
install-pkg (macOS-12, lightning, 3.10) success
install-pkg (macOS-12, notset, 3.7) success
install-pkg (macOS-12, notset, 3.10) success
install-pkg (windows-2022, app, 3.7) success
install-pkg (windows-2022, app, 3.10) success
install-pkg (windows-2022, lite, 3.7) success
install-pkg (windows-2022, lite, 3.10) success
install-pkg (windows-2022, pytorch, 3.7) success
install-pkg (windows-2022, pytorch, 3.10) success
install-pkg (windows-2022, lightning, 3.7) success
install-pkg (windows-2022, lightning, 3.10) success
install-pkg (windows-2022, notset, 3.7) success
install-pkg (windows-2022, notset, 3.10) success

These checks are required after the changes to .github/actions/pkg-check/action.yml, .github/actions/pkg-install/action.yml, .github/workflows/_build-packages.yml, .github/workflows/ci-pkg-install.yml, src/lightning_app/components/python/popen.py, src/lightning_app/components/python/tracer.py, src/lightning_app/core/app.py, src/lightning_app/core/flow.py, src/lightning_app/structures/dict.py, src/lightning_app/structures/list.py, src/lightning_app/utilities/packaging/build_config.py, src/pytorch_lightning/core/mixins/hparams_mixin.py, src/pytorch_lightning/utilities/parsing.py.


Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

@Borda Borda changed the title abstract pkg build [wip] CI: clean install & share pkg build [wip] Dec 9, 2022
@Borda
Copy link
Member Author

Borda commented Dec 9, 2022

it tries to load UI from the local destination instead of the one from the installed package:

FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/work/lightning/lightning/src/lightning_app/ui/index.html'

@github-actions github-actions bot added the app Generic label for Lightning App package label Dec 9, 2022
@Borda Borda changed the title CI: clean install & share pkg build [wip] CI: clean install & share pkg build Dec 9, 2022
@mergify mergify bot removed the has conflicts label Dec 12, 2022
Copy link
Contributor

@carmocca carmocca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I updated checkgroup for you

@carmocca carmocca added the ci Continuous Integration label Dec 12, 2022
@mergify mergify bot added the ready PRs ready to be merged label Dec 12, 2022
@github-actions github-actions bot removed the app Generic label for Lightning App package label Dec 13, 2022
@Borda Borda merged commit 18a4638 into master Dec 13, 2022
@Borda Borda deleted the ci/install-pkg branch December 13, 2022 05:55
run: |
mv pytorch_lightning pl
mv lightning_lite lit_lite
mv lightning_app lit_app
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain why move these directories?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so there is no accidental use of local version but importing the install version

python .actions/assistant.py copy_replace_imports --source_dir="./src" \
--source_import="pytorch_lightning,lightning_lite,lightning_app" \
--target_import="lightning.pytorch,lightning.lite,lightning.app"
rm -rf src/lightning
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why remove the source code?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because I want to test the main package on all subset of our doctests

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Borda added a commit that referenced this pull request Dec 14, 2022
* abstract pkg build
* share ci
* syntax
* Checkgroup
* folders
* whl 1st
* doctest

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>

(cherry picked from commit 18a4638)
Borda added a commit that referenced this pull request Dec 14, 2022
* abstract pkg build
* share ci
* syntax
* Checkgroup
* folders
* whl 1st
* doctest

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>

(cherry picked from commit 18a4638)
lantiga added a commit that referenced this pull request Dec 15, 2022
* update chlog

* CI: Add remote fetch (#16001)

Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local>
(cherry picked from commit 37fe3f6)

* Set the logger explicitly in tests (#15815)

(cherry picked from commit 9ed43c6)

* [App] Fix `AutoScaler` trying to replicate multiple works in a single machine (#15991)

* dont try to replicate new works in the existing machine

* update chglog

* Update comment

* Update src/lightning_app/components/auto_scaler.py

* add test

(cherry picked from commit c1d0156)

* Fix typo in PR titles generated by github-actions bot (#16003)

(cherry picked from commit 2dcebc2)

* Update docker requirement from <=5.0.3,>=5.0.0 to >=5.0.0,<6.0.2 in /requirements (#16007)

Update docker requirement in /requirements

Updates the requirements on [docker](https://github.com/docker/docker-py) to permit the latest version.
- [Release notes](https://github.com/docker/docker-py/releases)
- [Commits](docker/docker-py@5.0.0...6.0.1)

---
updated-dependencies:
- dependency-name: docker
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

(cherry picked from commit 4083b20)

* Update deepdiff requirement from <=5.8.1,>=5.7.0 to >=5.7.0,<6.2.3 in /requirements (#16006)

Update deepdiff requirement in /requirements

Updates the requirements on [deepdiff](https://github.com/seperman/deepdiff) to permit the latest version.
- [Release notes](https://github.com/seperman/deepdiff/releases)
- [Changelog](https://github.com/seperman/deepdiff/blob/master/docs/changelog.rst)
- [Commits](seperman/deepdiff@5.7.0...6.2.2)

---
updated-dependencies:
- dependency-name: deepdiff
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 5e705fa)

* app: update doctest_skip (#15997)

simple

Co-authored-by: hhsecond <sherin@grid.ai>
(cherry picked from commit 4fea6bf)

* CI: clean install & share pkg build (#15986)

* abstract pkg build
* share ci
* syntax
* Checkgroup
* folders
* whl 1st
* doctest

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>

(cherry picked from commit 18a4638)

* Adding hint to the logger's error messages (#16034)

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
Fixes #15143

(cherry picked from commit 7ce3825)

* fix publish

* Introduce `{Work,Flow}.lightningignore` (#15818)

(cherry picked from commit edd2b42)

* [App] Support running on multiple clusters (#16016)

(cherry picked from commit d3a7226)

* [App] Improve lightning connect experience (#16035)

(cherry picked from commit e522a12)

* Cleanup cluster waiting (#16054)

(cherry picked from commit 6458a5a)

* feature(cli): login flow fixes and improvements (#16052)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
(cherry picked from commit ebe7848)

* Add guards to cluster deletion from cli (#16053)

Adds guards to cluster deletion.
- If cluster has running apps -> throw an error
- If cluster has stopped apps -> confirm w/ user that apps and logs will be deleted

(cherry picked from commit 64d0ebb)

* Load app before setting LIGHTNING_DISPATCHED (#16057)

(cherry picked from commit 8d3339a)

* [App] Hot fix: Resolve detection of python debugger (#16068)

Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
(cherry picked from commit eae56ee)

* fix(cloud): detect and ignore venv (#16056)

Co-authored-by: Ethan Harris <ethanwharris@gmail.com>
(cherry picked from commit 3b323c8)

* version 1.8.5

* update chlog

Co-authored-by: thomas chaton <thomas@grid.ai>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
Co-authored-by: Akihiro Nitta <nitta@akihironitta.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Huy Đỗ <56794124+Al3xDo@users.noreply.github.com>
Co-authored-by: Ethan Harris <ethanwharris@gmail.com>
Co-authored-by: Luca Furst <rlfurst@gmail.com>
Co-authored-by: Yurij Mikhalevich <yurij@grid.ai>
Co-authored-by: Luca Antiga <luca.antiga@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci Continuous Integration ready PRs ready to be merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants