Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce CI runner count #1777

Merged
merged 7 commits into from
Sep 4, 2023
Merged

Reduce CI runner count #1777

merged 7 commits into from
Sep 4, 2023

Conversation

strickvl
Copy link
Contributor

@strickvl strickvl commented Sep 1, 2023

Implementing a refactoring of our runner matrix to reduce total numbers of runners needed for our CI tests.

  • Combined setup with lint+unit tests
  • Combined template tests with integration tests

Pre-requisites

Please ensure you have done the following:

  • I have read the CONTRIBUTING.md document.
  • If my change requires a change to docs, I have updated the documentation accordingly.
  • If I have added an integration, I have updated the integrations table and the corresponding website section.
  • I have added tests to cover my changes.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Other (add details above)

@strickvl strickvl added enhancement New feature or request internal To filter out internal PRs and issues tests Test suite additions or updates labels Sep 1, 2023
@strickvl strickvl changed the title WIP: Reduce CI runner count Reduce CI runner count Sep 1, 2023
Copy link
Contributor

@fa9r fa9r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Contributor

@AlexejPenner AlexejPenner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🦭

Copy link
Contributor

@avishniakov avishniakov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, nice step forward! Left a few comment and we need to find out why CI failed (looks like there are unseen errors with setup happening)

.github/workflows/integration-test.yml Outdated Show resolved Hide resolved
.github/workflows/integration-test.yml Outdated Show resolved Hide resolved
Copy link
Contributor

@avishniakov avishniakov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to merge 👍🏼

@avishniakov
Copy link
Contributor

@strickvl let's merge it before #1766 - this one is more critical and I'll resolve conflicts there.

@strickvl
Copy link
Contributor Author

strickvl commented Sep 4, 2023

@avishniakov agreed. the failing test is lightgbm integration test, but I ran it locally and works fine. WDYT? the failure message doesn't really help debug what's up there. And the test works everywhere else.

@avishniakov
Copy link
Contributor

@avishniakov agreed. the failing test is lightgbm integration test, but I ran it locally and works fine. WDYT? the failure message doesn't really help debug what's up there. And the test works everywhere else.

I would merge it - don't see this failure anyhow related to your work in this PR.

@strickvl strickvl merged commit 07c0313 into develop Sep 4, 2023
28 of 29 checks passed
@strickvl strickvl deleted the feature/reduce-ci-runner-count branch September 4, 2023 14:13
avishniakov added a commit that referenced this pull request Sep 4, 2023
* Add `bandit` to CI for security linting (#1775)

* ignore bandit false positives

* add timeout for requests call

* ignore bandit false positive

* ignore bandit false positives

* ignore bandit false positives

* update as per bandit suggestions

* add bandit as a dependency

* add security checks to lint-unit-test workflow

* add safety to dev dependencies + update version

* loosen safety dependency pin

* remove safety dependency

* Add `mlstacks` compatibility check to CI (#1767)

* add compatibility check to CI

* add compatibility check to earlier in CI

* remove dependency on integration tests from CI

* add compatibility check to earlier in CI

* remove compatibility check from integration tests

---------

Co-authored-by: Safoine El Khabich <34200873+safoinme@users.noreply.github.com>

* extend `StepContext` visibility to materializers (#1769)

* extend StepContext visibility to materializers

* skip output processing on failed step

* PR comments

* Revert GH changes to fix colima bug in macos gh (#1779)

* revert the changes done on gh in macos due to failing bug in colima

* revert the changes done on gh in macos due to failing bug in colima

* reinstall qemu

* Reduce CI runner count (#1777)

* combine setup with unit tests

* combine integration with template tests

* further reduce runner count

* remove dynamic uses

---------

Co-authored-by: Safoine El Khabich <34200873+safoinme@users.noreply.github.com>

* Add auto-population of E2E example from template (#1766)

* add checked-out template

* add setup to action

* auto-update e2e

* add OPT_IN_OUT_EMAIL

* pass email

* let e2e auto populate

* debug

* fix mkdir

* debug

* debug

* update commit command

* use ref_name

* update ref

* add ref to checkout

* Auto-update of template

* update template

* Auto-update of template

* restore full CI

* fix branch changes check

* debug

* restore full CI

---------

Co-authored-by: GitHub Actions <actions@github.com>

* fix step name in ci

---------

Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com>
Co-authored-by: Safoine El Khabich <34200873+safoinme@users.noreply.github.com>
Co-authored-by: GitHub Actions <actions@github.com>
@fa9r fa9r mentioned this pull request Sep 11, 2023
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request internal To filter out internal PRs and issues tests Test suite additions or updates
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants