Migrate to GitHub acitions #1342

jernejfrank · 2025-06-14T23:13:38Z

Addressing #1313 and uses uv.

I don't have access to CircleCI to see the settings there (I assume the config with jobs and worklof is only part of it, since there's no mention of OS inside), would be good to let me know if anything else needs to be configured for the github actions workflow.

Changes

replaced circleci config with github actions
minor linting and bugfixes

How I tested this

Need to invoke CI from main repo

Notes

Fail-fast is disabled for now, so that I can see all of the tests in the matrix failing
Not sure if we need to test on all platforms
Adds the option to run the tests manually through the actions tab
some test (mostly plugins) were running only on specific python versions and run now on all supported python versions of the plugin package:
- Ray, Dask, PySpark 3.9-3.12
- pre-commit 3.8-3.12 (sometimes important to spot things like needing from __future__ import annotations)
- pandas, polars, narwhals 3.8-3.12
- Vaex 3.8-3.10

Checklist

PR has an informative and human-readable title (this will be pulled into the release notes)
Changes are limited to a single goal (no scope creep)
Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
Any change in functionality is tested
New functions are documented (with a description, list of inputs, and expected output)
Placeholder code is flagged / future TODOs are captured in comments
Project documentation has been updated if adding/changing functionality.

skrawcz · 2025-06-15T03:38:46Z

I don't have access to CircleCI to see the settings there (I assume the config with jobs and worklof is only part of it, since there's no mention of OS inside), would be good to let me know if anything else needs to be configured for the github actions workflow.

@jernejfrank you can see everything in the .circleci yaml and what it references. It's similar to github workflows, but we have more flexibility on when a workflow is triggered with the github way. So shouldn't be hard to replicate.

skrawcz · 2025-06-15T03:40:39Z

pyproject.toml

 ]
 dependencies = [
-    "numpy",
+    "numpy<2",


this isn't good since 2.0 has been out for a while.

Forgot to remove when debugging some other dependencies issues. Removed now.

skrawcz · 2025-06-15T03:40:44Z

pyproject.toml

  "dlt",
  # furo -- install from main for now until the next release is out:
-  "furo @ git+https://github.com/pradyunsg/furo@main",
+  "furo",


skrawcz · 2025-06-15T03:43:49Z

The SDK tests aren't passing because spark changed things. I think we just need to update the tests...

jernejfrank · 2025-06-15T07:25:20Z

I don't have access to CircleCI to see the settings there (I assume the config with jobs and worklof is only part of it, since there's no mention of OS inside), would be good to let me know if anything else needs to be configured for the github actions workflow.

@jernejfrank you can see everything in the .circleci yaml and what it references. It's similar to github workflows, but we have more flexibility on when a workflow is triggered with the github way. So shouldn't be hard to replicate.

@skrawcz Ah ok, then all is migrated! I thought there might be something more since the .circleci yaml doesn't mention anything about OS to test on, etc..

Also added this to the notes of the pr:

some test (mostly plugins) were running only on specific python versions and run now on all supported python versions of the plugin package:
- Ray, Dask, PySpark 3.9-3.12
- pre-commit 3.8-3.12 (sometimes important to spot things like needing from __future__ import annotations)
- pandas, polars, narwhals 3.8-3.12
- Vaex 3.8-3.10

Will be fixing broken tests, but may take a while!

jernejfrank · 2025-06-17T21:57:18Z

Right, so SDK was passing but now the CI dissapeared. Any thoughts @skrawcz?

I am also thinking of limiting it to Linux (for now) since I saw that windows has some issues with package installation when using uv since it does a better job at finding problems than pip.

skrawcz · 2025-06-18T01:07:03Z

Right, so SDK was passing but now the CI dissapeared. Any thoughts @skrawcz?

Can you rebase against main? (you might want to squash your commits to make that easier to do if it makes sense to squash some of them)

I am also thinking of limiting it to Linux (for now) since I saw that windows has some issues with package installation when using uv since it does a better job at finding problems than pip.

That's fine I think.

Limit to linux os for now

Fix Polars hist lower bound tests Fix test by changing sql.DataFrame to sql.classic.DataFrame Fix sanitize error to work on CI

jernejfrank · 2025-06-19T22:00:09Z

@skrawcz ok, got the tests sorted out and did a rebase. Let me know if you're happy with squash merge or need me to organize the commits :)

skrawcz · 2025-06-20T02:52:04Z

.github/workflows/hamilton-main.yml

+    name: "Unit Tests"
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false # will change this to true at the end, but want to see tests failing on all use cases


need to flip this?

actually I think we want this to be false always.

skrawcz

Let's squash merge this into a single commit. This will make things simpler to rebase that way ; I don't think we need to keep the individual commits, right?

I usually craft a commit message, and then leave the prior squashed commits below it!

jernejfrank requested a review from skrawcz June 14, 2025 23:13

skrawcz reviewed Jun 15, 2025

View reviewed changes

jernejfrank force-pushed the migrate_to_github_acitions branch from 1efc74e to ffccf31 Compare June 17, 2025 19:55

jernejfrank added 4 commits June 18, 2025 07:48

Migrate CircleCI to actions

19b020b

Limit to linux os for now

Fix linting issues

d58351a

Fix tests and minor bug issues

0a58b9c

Fix SDK

a21f800

Fix Polars hist lower bound tests Fix test by changing sql.DataFrame to sql.classic.DataFrame Fix sanitize error to work on CI

jernejfrank force-pushed the migrate_to_github_acitions branch from ffccf31 to a21f800 Compare June 18, 2025 06:51

jernejfrank added 7 commits June 18, 2025 23:25

Fix pandas/polars plugin tests

862ba87

Pin dask minimal dependency that resolve bug

10fc45d

Fix ray venv bug

e27433c

Add grpcio dependency for spark

ab72c56

Try force reinstall grcpio stuff

c16ed7d

Disable pyspark ANSI

1a467b0

Try to fix SparkInputValidator test

605497d

skrawcz reviewed Jun 20, 2025

View reviewed changes

skrawcz approved these changes Jun 20, 2025

View reviewed changes

Add reason for fail-safe false

6383202

jernejfrank merged commit 3c32aca into apache:main Jun 20, 2025
8 checks passed

Migrate to GitHub acitions #1342

Migrate to GitHub acitions #1342

Uh oh!

Conversation

jernejfrank commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

How I tested this

Notes

Checklist

Uh oh!

skrawcz commented Jun 15, 2025

Uh oh!

skrawcz Jun 15, 2025

Choose a reason for hiding this comment

Uh oh!

jernejfrank Jun 15, 2025

Choose a reason for hiding this comment

Uh oh!

skrawcz Jun 15, 2025

Choose a reason for hiding this comment

Uh oh!

skrawcz commented Jun 15, 2025

Uh oh!

jernejfrank commented Jun 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jernejfrank commented Jun 17, 2025

Uh oh!

skrawcz commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jernejfrank commented Jun 19, 2025

Uh oh!

skrawcz Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

skrawcz Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

skrawcz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jernejfrank commented Jun 14, 2025 •

edited

Loading

jernejfrank commented Jun 15, 2025 •

edited

Loading

skrawcz commented Jun 18, 2025 •

edited

Loading