THRIFT-5564: add GitHub action for python 2.x and 3.x #2787

jimexist · 2023-04-14T15:26:47Z

closes THRIFT-5564: add GitHub action for python 2.x and 3.x #2596

Did you create an Apache Jira ticket? (Request account here, not required for trivial changes)
If a ticket exists: Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?
Did you squash your changes to a single commit? (not required, but preferred)
Did you do your best to avoid breaking changes? If one was needed, did you label the Jira ticket with "Breaking-Change"?
If your change does not involve any code, include [skip ci] anywhere in the commit message to free up build resources.

update

fishy · 2023-04-14T15:51:32Z

I don't know why this didn't trigger github actions, probably there are something wrong in the yaml makes it failed to parse the config?

jimexist · 2023-04-14T15:59:49Z

I don't know why this didn't trigger github actions, probably there are something wrong in the yaml makes it failed to parse the config?

yes it was an indentation but which is now fixed

fishy · 2023-04-14T16:28:47Z

the 3.x one failed, we probably should also set fail-fast to false so that one failure won't auto cancel other jobs.

jimexist · 2023-04-15T00:17:11Z

the 3.x one failed, we probably should also set fail-fast to false so that one failure won't auto cancel other jobs.

ModuleNotFoundError: No module named 'thrift'

could it be because of some path handling logic not being mirrored in the GitHub workflow code?

jimexist · 2023-04-15T10:24:38Z

@fishy i have no knowledge of python lib tests, can you tell the cause here? https://github.com/apache/thrift/actions/runs/4707202819/jobs/8349103781?pr=2787

jimexist · 2023-04-15T10:32:49Z

@fishy i have no knowledge of python lib tests, can you tell the cause here? https://github.com/apache/thrift/actions/runs/4707202819/jobs/8349103781?pr=2787

turns out i need to backport a list of libraries for python 2

Fokko

I have some small comments, great to have this in the CI as well 👍🏻

Fokko · 2023-04-17T07:17:44Z

.github/workflows/build.yml

+    runs-on: ubuntu-20.04
+    strategy:
+      matrix:
+        python-version: ["2.x", "3.x"]


I would drop Python v2. Python 2 has been sunsetted 1st of January 2020: https://www.python.org/doc/sunset-python-2/

It would be nice to have a matrix to test against different versions of Python 3 as well:

https://github.com/apache/iceberg/blob/9dd00fab1dc160bb531091cbdda8f19ef5b3a32e/.github/workflows/python-ci.yml#L39-L42

This way you can also be more explicit in the supported Python versions (sometimes stuff breaks). Instead of:

thrift/lib/py/setup.py

Line 136 in 1e3d90d

'Programming Language :: Python :: 3',

You can be more explicit in versions:
https://github.com/apache/airflow/blob/8d81963c014398a7ab14505fd8e27e432f1aaf5c/setup.cfg#L39-L42

I would drop Python v2. Python 2 has been sunsetted 1st of January 2020: https://www.python.org/doc/sunset-python-2/

feel free to review #2588

jimexist · 2023-04-17T13:17:19Z

i am not actually an expert in python library code of thrift and not sure why ssl is still failing. anyone wish to take a look?

install openssl remove 3.6 setup daemon try 3.6 fix lib path add backports of py Update build.yml use sudo add install-exec-hook

fishy · 2023-04-17T18:00:34Z

ModuleNotFoundError: No module named 'thrift'

could it be because of some path handling logic not being mirrored in the GitHub workflow code?

possibly. I'm getting the same error locally when trying to run the tests for python3 as well, so I guess some setup steps are missing but I don't know which (I'm not familiar with the python bootstraps either).

jimexist · 2023-04-19T10:26:18Z

ModuleNotFoundError: No module named 'thrift'

could it be because of some path handling logic not being mirrored in the GitHub workflow code?

possibly. I'm getting the same error locally when trying to run the tests for python3 as well, so I guess some setup steps are missing but I don't know which (I'm not familiar with the python bootstraps either).

it's due to a naming convention change which i have fixed in this pull request. but now am hitting ssl errors which i have no idea how to move pass

jimexist · 2023-04-19T10:27:18Z

@fishy how do you think of marking the failing tests skipped and merge this pull request first. this pull request is at least marginally better than the status quote which is without any CI.

fishy

my comments can totally be done in a follow up PR instead. I certainly agree that this is already much better than what we have today.

fishy · 2023-04-19T16:52:00Z

.github/workflows/build.yml

+      #   run: sudo make -C lib/py install-exec-hook
+
+      - name: Run make check for python
+        run: make -C lib/py check


we actually have two sets of tests, one under lib/py (which is the one here) and another under test/py. can you also add a step to run make -C test/py check?

fishy · 2023-04-19T16:54:00Z

.github/workflows/build.yml

  cross-test:
    needs:
      - lib-java-kotlin
      - lib-swift
      - lib-rust
      - lib-go
+      - lib-python


we actually also need to add python (I'm not sure if it's gonna be py3 or python) to line 483 (PRECROSS_LANGS) to actually run the cross tests with python.

fishy · 2023-04-19T16:55:50Z

lib/py/src/server/TNonblockingServer.py

@@ -268,7 +268,7 @@ def prepare(self):
        self.socket.listen()
        for _ in range(self.threads):
            thread = Worker(self.tasks)
-            thread.setDaemon(True)
+            thread.daemon = True


🔕 oh I see setDaemon was deprecated since python 3.10.

jimexist added 5 commits April 14, 2023 23:17

update .github workflow to consolidate config

0e90ee5

add lib python

4d34962

Update build.yml

2b77fe6

update

rust => rs

9727f8e

Update build.yml

4f15c7a

update

Merge branch 'apache:master' into setup-github-action-py2

7735b28

Jens-G added the ci label Apr 15, 2023

Fokko reviewed Apr 17, 2023

View reviewed changes

update

02b09b0

install openssl remove 3.6 setup daemon try 3.6 fix lib path add backports of py Update build.yml use sudo add install-exec-hook

jimexist force-pushed the setup-github-action-py2 branch from 6a6c0ce to 02b09b0 Compare April 17, 2023 13:18

fishy mentioned this pull request Apr 17, 2023

THRIFT-5693 - fix bug in serialization of enum default values #2783

Open

5 tasks

add ssl test skip

1cdcefb

fishy approved these changes Apr 19, 2023

View reviewed changes

jimexist merged commit 6f33900 into apache:master Apr 19, 2023

jimexist deleted the setup-github-action-py2 branch April 19, 2023 23:39

jimexist mentioned this pull request Apr 20, 2023

[CI] update github workflow for python and include test folder #2789

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

THRIFT-5564: add GitHub action for python 2.x and 3.x #2787

THRIFT-5564: add GitHub action for python 2.x and 3.x #2787

jimexist commented Apr 14, 2023 •

edited

Loading

fishy commented Apr 14, 2023

jimexist commented Apr 14, 2023

fishy commented Apr 14, 2023

jimexist commented Apr 15, 2023

jimexist commented Apr 15, 2023

jimexist commented Apr 15, 2023

Fokko left a comment

Fokko Apr 17, 2023

Fokko Apr 17, 2023

jimexist Apr 17, 2023

jimexist commented Apr 17, 2023

fishy commented Apr 17, 2023

jimexist commented Apr 19, 2023

jimexist commented Apr 19, 2023 •

edited

Loading

fishy left a comment

fishy Apr 19, 2023

fishy Apr 19, 2023

fishy Apr 19, 2023

THRIFT-5564: add GitHub action for python 2.x and 3.x #2787

THRIFT-5564: add GitHub action for python 2.x and 3.x #2787

Conversation

jimexist commented Apr 14, 2023 • edited Loading

fishy commented Apr 14, 2023

jimexist commented Apr 14, 2023

fishy commented Apr 14, 2023

jimexist commented Apr 15, 2023

jimexist commented Apr 15, 2023

jimexist commented Apr 15, 2023

Fokko left a comment

Choose a reason for hiding this comment

Fokko Apr 17, 2023

Choose a reason for hiding this comment

Fokko Apr 17, 2023

Choose a reason for hiding this comment

jimexist Apr 17, 2023

Choose a reason for hiding this comment

jimexist commented Apr 17, 2023

fishy commented Apr 17, 2023

jimexist commented Apr 19, 2023

jimexist commented Apr 19, 2023 • edited Loading

fishy left a comment

Choose a reason for hiding this comment

fishy Apr 19, 2023

Choose a reason for hiding this comment

fishy Apr 19, 2023

Choose a reason for hiding this comment

fishy Apr 19, 2023

Choose a reason for hiding this comment

jimexist commented Apr 14, 2023 •

edited

Loading

jimexist commented Apr 19, 2023 •

edited

Loading