Add Windows CI to kedro-starters #89
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
added 9 commits
May 18, 2022 16:36
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
AhdraMeraliQB
commented
May 18, 2022
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
* Try cloning env in before scenario Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Try again Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Try find line Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Try find line 2 Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Try find line 3 Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Try without venv.main step Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Try activating env at every windows build step Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Activate env before running tests/linting Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Clean up environment Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Try adding hadoop setup step Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Try adding hadoop setup step Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Clean up Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Make formatting consistent Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> Co-authored-by: Merel Theisen <merel.theisen@quantumblack.com>
|
The test for running pyspark-iris fails for the windows builds, the error messages have been captured and are appended below. However, perhaps this should be addressed in a separate issue, as I would say it falls out of scope for the current one. -- |
merelcht
reviewed
May 25, 2022
There was a problem hiding this comment.
Great work @AhdraMeraliQB ! I'm happy with skipping the pyspark-iris test on Windows. Ideally we would add in a bit of logic so the builds pass and you don't need an admin to merge. Basically a check that says "if its the pyspark-iris test and running windows, skip the test". See: https://stackoverflow.com/questions/36482419/how-do-i-skip-a-test-in-the-behave-python-bdd-framework and https://github.com/kedro-org/kedro/blob/main/features/build_docs.feature#L3 + https://github.com/kedro-org/kedro/blob/main/features/environment.py#L41 for how you can add tags to behave test to skip.
added 2 commits
May 25, 2022 17:07
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
noklam
reviewed
May 31, 2022
noklam
reviewed
May 31, 2022
.circleci/config.yml
Outdated
| name: Install Python dependencies | ||
| command: | | ||
| pip install git+https://github.com/kedro-org/kedro@main | ||
| cd package && pip install -r requirements.txt -U |
There was a problem hiding this comment.
Where is this package folder coming from?
There was a problem hiding this comment.
Nice catch - looks like we're installing kedro and it's requirements, and then installing them again under the kedro-starters requirements; I'll remove the redundant step.
SajidAlamQB
reviewed
May 31, 2022
There was a problem hiding this comment.
We might want JDK set up, we could try the win JDK setup we have on the Kedro repo. Having this installed may fix the pyspark-iris issues since it's reliant on Java.
noklam
reviewed
May 31, 2022
features/environment.py
Outdated
Comment on lines
77
to
80
| def after_scenario(context, scenario): | ||
| rmtree(str(context.temp_dir)) | ||
| rmtree(str(context.venv_dir)) | ||
|
|
||
|
|
||
| def rmtree(top): | ||
| if os.name != "posix": | ||
| for root, _, files in os.walk(top, topdown=False): | ||
| for name in files: | ||
| os.chmod(os.path.join(root, name), stat.S_IWUSR) | ||
| shutil.rmtree(top) | ||
| for path in _PATHS_TO_REMOVE: | ||
| # ignore errors when attempting to remove already removed directories | ||
| shutil.rmtree(path, ignore_errors=True) |
There was a problem hiding this comment.
Is this just refactoring or necessary to make it work? context and scenario are now redundant arguments. Does it still belong to after_scenario or it should be after_all?
There was a problem hiding this comment.
With how behave is set up it always passes the context through, but scenario was indeed redundant and has been removed. Thanks for the catch!
noklam
reviewed
May 31, 2022
There was a problem hiding this comment.
Thanks for making this work! Leave a couples of small comments.
I also notice on the CircleCI there is a note, on kedro main repo we don't have this and we can see all the test results within CircleCI UI.
Your output is too large to display in the browser.
Only the last 400000 characters are displayed
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
@SajidAlamQB that's a good shout - will be sure to address in the pyspark-iris issue 👍 |
@noklam I think that might be built into CircleCI for larger outputs; I'm unsure that anything was changed manually to trigger this 🤔 |
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Co-authored-by: Nok <nok.lam.chan@quantumblack.com>
merelcht
approved these changes
Jun 7, 2022
There was a problem hiding this comment.
Awesome work @AhdraMeraliQB ! ⭐ 🎉 Great to finally have windows builds for our starters.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Ahdra Merali ahdra.merali@quantumblack.com
Motivation and Context
Kedro-starters was lacking Windows CI. This PR rectifies this by adding Windows builds to the e2e testing. The setup has also been changed to make use of conda environments. The tests themselves have not been changed.
Checklist