Add Gitpod configuration #1232

antonymilne · 2022-02-10T10:15:30Z

This PR add a Gitpod configuration file that makes it much easier to develop on this repo using Gitpod. Updates to contributing guide to follow in a separate PR.

What is Gitpod?

Quoting the Gitpod website:

Every day developers waste millions of hours switching contexts between projects and branches, setting up developer environments, or simply waiting for builds to complete. This friction is not only time consuming but is a serious barrier for contributors. Reading through a long list of setup instructions and messing up your own laptop is just too much effort.

Here’s how Gitpod works: You are on GitHub and want to start working on a project. Instead of going through the manual setups, etc. you simply click a button to start a ready-to-code dev environment in your browser. You can now code, review, or just try out the project. Once you are done you simply close the tab.

Basically it gives you a way of spinning up an ephemeral dev environment on the cloud. I have found this to be incredibly useful and wish to share the joy with everyone.

Why would I want to use Gitpod on Kedro?

There are many use cases, but the most obvious one is that it massively streamlines the process for open source contributors. Following our current development setup guidelines requires a certain level of expertise and is time-consuming (make install-test-requirements takes 10+ minutes for me locally).

With Gitpod, a contributor only needs to click one button to launch in a couple of minutes an environment that includes:

all Python dependencies installed 🐍
pre-commit hooks installed 🪝
the checked out branch of Kedro installed in editable/develop mode 🧑‍💻
a test project to run against 🧪
commits automatically DCO signed-off 👮

I'm a maintainer who already has a local development environment setup. Should I still be interested in Gitpod?

Absolutely. I still do most of my development locally in PyCharm but find Gitpod very valuable for doing small tasks while minimising disruption to my local workflow. I use it pretty much every day for the following:

Reviewing PRs if I need more than just the GitHub interface ✅ Gitpod opens up in the context of the PR, so you can try it out without needing to checkout the branch locally
Resolving conflicts on merge-main-to-develop ⚠️
Making small bug fixes 🐛
Documentation updates 📝 Unlike changes you make on GitHub directly, you can build the docs to see what it looks like

More generally, Gitpod is very useful because it provides a clean environment and hence a good source of truth:

A test is failing on CI but not your local environment. Which is correct? Just spin up a Gitpod instance and try it out there.
Reproduce bug reports.
Checking Python dependencies. No need to worry about messing up your own local environment or switching to a different environment to work on a different version of Python.

It's also worth noting that the Gitpod environment is much faster than my local one (e.g. make e2e-test runs in about half the time).

✨ Actually it's also possible to use the Gitpod environment from your own local IDE! This means you get all the advantages of the ephemeral dev environment but can still use the editor you like with the keyboard shortcuts etc. that you're used to. I've tried this with PyCharm and it was a bit clunky (they're still working on it apparently); with VS Code it was super slick and I actually couldn't tell I was using a remote container at all. ✨

How much does it cost?

Nothing! It's completely free to us and to the users. On the default free plan you get 50 hours per month. If you're a professional Kedro maintainer then you can apply to get professional open source membership which gives you an unlimited number of hours per month.

Tips

You can launch Gitpod anywhere in our repo by prefixing the GitHub URL with gitpod.io/#. The browser bookmarklet or browser extension make it even easier
If you're using the in-browser editor then alter your settings to enable copy & paste and allow popups
To preview documentation after make build-docs, the easiest way is to run python -m http.server -d docs/build/html. This will open the HTML up in your browser
To change Python version, use pyenv. This is quite slow because it builds Python from source. If it's something people end up doing a lot then we can make it much slicker by pre-installing the latest 3.7.x, 3.8.x, etc.

Checklist

Read the contributing guidelines
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the RELEASE.md file
Added tests to cover my changes

Signed-off-by: Antony Milne <antony.milne@quantumblack.com>

antonymilne · 2022-02-10T10:22:47Z

test_requirements.txt

@@ -33,7 +33,7 @@ Pillow~=8.0; python_version == '3.6'
 Pillow~=9.0; python_version > '3.6'
 plotly>=4.8.0, <6.0
 pre-commit~=1.17
-psutil==5.6.7
+psutil==5.9.0


Not sure why I was getting an with psutils==5.6.7 - it worked fine for months until this morning 🤔 To be checked whether we should change this or not.

Fixed by reverting to a legacy Dockerfile.

Signed-off-by: Antony Milne <antony.milne@quantumblack.com>

antonymilne · 2022-02-10T10:39:40Z

.gitpod.yml

+      pip install /workspace/kedro --no-deps
+      yes project | kedro new -s pandas-iris
+    command: |
+      pip install /workspace/kedro --no-deps


TODO: this should be -e.

Fixed by reverting to a legacy Dockerfile.

datajoely · 2022-02-10T11:57:39Z

This is really cool - can gitpod allow us to spin up a Windows env?

Signed-off-by: Antony Milne <antony.milne@quantumblack.com>

antonymilne · 2022-02-11T10:00:18Z

.gitpod.yml

@@ -0,0 +1,45 @@
+image: gitpod/workspace-full:legacy-dazzle-v1


This is why things didn't work properly yesterday! Gitpod changed their system for building Docker images, and the latest gitpod/workspace-full (used by default) broke some Python stuff. Until it's fixed I've set this to use the old Docker image.

antonymilne · 2022-02-11T10:31:17Z

This is really cool - can gitpod allow us to spin up a Windows env?

Discussed this offline but putting it here so everyone can see...

I was wondering this also as it might be a nice way to debug stuff on Windows. I think the basic answer is no. This is what they say, which I don't really understand tbh...

you can't use a windows dockerimage as the workspace image but you might be able to run windows-servercore with qemu from the ISO image

You can only have one gitpod setup per repo, so if we could use Gitpod to try stuff out on Windows then we would need to put the setup on a separately forked repo. Otherwise every time you launch Gitpod on the main kedro repo it would use the Windows setup.

antonymilne · 2022-02-11T10:55:05Z

I've requested reviews from lots of people just for visibility if you're interested in this. No need to look if you're not interested though!

merelcht

This is awesome!! 🚀 Thank you so much for investing time in learning Gitpod and setting it up for Kedro! 🏆🌟 I can't wait to start using it 🥳

lorenabalan

This is an amazing effort and an even more brilliant write-up!!! 🏆 🎖️ 💯 🥇 🎊
Could we write down some of this knowledge or suggestions into our contributing docs? I'm thinking especially of the stuff you've listed under Tips, this could be super useful! Let's also make some noise / show some love on Discord. 🙌 🎉

limdauto

Amazing!!!

SajidAlamQB

Amazing work, can't wait to use it! 💥

antonymilne · 2022-02-11T15:52:25Z

Thanks for the nice comments all! @lorenabalan I'm definitely going to put this in the contributing guide including the tips, and then let's make some noise about it on slack and discord. I just want to try it out a little more to be sure it's all working, since we only just got the app added to the repo.

Signed-off-by: Laurens Vijnck <laurens_vijnck@mckinsey.com>

antonymilne and others added 6 commits February 1, 2022 10:57

Create .gitpod.yml

2ad716c

Update .gitpod.yml

9f8c4a1

Document .gitpod.yml

c17c530

Signed-off-by: Antony Milne <antony.milne@quantumblack.com>

Merge branch 'kedro-org:main' into main

626bc1c

Merge branch 'kedro-org:main' into main

8480d64

Merge branch 'kedro-org:main' into main

075c023

antonymilne self-assigned this Feb 10, 2022

antonymilne requested a review from idanov as a code owner February 10, 2022 10:15

Bump psutils

b56fc20

Signed-off-by: Antony Milne <antony.milne@quantumblack.com>

antonymilne commented Feb 10, 2022

View reviewed changes

antonymilne marked this pull request as draft February 10, 2022 10:36

Remove editable mode

83ad8e2

Signed-off-by: Antony Milne <antony.milne@quantumblack.com>

antonymilne commented Feb 10, 2022

View reviewed changes

antonymilne mentioned this pull request Feb 10, 2022

Add Gitpod configuration kedro-org/kedro-viz#735

Merged

11 tasks

antonymilne and others added 3 commits February 10, 2022 23:10

Add no-use-pep517

e020c47

Signed-off-by: Antony Milne <antony.milne@quantumblack.com>

Revert to old Dockerfile

72dd61d

Revert to old Dockerfile

638f98b

Signed-off-by: Antony Milne <antony.milne@quantumblack.com>

antonymilne commented Feb 11, 2022

View reviewed changes

antonymilne added 2 commits February 11, 2022 10:31

Merge branch 'main' into main

53990d2

Merge branch 'main' into main

b4a663d

antonymilne marked this pull request as ready for review February 11, 2022 10:53

antonymilne requested review from limdauto, lorenabalan, merelcht, SajidAlamQB and yetudada February 11, 2022 10:54

antonymilne requested a review from datajoely February 11, 2022 10:55

merelcht approved these changes Feb 11, 2022

View reviewed changes

lorenabalan approved these changes Feb 11, 2022

View reviewed changes

limdauto approved these changes Feb 11, 2022

View reviewed changes

SajidAlamQB approved these changes Feb 11, 2022

View reviewed changes

Merge branch 'main' into main

94fd63e

antonymilne merged commit 33b1c88 into kedro-org:main Feb 11, 2022

antonymilne mentioned this pull request Feb 15, 2022

<Title> #1253

Closed

lvijnck pushed a commit to lvijnck/kedro that referenced this pull request Apr 7, 2022

Add Gitpod configuration (kedro-org#1232)

ab1b47a

Signed-off-by: Laurens Vijnck <laurens_vijnck@mckinsey.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Gitpod configuration #1232

Add Gitpod configuration #1232

antonymilne commented Feb 10, 2022 •

edited by gitpod-io bot

Loading

antonymilne Feb 10, 2022

antonymilne Feb 11, 2022

antonymilne Feb 10, 2022

antonymilne Feb 11, 2022

datajoely commented Feb 10, 2022

antonymilne Feb 11, 2022

antonymilne commented Feb 11, 2022

antonymilne commented Feb 11, 2022

merelcht left a comment

lorenabalan left a comment

limdauto left a comment

SajidAlamQB left a comment

antonymilne commented Feb 11, 2022

		@@ -0,0 +1,45 @@
		image: gitpod/workspace-full:legacy-dazzle-v1

Add Gitpod configuration #1232

Add Gitpod configuration #1232

Conversation

antonymilne commented Feb 10, 2022 • edited by gitpod-io bot Loading

What is Gitpod?

Why would I want to use Gitpod on Kedro?

I'm a maintainer who already has a local development environment setup. Should I still be interested in Gitpod?

How much does it cost?

Tips

Checklist

antonymilne Feb 10, 2022

Choose a reason for hiding this comment

antonymilne Feb 11, 2022

Choose a reason for hiding this comment

antonymilne Feb 10, 2022

Choose a reason for hiding this comment

antonymilne Feb 11, 2022

Choose a reason for hiding this comment

datajoely commented Feb 10, 2022

antonymilne Feb 11, 2022

Choose a reason for hiding this comment

antonymilne commented Feb 11, 2022

antonymilne commented Feb 11, 2022

merelcht left a comment

Choose a reason for hiding this comment

lorenabalan left a comment

Choose a reason for hiding this comment

limdauto left a comment

Choose a reason for hiding this comment

SajidAlamQB left a comment

Choose a reason for hiding this comment

antonymilne commented Feb 11, 2022

antonymilne commented Feb 10, 2022 •

edited by gitpod-io bot

Loading