Code quality and release period

[benz0li]

During the past year, roughly every second release had major bugs rendering some features of code-server unusable.

The current release period is closer to two month rather than one month.

= 3-4 usable code-server releases per year.

---

I don't depend on the latest version of code-server. But if the currently usable code-server version should get half a year behind of VS Code at some point, things will get complicated again with the Jupyter and Python extension.

@jsjoeio What are your plans to improve the code quality and to comply with the 1-month release cycle?

[Newtoniano]

I guess the recent announcement of Vscode server might render this project obsolete, or restrict its use cases once that gets out of preview and more features are added. The fact it's gonna have full support for all extensions including proprietary ones, PWA and likely more up to date releases are huge advantages.

[jsjoeio]

Thanks @benz0li for bringing this up and the feedback! Code quality has definitely not been where we want it.

In terms of quality, we've been focused on it for the last year but still not where we want to be. We've done the following:

• increased code coverage via unit tests 47% -> 72%. our goal is 80%
• added end-to-end tests with Playwright
• added shell script tests
• added integration tests

So we're trying, but clearly things are slipping through still. For instance, the terminal bug happened because our tests don't run against a proxy/non-root. We are working on adding a setup to prevent that from happening again. For the version bug, we cached VS Code when we released 4.5.0.

to comply with the 1-month release cycle

If I may nitpick this a bit, I wouldn't say "comply with 1-month release cycle." We want a 1-month cycle and we're shooting for it, but with a small team size, it wouldn't be perfect. I would say we will do what we can to make that happen but it may not occassionally.

Lastly, I want to say thanks again for bringing this up! This is an amazing project and community and we want you, and others, to continue bringing these things up so that we can all figure out how to fix them together.

---

We're also open to other ideas! We would love any support for adding more tests to increasing the code quality or other suggestions the community has.

[benz0li]

to comply with the 1-month release cycle

If I may nitpick this a bit, I wouldn't say "comply with 1-month release cycle." We want a 1-month cycle and we're shooting for it, but with a small team size, it wouldn't be perfect. I would say we will do what we can to make that happen but it may not occassionally.

@jsjoeio You are doing a tremendous job. Thank you and your team for all your work.

We're also open to other ideas! We would love any support for adding more tests to increasing the code quality or other suggestions the community has.

I could do some manual smoke tests before any release – just ping me to do so. If you do not get an answer within 48 hours (depending on my availability) simply go ahead.
ℹ️ My focus is on the applicability in the field of Data Science, i.e. proxied JupyterLab integration, working terminal as well as compatibility with extensions ms-python.python, ms-toolsai.jupyter, Ikuyadeu.r and julialang.language-julia.

[jsjoeio]

We appreciate that but we can always do better! <3

I could do some manual smoke tests before any release – just ping me to do so. If you do not get an answer within 48 hours (depending on my availability) simply go ahead.

Appreciate you offering that! I'm wondering though how we might automate it. I guess Step 1 would be simply adding an e2e test to make sure you can install the Jupyter and Python extensions. We'd have to pin the versions though so then we could include .vsix files in source code and not rely on a connection for the tests to pass.

@code-asher any thoughts on these e2e tests and making sure code-server works for data scientists?

[benz0li]

I guess Step 1 would be simply adding an e2e test to make sure you can install the Jupyter and Python extensions. We'd have to pin the versions though so then we could include .vsix files in source code and not rely on a connection for the tests to pass.

I don't see how installing extensions should fail. A .vsix file is merely a .zip file containing some additional files next to the extension [folder] itself.
ℹ️ Using the CLI (code-server --install-extension) always installs the latest (and possibly incompatible) version of an extension.
👉 See also #5031 and microsoft/vscode#153201.

@code-asher any thoughts on these e2e tests and making sure code-server works for data scientists?

IMHO That's beyond the scope of a stable release of code-server. I am happy to provide pre-releases on demand at https://vscode-r.jupyter.b-data.ch. (To look into #5335 for example)

FYI @fritterhoff

[jsjoeio]

I don't see how installing extensions should fail. A .vsix file is merely a .zip file containing some additional files next to the extension [folder] itself.

Ah sorry, my message was confusing. I meant if you do code-server --install-extension <ext-name> it will require an internet connection vs. code-server --install-extension <vsix file> doesn't so less likely for tests to flake.

IMHO That's beyond the scope of a stable release of code-server.

Maybe! But if we have community members and potential customers relying on Jupyter to work, we need to make sure it doesn't break between releases.

[code-asher]

I think we have a few problems.

1. Not enough manual testing

This testing is tedious and I have done a very poor job smoke testing releases ever since my primary focus was moved away from code-server so I am largely to blame here.

Updating Code can seem benign but doing so often breaks code-server unexpectedly and due to the architecture there is no guarantee that something which works when running via yarn watch (which is what we use when updating Code or making changes) works with a full build for production.

That said, I think if we test all the flows touched by our patches behind a reverse proxy before each release we should be in a good position. After each Code update we should do the same (with a full build, not yarn watch as that has caused us to miss multiple significant regressions).

Flows not touched by our patches are not as critical I think.

We might benefit from our own version of https://vscode-r.jupyter.b-data.ch/ that gets deployed on each PR and gets linked in the PR comments to lower the friction of testing a PR or a release.

Maybe release candidates (as pre-releases) would be a good call. We could require a minimum of a week before promoting the candidate to full release (assuming no release blockers) or something like that.

2. Not enough automated testing

If we find ourselves with time to write tests we might want to focus on e2e tests since I think for code-server they are more valuable than any other test even though they do not contribute to code coverage.

E2e tests should focus on covering patched flows.

3. Not marking regressions as release blockers.

Pretty straightforward. If an issue reports a regression the next release should not continue without resolving the issue. No exceptions IMO. We could use labels and/or put it on the milestone.

4. Keeping up to date with Code remains difficult.

I think we struggle to allocate time for updates and updating always takes longer than we think. The dream is to just update the submodule and away we go but it is almost never that easy.

We have talked about including code-server in planning sessions but we have not followed through.

Long-term we should think about upstreaming as much as possible and see if we can eventually deprecate code-server in favor of upstream's code-server. I think we should have an easier go of it now that they seem to be documenting ways to self-host (although it is still in private preview). Maybe we can start by trying upstream's code-server in Coder OSS.

tl;dr
Most important takeaways:

1. We need to test all patched flows on each release
2. Regressions must block releases

Secondary takeaways:

1. Test all patched flows with a full build after updates, not just with yarn watch
2. Prioritize e2e tests over other types
3. Automatic PR deploys would be nice
4. Try release candidates
5. Try to deprecate in favor of upstream
6. Try out upstream code-server in Coder OSS or elsewhere
7. Include code-server in planning sessions