Most relevant area for this enhancement?
PDE
Feature to enhance
Maintainability
Enhancement details
I’ve been curious about using another open source code editor as the core user interface for the Processing Development Environment (PDE). Some high level goals for such an implementation would be:
- Ease our maintenance burdens by leveraging a larger open source project
- Use a newer front-end UI toolkit to access a larger pool of potential contributors
- And most importantly, continue to provide a UI that is friendly to beginners and artists who don’t want to write their own build scripts, as the PDE has done for nearly 25 years!
The part of the VSCode source released under the MIT License is the core of several other programming environments, including Cursor and Theia. There are some important features we can leverage from the VSCode codebase: developer extensions, syntax highlighting for many different programming languages, internationalization, telemetry, and AI code support. VSCode’s user interface is built with Typescript and Electron, which are both widely used in front-end web development. VSCode is also a project maintained by a well-resourced team of engineers and designers at Microsoft, who can put a lot more time into maintenance and security than our small (but mighty) team from Processing Foundation.
I used Copilot/Claude to build a PDE-like UI from a fork of the VSCode project for evaluative purposes, based on a suggestion from Charlotte. This proof of concept incorporates some of the PDE’s most important beginner-friendly features in the VSCode UI, such as the welcome screen and a toolbar with run & stop buttons. It uses the Processing VSCode Extension for sketch execution. The code is visible in this repository & branch. This document provides an overview of that prototype and enumerates unresolved issues for consideration.
The prototype is not being presented for code review, as it is not aligned with the processing4 repo’s AI policy. Creating this prototype was a useful exercise for me to understand what parts of the VSCode repository might be most relevant for creating a PDE-like experience. This documentation contains my notes and what I learned from reading/reviewing the prototype implementation.
I welcome community feedback on this approach, suggestions for other open source editors to evaluate as an alternative to VSCode, and comparisons to other possible PDE evolutions from other Processing Foundation projects (including processing4, the p5.js Web Editor, and the prototype collaborative editor).
VSCode Prototype Implementation
Most customizations in this prototype are self-contained in Extensions, with the SketchToolbar (where the play & stop buttons live) as one notable exception. This bodes well for keeping the project up to date with the VSCode mainline for important security patches and bugfixes.
Workbench Extension: Beginner Mode
The BeginnerMode Workbench extension hides some of the VSCode UI by overriding user preferences about which panels, buttons, and controls to show. All side panels are hidden, as well as the command center and AI-assistance interface elements. This extension also hides the debugger and console by default, but the console is revealed when the user runs code.
Side note: Claude picked the name for this class based on how I described the core audience for the PDE, we could definitely name it something else.
Copilot AI Extensions Disabled
All AI code generation extensions are disabled in this prototype, to 1) keep the UI minimal and 2) give educators the option to focus on teaching without AI. Some custom code is necessary for this because Microsoft really doesn’t want people to turn Copilot off!
If we do stick with VSCode as the base implementation, we’d have the option to easily add in AI support if it seems important to our users—or someone else could fork the project and re-enable it to suit their needs. We can also look at how VSCodium, another open source fork, approaches removing AI support from the app (and maybe also the codebase) if that is a preferable option.
Sketch Toolbar
A new Sketch Toolbar component, which contains the run & stop buttons, appears between the title bar and editor. The activity bar, status bar, command center, and other advanced features are hidden from VSCode’s Title Bar to keep things simple for beginners. The changes for the Sketch Toolbar are the main changes that were not self-contained in their own extension, as the toolbar needs to be inserted into a part of the UI that Extensions can’t access.
Repurposing the Processing VSCode Extension (and using other Extensions too)
The prototype includes our existing Processing VSCode extension as the engine for building and running sketches, which means we utilize the Processing CLI to build sketches and access the LSP. The Processing CLI currently requires the whole PDE to already be installed on the user’s computer, so we’ll want to work toward making the CLI standalone to remove the dependency on the current PDE implementation.
Authors of new Modes (to use terminology from processing4) in this prototype version of the PDE can also write an Extension that would be useful to VSCode users. Working on a libprocessing Python mode is one of my next steps for this proof of concept project.
Welcome Screen
The processingWelcome Workbench Extension introduces a new empty window screen based on the welcome screen implemented in recent versions of processing4. I have yet to try to add images or styling to this page, but this should be easy to customize. There are some quirks with how this is hooked up to the filesystem and my existing mental model of a Sketchbook as a long time Processing user, but they are likely easy to fix with additional time and didn’t feel essential to include in a proof of concept prototype.
Future Considerations
While this prototype suggests a PDE-like interface on the VSCode codebase is possible, there are some other key issues to resolve in determining the right direction for the next version of the PDE.
License compatibility with PDE and Processing CLI
The processing4 repo uses LGPL to license the Processing Core and GPL to license the PDE. I don’t fully understand what this means for bundling the Processing VSCode Extension (and perhaps the Processing CLI) in an MIT License project (VSCode Editor fork).
UI Design
Sketch Tabs
In Java mode, the PDE supports multiple tabs for organizing code, which the preprocessor assembles into a single class. How would this be designed and implemented in the new PDE?
Sketchbook & Examples Browsers
VSCode seems to prefer panels and tabs for Extensions over new independent windows, which is in conflict with the current UI design of the PDE. We’d need to come up with a workaround to have more popup windows in VSCode, or come up with a new UI design for browsing the user’s sketchbook folder and examples.
Importing Libraries and adding new Modes
The PDE could use VSCode Extensions directly through Open VSX (note: Microsoft’s Marketplace code is not open source and can’t be accessed in forks of VSCode). This prototype has an updated product.json file for Open VSX, but doesn’t yet have a UI to handle contributions. I’m also curious to learn if there’s a way we can vet Extensions as compatible (or not) with the PDE if we use Open VSX. Could we keep a list of allowed Extensions, and how would this work? Could we use this to replace the Contribution Manager?
What easy UI can we provide to beginners for browsing and importing libraries needed in their projects? Can we find an existing Extension that provides a friendly interface for package managers like maven and npm?
How will the user find and download new Modes? VSCode has good detection for different file types and supported syntax highlighting, so we could likely forgo the dropdown selector that the current PDE has. What should the behavior be if the user has different types of code in one folder? To what extent should the PDE try to shape what can and can’t be a single sketch/folder?
Interface Design & processing4 feature parity TODO list
Telemetry
Microsoft’s proprietary telemetry tools are disabled in the MIT License version of the repository. However, we could probably define our own endpoint for reporting if wanted. Should we? What are the privacy and GDPR considerations? It would be nice to know more about which features our users can’t live without, but other forks committed to open source values (like VSCodium) disable the telemetry features. I also haven’t researched this deeply enough to know how invasive the existing infrastructure would be for our users, maybe there is a middle-ground solution where we report a smaller set of data than what Microsoft collects.
tl;dr
VSCode fork with simplified UI, including Processing-style run & stop buttons
- Minimal changes to core VSCode UI needed, indicating it may be easy to keep the project up to date with changes from Microsoft’s well-resourced engineering team
- Syntax highlighting & building the project are accomplished with the existing Processing VSCode Extension (using the Processing CLI and LSP)
- AI code assist disabled in this prototype UI, but could easily be added back in if needed or desired
- Includes simple welcome screen similar to processing4 as crude proof of concept
Open issues and questions
- Licensing challenges between processing4 CLI (LGPL) and VSCode (MIT License) may impact how we can bundle the Processing Extension in the app
- Unimplemented features that need more research: importing libraries, tabs, support for multiple modes
- TODO list with processing4 features not represented in this prototype are listed above, to give a sense of project scope
- Telemetry: do we want it and how invasive is what’s there, if used?
Would you like to help implement this enhancement?
I’m not sure yet
Most relevant area for this enhancement?
PDE
Feature to enhance
Maintainability
Enhancement details
I’ve been curious about using another open source code editor as the core user interface for the Processing Development Environment (PDE). Some high level goals for such an implementation would be:
The part of the VSCode source released under the MIT License is the core of several other programming environments, including Cursor and Theia. There are some important features we can leverage from the VSCode codebase: developer extensions, syntax highlighting for many different programming languages, internationalization, telemetry, and AI code support. VSCode’s user interface is built with Typescript and Electron, which are both widely used in front-end web development. VSCode is also a project maintained by a well-resourced team of engineers and designers at Microsoft, who can put a lot more time into maintenance and security than our small (but mighty) team from Processing Foundation.
I used Copilot/Claude to build a PDE-like UI from a fork of the VSCode project for evaluative purposes, based on a suggestion from Charlotte. This proof of concept incorporates some of the PDE’s most important beginner-friendly features in the VSCode UI, such as the welcome screen and a toolbar with run & stop buttons. It uses the Processing VSCode Extension for sketch execution. The code is visible in this repository & branch. This document provides an overview of that prototype and enumerates unresolved issues for consideration.
The prototype is not being presented for code review, as it is not aligned with the processing4 repo’s AI policy. Creating this prototype was a useful exercise for me to understand what parts of the VSCode repository might be most relevant for creating a PDE-like experience. This documentation contains my notes and what I learned from reading/reviewing the prototype implementation.
I welcome community feedback on this approach, suggestions for other open source editors to evaluate as an alternative to VSCode, and comparisons to other possible PDE evolutions from other Processing Foundation projects (including processing4, the p5.js Web Editor, and the prototype collaborative editor).
VSCode Prototype Implementation
Most customizations in this prototype are self-contained in Extensions, with the SketchToolbar (where the play & stop buttons live) as one notable exception. This bodes well for keeping the project up to date with the VSCode mainline for important security patches and bugfixes.
Workbench Extension: Beginner Mode
The BeginnerMode Workbench extension hides some of the VSCode UI by overriding user preferences about which panels, buttons, and controls to show. All side panels are hidden, as well as the command center and AI-assistance interface elements. This extension also hides the debugger and console by default, but the console is revealed when the user runs code.
Side note: Claude picked the name for this class based on how I described the core audience for the PDE, we could definitely name it something else.
Copilot AI Extensions Disabled
All AI code generation extensions are disabled in this prototype, to 1) keep the UI minimal and 2) give educators the option to focus on teaching without AI. Some custom code is necessary for this because Microsoft really doesn’t want people to turn Copilot off!
If we do stick with VSCode as the base implementation, we’d have the option to easily add in AI support if it seems important to our users—or someone else could fork the project and re-enable it to suit their needs. We can also look at how VSCodium, another open source fork, approaches removing AI support from the app (and maybe also the codebase) if that is a preferable option.
Sketch Toolbar
A new Sketch Toolbar component, which contains the run & stop buttons, appears between the title bar and editor. The activity bar, status bar, command center, and other advanced features are hidden from VSCode’s Title Bar to keep things simple for beginners. The changes for the Sketch Toolbar are the main changes that were not self-contained in their own extension, as the toolbar needs to be inserted into a part of the UI that Extensions can’t access.
Repurposing the Processing VSCode Extension (and using other Extensions too)
The prototype includes our existing Processing VSCode extension as the engine for building and running sketches, which means we utilize the Processing CLI to build sketches and access the LSP. The Processing CLI currently requires the whole PDE to already be installed on the user’s computer, so we’ll want to work toward making the CLI standalone to remove the dependency on the current PDE implementation.
Authors of new Modes (to use terminology from processing4) in this prototype version of the PDE can also write an Extension that would be useful to VSCode users. Working on a libprocessing Python mode is one of my next steps for this proof of concept project.
Welcome Screen
The processingWelcome Workbench Extension introduces a new empty window screen based on the welcome screen implemented in recent versions of processing4. I have yet to try to add images or styling to this page, but this should be easy to customize. There are some quirks with how this is hooked up to the filesystem and my existing mental model of a Sketchbook as a long time Processing user, but they are likely easy to fix with additional time and didn’t feel essential to include in a proof of concept prototype.
Future Considerations
While this prototype suggests a PDE-like interface on the VSCode codebase is possible, there are some other key issues to resolve in determining the right direction for the next version of the PDE.
License compatibility with PDE and Processing CLI
The processing4 repo uses LGPL to license the Processing Core and GPL to license the PDE. I don’t fully understand what this means for bundling the Processing VSCode Extension (and perhaps the Processing CLI) in an MIT License project (VSCode Editor fork).
UI Design
Sketch Tabs
In Java mode, the PDE supports multiple tabs for organizing code, which the preprocessor assembles into a single class. How would this be designed and implemented in the new PDE?
Sketchbook & Examples Browsers
VSCode seems to prefer panels and tabs for Extensions over new independent windows, which is in conflict with the current UI design of the PDE. We’d need to come up with a workaround to have more popup windows in VSCode, or come up with a new UI design for browsing the user’s sketchbook folder and examples.
Importing Libraries and adding new Modes
The PDE could use VSCode Extensions directly through Open VSX (note: Microsoft’s Marketplace code is not open source and can’t be accessed in forks of VSCode). This prototype has an updated product.json file for Open VSX, but doesn’t yet have a UI to handle contributions. I’m also curious to learn if there’s a way we can vet Extensions as compatible (or not) with the PDE if we use Open VSX. Could we keep a list of allowed Extensions, and how would this work? Could we use this to replace the Contribution Manager?
What easy UI can we provide to beginners for browsing and importing libraries needed in their projects? Can we find an existing Extension that provides a friendly interface for package managers like maven and npm?
How will the user find and download new Modes? VSCode has good detection for different file types and supported syntax highlighting, so we could likely forgo the dropdown selector that the current PDE has. What should the behavior be if the user has different types of code in one folder? To what extent should the PDE try to shape what can and can’t be a single sketch/folder?
Interface Design & processing4 feature parity TODO list
Telemetry
Microsoft’s proprietary telemetry tools are disabled in the MIT License version of the repository. However, we could probably define our own endpoint for reporting if wanted. Should we? What are the privacy and GDPR considerations? It would be nice to know more about which features our users can’t live without, but other forks committed to open source values (like VSCodium) disable the telemetry features. I also haven’t researched this deeply enough to know how invasive the existing infrastructure would be for our users, maybe there is a middle-ground solution where we report a smaller set of data than what Microsoft collects.
tl;dr
VSCode fork with simplified UI, including Processing-style run & stop buttons
Open issues and questions
Would you like to help implement this enhancement?
I’m not sure yet