Produce and Publish the set of vscode built in extensions
- Produce and Publish the set of vscode built in extensions
TL;DR: Eclipse Theia requires a set of about 80 so-called "vscode built-in extensions". We build/package them ourselves, from the MIT-licensed vscode repository, and then publish them individually to Open VSX. We do not modify the extensions, other than a couple of very minor adaptations, to make them suitable to work as standalone extensions.
The "vscode builtins", "vscode built-ins" or "vscode built-in extensions" are a set of extensions whose code resides in the public vscode repository. They are built along and bundled as a group, in the Visual Studio Code product as well as in products based on Code OSS and derivatives such as VSCodium. As such, they are not made available as individual .vsix packages, for use in other IDE applications, outside of the vscode family. This is why we have this repo here - to build, package and individually publish, the various built-in extensions.
Every sub-folder of vscode/extensions/ is one built-in vscode extension (with a couple of exceptions like node_modules after a build)
-
Some built-in extensions have no source code as such, and contribute the means to perform static rule-based syntax highlighting, through a TextMate grammar, and sometimes also some code snippets, for a particular programming language. E.g.
vscode.bat,vscode.javascript. -
A few built-in extensions contribute advanced language support for a given language, including a Language Server, a debug adapter, command/tasks contributions and more. e.g.
vscode.typescript-language-features,vscode.markdown-language-features,vscode.html-language-features, 'vscode.json-language-features'. Finally, some built-ins contribute misc potentially useful features, like various themes, integration to a 3PP services likevscode.github*and more. -
Integration points: some built-in extensions provide an API, for other extensions to use. For example, the two git built-in extensions provide runtime APIs for extenders to use, e.g. to be kept informed of the current active git repository and checked-out branch:
See also: Consuming Builtin and External VS Code Extensions
In Theia-based application, there is often a need for the features provided by various built-ins. To obtain some or all of the builtin extensions, the IDE author can include all built-in extensions by adding to the app's package.json, the URL of one of our builtins extension pack - open-vsx.org, produced and published alongside the builtin extensions. It's possible to exclude unwanted extensions, referenced by the pack, by adding them to the theiaPluginsExcludeIds list. It's also possible to curate the exact list of wanted built-ins and add them individually to the application's package.json and let the user install more later if needed. In theory they could decide to bundle no built-ins at all, if that fits their purpose.
From the example Theia application, in section "theiaPlugins":
theiaPlugins": {
"vscode.markdown-language-features": "https://open-vsx.org/api/vscode/markdown-language-features/1.64.2/file/vscode.markdown-language-features-1.64.2.vsix",
"eclipse-theia.builtin-extension-pack": "https://open-vsx.org/api/eclipse-theia/builtin-extension-pack/1.70.2/file/eclipse-theia.builtin-extension-pack-1.70.2.vsix",
"EditorConfig.EditorConfig": "https://open-vsx.org/api/EditorConfig/EditorConfig/0.14.4/file/EditorConfig.EditorConfig-0.14.4.vsix",
"dbaeumer.vscode-eslint": "https://open-vsx.org/api/dbaeumer/vscode-eslint/2.1.20/file/dbaeumer.vscode-eslint-2.1.20.vsix"
},How to exclude certain pack extensions:
"theiaPluginsExcludeIds": [
"ms-vscode.js-debug-companion",
"vscode.extension-editing",
"vscode.git",
"vscode.git-base",
"vscode.github",
"vscode.github-authentication",
"vscode.ipynb",
"vscode.microsoft-authentication",
"ms-vscode.references-view"
]Eclipse Theia is an Eclipse Foundation project, we follow the Foundation's IP process regarding 3PPs (runtime dependencies as well as "build and test" dependencies, obtained from the public npm registry). See the Eclipse Foundation Project Handbook IP section for more details. We use dash-licenses to obtain (mostly) automated approval for the 3PP dependencies used to build the built-in extensions.
For any new version of the built-in extensions we want to produce and publish, we:
- open an IP Check ticket on the Eclipse Foundation Gitlab, attach the source code of all the extension we use
- Run the 3PP License check[*] on that version of the built-ins, automatically open IP tickets for new found dependencies of unclear license. See below for more details about these tickets
When all tickets have been approved, we can proceed with publishing the new version.
[*] details here
Result: 2 dependencies were flagged for more scrutiny:
npm/npmjs/-/jschardet/3.0.0, LGPL-2.1 AND LGPL-2.1+, restricted, clearlydefined
npm/npmjs/@octokit/openapi-types/13.6.0, MIT, restricted, clearlydefined
vscode-builtin-extensions$ export DASH_LICENSES_TOKEN=<token>
vscode-builtin-extensions$ find vscode/extensions -name yarn.lock ! -path '*node_modules*' -exec bash -c "cd \`dirname {}\` && pwd && java -jar $DASH_LICENSES_JAR yarn.lock -timeout 120 -batch 20 -summary ./summary.txt -review -token $DASH_LICENSES_TOKEN -project ecd.theia" \;/home/lmcmcds/marc/vscode-builtin-extensions/vscode/extensions/hlslOne ticket was opened by dash-licenses: "npm/npmjs/@octokit/openapi-types/13.6.0"
Open a ticket towards the Eclipse Foundation IP Team here (gitlab.eclipse.org)
Click new issue,
Title: "project/ecd.theia/-/vscode-builtin-extensions/<version>"
e.g.:
Title: project/ecd.theia/-/vscode-builtin-extensions/1.72.2
Description -> "Choose a template": vet-third-party
Follow the instructions of the top of the template, and start to fill-in the content. Take inspiration:
Project: [ecd.theia](https://projects.eclipse.org/projects/ecd.theia)
For more information about `vscode built-in extensions` please see: https://github.com/eclipse-theia/vscode-builtin-extensions/wiki/Produce-and-Publish-the-set-of-vscode-built-in-extensionsWe will produce a source archive for the built-ins we want to publish and attach it to the IP ticket, for the IP team to analyze.
# only keep sources, not build artifacts
vscode-builtin-extensions/vscode/extensions$ git clean -ffdx
# clean-upo test content
vscode-builtin-extensions/vscode/extensions$ find -iname 'test' -execdir rm -rf {} \;
vscode-builtin-extensions/vscode/extensions$ rm -rf ./vscode-colorize-tests
vscode-builtin-extensions/vscode/extensions$ cd ..
vscode-builtin-extensions/vscode$ tar czvf vscode-builtin-extensions-1.72.2.tar.gz ./extensions/
vscode-builtin-extensions-1.72.2.tar.gz is the archive to attach to the IP check ticket.
Copy the file somewhere handy. Open a file navigator and find the archive. Drag it to the browser tab where the IP ticket is being composed, in the description of the ticket. It will look like:
[vscode-builtin-extensions-1.72.2.tar.gz](/uploads/1494b604e895696df3cc848eff744973/vscode-builtin-extensions-1.72.2.tar.gz)Go into preview more and copy the link. It will be something like:
https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab/uploads/1494b604e895696df3cc848eff744973/vscode-builtin-extensions-1.72.2.tar.gzUse this URL for the Source field in the basic information section:
# Basic Information
- License: MIT
- Copyright Holder: Microsoft
- [Git repository](https://github.com/microsoft/vscode/tree/1.70.2/extensions)
- [Binary]()
- [Full Source](https://github.com/microsoft/vscode/archive/refs/tags/1.70.2.zip)
- [Source](https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab/uploads/1494b604e895696df3cc848eff744973/vscode-builtin-extensions-1.72.2.tar.gz)See the IP Check ticket created for the built-ins 1.72.2, created along with this guide.
At the moment this is only informational. This check uncovers known vulnerabilities in 3PP dependencies, in specific versions of dependencies, used by the current set of built-ins. There used to be a lot when we were a couple of years late in producing a set of older built-ins. The portrait has recently improved a lot, because we have made rapid progress in supporting newer vscode extensions API versions. This means the vscode built-in extensions we produced recently are based on more recent packages, that have accumulated less vulnerability reports. When we catch-up and stay on top of new APIs as they are released, the built-in extensions we build will have close to zero vulnerabilities, in the moment when we produce and publish them, just like their twins that are part of the most recently released version of Visual Studio Code product.
See here for how to run yarn audit, for the full set of built-in extensions.
Start with the latest master state:
cd vscode-builtin-extensions
git clean -ffdx
git checkout master
git pull origin masterCreate a branch for the vscode builtins you intend to build/publish:
BIVERSION="1.72.2" # for example
git branch $BIVERSION
git checkout $BIVERSIONCheckout vscode git submodule at wanted version
git submodule init
git submodule update
cd vscode
git clean -ffdx
git checkout $BIVERSION
cd ..Build the built-ins, package them as .vsix and generate an extension pack:
yarn && yarn build:extensions
# proceed if everything built correctly
yarn package-vsix:latest && yarn create-extension-pack:latestThe extensions, including the pack, will be under folder dist, each packaged as a .vsix file.
vscode-builtin-extensions$ ls dist/
bat-1.72.1.vsix html-language-features-1.72.1.vsix references-view-1.72.1.vsix
builtin-extension-pack-1.72.1.vsix ini-1.72.1.vsix restructuredtext-1.72.1.vsix
builtin-notebook-renderers-1.72.1.vsix ipynb-1.72.1.vsix ruby-1.72.1.vsix
clojure-1.72.1.vsix jake-1.72.1.vsix rust-1.72.1.vsix
coffeescript-1.72.1.vsix java-1.72.1.vsix scss-1.72.1.vsix
configuration-editing-1.72.1.vsix javascript-1.72.1.vsix search-result-1.72.1.vsix
cpp-1.72.1.vsix json-1.72.1.vsix shaderlab-1.72.1.vsix
csharp-1.72.1.vsix json-language-features-1.72.1.vsix shellscript-1.72.1.vsix
css-1.72.1.vsix julia-1.72.1.vsix simple-browser-1.72.1.vsix
css-language-features-1.72.1.vsix latex-1.72.1.vsix sql-1.72.1.vsix
dart-1.72.1.vsix less-1.72.1.vsix swift-1.72.1.vsix
debug-auto-launch-1.72.1.vsix log-1.72.1.vsix theme-abyss-1.72.1.vsix
debug-server-ready-1.72.1.vsix lua-1.72.1.vsix theme-defaults-1.72.1.vsix
diff-1.72.1.vsix make-1.72.1.vsix theme-kimbie-dark-1.72.1.vsix
docker-1.72.1.vsix markdown-1.72.1.vsix theme-monokai-1.72.1.vsix
emmet-1.72.1.vsix markdown-language-features-1.72.1.vsix theme-monokai-dimmed-1.72.1.vsix
extension-editing-1.72.1.vsix markdown-math-1.72.1.vsix theme-quietlight-1.72.1.vsix
fsharp-1.72.1.vsix media-preview-1.72.1.vsix theme-red-1.72.1.vsix
git-1.72.1.vsix merge-conflict-1.72.1.vsix theme-solarized-dark-1.72.1.vsix
git-base-1.72.1.vsix npm-1.72.1.vsix theme-solarized-light-1.72.1.vsix
github-1.72.1.vsix objective-c-1.72.1.vsix theme-tomorrow-night-blue-1.72.1.vsix
github-authentication-1.72.1.vsix perl-1.72.1.vsix typescript-1.72.1.vsix
go-1.72.1.vsix php-1.72.1.vsix typescript-language-features-1.72.1.vsix
groovy-1.72.1.vsix php-language-features-1.72.1.vsix vb-1.72.1.vsix
grunt-1.72.1.vsix powershell-1.72.1.vsix vscode-theme-seti-1.72.1.vsix
gulp-1.72.1.vsix pug-1.72.1.vsix xml-1.72.1.vsix
handlebars-1.72.1.vsix python-1.72.1.vsix yaml-1.72.1.vsix
hlsl-1.72.1.vsix r-1.72.1.vsix
html-1.72.1.vsix razor-1.72.1.vsixThis section assumes you have a local clone of the main Theia repo.
cd theia # go in the theia main git repository folderIf required, step the supported vscode API in file dev-packages/application-package/src/api.ts. Luckily, it did not need to be done in this case:
/**
* The default supported API version the framework supports.
* The version should be in the format `x.y.z`.
*/
export const DEFAULT_SUPPORTED_API_VERSION = '1.72.2';Clean-build the example application:
git clean -ffdx
yarn && yarn browser buildIf already present, delete folder plugins. We will instead use the built-ins we previously built
rm -rf plugins
mkdir pluginsCopy the builtin extensions we produced, to Theia's plugins folder.
cd vscode-builtin-extensions # back to vscode-builtin0-extensions repo
cp -a dist/* ../theia/plugins # adjust according to where your theia folder residesGet rid of a few builtins that will interfere with testing (note: we keep these extensions where they were generated, but remove them from our test Theia application):
cd theia # back to theia repo
rm -rf plugins/ipynb-*
rm -rf plugins/extension-editing-*
rm -rf plugins/github-*To test vscode builtin git, we need to remove the Theia-specific git extension from the example application:
theia$ git diff
diff --git a/examples/browser/package.json b/examples/browser/package.json
index da42a2b6004..f5f52d7ffcd 100644
--- a/examples/browser/package.json
+++ b/examples/browser/package.json
@@ -25,7 +25,6 @@
"@theia/file-search": "1.36.0",
"@theia/filesystem": "1.36.0",
"@theia/getting-started": "1.36.0",
- "@theia/git": "1.36.0",
"@theia/keymaps": "1.36.0",
"@theia/markers": "1.36.0",
"@theia/memory-inspector": "1.36.0",
diff --git a/examples/electron/package.json b/examples/electron/package.json
index 1f81d6076c6..35d1b6d927c 100644
--- a/examples/electron/package.json
+++ b/examples/electron/package.json
@@ -27,7 +27,6 @@
"@theia/file-search": "1.36.0",
"@theia/filesystem": "1.36.0",
"@theia/getting-started": "1.36.0",
- "@theia/git": "1.36.0",
"@theia/keymaps": "1.36.0",
"@theia/markers": "1.36.0",
"@theia/memory-inspector": "1.36.0",Re-build and start the example application (e.g. the browser version):
yarn && yarn browser build
yarn browser start- Connect to
localhost:3000with a browser - Observe backend log for new exceptions, specially during activation of builtin extensions
- quick TypeScript test
- quick JSON test
- quick git test
- Submit PR for current builtins versions for review and merge.
Register issues for problems found. Some problems may require changing how we build or package, in which case a fix would be made on vscode-builtin-extensions as part of the ongoing release PR. If the issue is with the upstream Theia repo, we open the issue there.
While testing buitins 1.72.2, we found the following:
- RangeError: Maximum call stack size exceeded with recent vscode.html builtin extension #12434
- [builtins] [proposed API] [vscode.markdown-language-features]: Theia misses proposed API:
Document Paste - [builtins] [proposed API] [vscode.git@1.72.2]: Theia misses proposed API:
Edit session identifier provider
Once you are confident that the new set of builtins do not have obvious issues, you can proceed with publishing them to open-vsx.org. It's ok if there are issues that will later be fixed in Theia - older version of the builtin can be temporarily used instead in most cases.
Commit your changes:
cd vscode-builtin-extensions # back to vscode-builtin0-extensions repo
git branch # confirm you are in the "version" branch, e.g. "1.72.2"
# add any file you modified and commit
git add vscode/ # the change in checked-out version of the vscode git sub-module
git commit -s -m "changes for builtins v1.72.2"
git push origin 1.72.2 # from there, create a PR.Publishing is done using GitHub Actions. In the vscode-builtin-extensions repo, a publish token for open-vsx.org has been set, that can be used to publish under the identity of the openvsx publish bot. This means that we need to automatically obtain the extensions to publish, without manual intervention on our part.
Before publishing: Make sure that you have performed the built-ins 3PP license check. See here
Our GitHub workflow is aware of two "magic" git branches, that we use to publish:
They are respectively used to trigger the run of a workflow that will attempt to publish a "latest" or "next" version of the built-ins set.
If required, update the relevant GitHub workflow. In this case, vscode-builtin-extensions/.github/workflows/publish-vsx-specific-latest.yml
For example if a new node version is required to build, update that. If the build/packaging procedure changes, reflect the change in the workflow(s). You may want to include these changes in your PR under review.
git checkout 1.72.2
git branch -D ovsx-publish # if the branch already exists
git branch ovsx-publish
git checkout ovsx-publishWhen ready, push the branch to the upstream repo: this will trigger an attempt at building and publishing the builtins: note: this requires write permission in upstream repo, i.e. being a committer
git checkout ovsx-publish
git push origin ovsx-publish
# if will probably fail because we already have an old version on that branch upstream. Use the force:
git push -f origin ovsx-publish
# for posterity, keep a copy of the publish branch. This way we can later know what exactly was published
git branch old-ovsx-publish-1.72.0
git push origin old-ovsx-publish-1.72.0Go in the "Actions" tab to follow the publishing
New vscode proposed API seem to often first be introduced in built-in extensions. Other, non built-in extensions, are not allowed to use proposed API without serious consequences: not allowed published on the Visual Studio Marketplace and if otherwise distributed, cannot be used in vanilla Visual Studio Code. Instead, the cutting-edge Insiders distribution of vscode must (allegedly) be used (as per documentation above - the requirement is not mentioned in other places in the documentation, so we are not sure).
"They [proposed API] are subject to change, only available in Insiders distribution and cannot be used in published extensions."
This brings the potential need for some tracking to happen, regarding built-ins and proposed API. So far, we have noticed very obvious cases built-ins not working in Theia apps, often because they fail during activation and are rendered useless after. However some more subtle use of incompatible proposed API can occur, e.g. only triggered in a special context. In consequence, it might be good to:
- do a deep pass, checking that all the currently used proposed API in the built-ins is in-line with what the code expects.
- take note whether each proposed API is fully implemented or stubbed
- catch-up to current proposed API, adding stubs or implementations as needed.
- once we have caught-up, update monthly to follow most recent changes in vscode
- produce the built-ins of corresponding version - they should work with the current version of Theia. Most will keep on working with future versions of Theia, unless some proposed API is changed in the meantime.
- consider plan to replace stubbed proposed API, that we know is used in the built-ins, with a proper implementation.
In Theia we have at least stubs for most proposed API, that are used in the built-ins (< 1.72.2) . We are currently missing a few relatively recent ones. If we want to continue to benefit from the features offered by recent built-ins from vscode, we have to keep pace, and implement some workable version of the used proposed API. We also must be ready to adapt if the API changes, which is not unlikely over several releases. We must also follow when Proposed API becomes solid API, with or without modifications from its recent Proposed version.
In vscode and derivatives, they have us at an advantage here - since the built-ins are always built and bundled with vscode, they will be aligned with how this API is used in the built-ins. There is no chance for "de-synchronization" between the built-ins and the Proposed API. In Theia applications, we do not have that luxury; rather we need to make sure that the built-ins, used a given version of the Theia framework, will find resolution to the API they are using, including proposed API.
How to more formally handle proposed API in Theia is under community discussion. This affects compatibility of built-ins we produce, when run in Theia-based applications.
Here's how to see the proposed API used by each built-in of a given version:
# list proposed API used in built-ins:
vscode-builtin-extensions$ git submodule foreach "grep -irn enabledApiProposals --include=package.json | cut -d ':' -f 1 | xargs jq --raw-output '{filename: input_filename, enabledApiProposals: .enabledApiProposals}'"
# save the output to a file for later comparison with other versions
vscode-builtin-extensions$ git submodule foreach "grep -irn enabledApiProposals --include=package.json | cut -d ':' -f 1 | xargs jq --raw-output '{filename: input_filename, enabledApiProposals: .enabledApiProposals}'" > ./vscode-builtins-enabled-proposals-1.72.2.txtSee below, the output for 1.72.2. A couple of proposed API were found to be missing during our tests:
- vscode.proposed.documentPaste.d.ts, vscode blog post, Theia issue
- vscode.proposed.editSessionIdentityProvider.d.ts, Theia issue
Going through the list systematically, making sure our version of each API works well enough, and adapting to new or modified API each vscode release, would be the solid way to go.
Entering 'vscode'
{
"filename": "extensions/vscode-test-resolver/package.json",
"enabledApiProposals": [
"resolvers"
]
}
{
"filename": "extensions/git/package.json",
"enabledApiProposals": [
"diffCommand",
"contribEditorContentMenu",
"contribViewsWelcome",
"editSessionIdentityProvider",
"scmActionButton",
"scmSelectedProvider",
"scmValidation",
"tabInputTextMerge",
"timeline",
"contribMergeEditorMenus"
]
}
{
"filename": "extensions/html-language-features/package.json",
"enabledApiProposals": [
"extensionsAny"
]
}
{
"filename": "extensions/github/package.json",
"enabledApiProposals": [
"contribShareMenu",
"contribEditSessions"
]
}
{
"filename": "extensions/github-authentication/package.json",
"enabledApiProposals": [
"extensionLog"
]
}
{
"filename": "extensions/ipynb/package.json",
"enabledApiProposals": [
"documentPaste",
"diffContentOptions"
]
}
{
"filename": "extensions/typescript-language-features/package.json",
"enabledApiProposals": [
"resolvers",
"workspaceTrust",
"workspaceEditIsRefactoring"
]
}
{
"filename": "extensions/markdown-language-features/package.json",
"enabledApiProposals": [
"documentPaste"
]
}
{
"filename": "extensions/debug-server-ready/package.json",
"enabledApiProposals": [
"terminalDataWriteEvent"
]
}
{
"filename": "extensions/search-result/package.json",
"enabledApiProposals": [
"documentFiltersExclusive"
]
}
{
"filename": "extensions/vscode-api-tests/package.json",
"enabledApiProposals": [
"authSession",
"contribViewsRemote",
"customEditorMove",
"diffCommand",
"documentFiltersExclusive",
"editorInsets",
"extensionRuntime",
"extensionsAny",
"externalUriOpener",
"fileSearchProvider",
"findTextInFiles",
"fsChunks",
"notebookCellExecutionState",
"notebookContentProvider",
"notebookControllerKind",
"notebookDebugOptions",
"notebookDeprecated",
"notebookLiveShare",
"notebookMessaging",
"notebookMime",
"portsAttributes",
"quickPickSortByLabel",
"resolvers",
"scmActionButton",
"scmSelectedProvider",
"scmValidation",
"taskPresentationGroup",
"terminalDataWriteEvent",
"terminalDimensions",
"envShellEvent",
"testCoverage",
"testObserver",
"textSearchProvider",
"timeline",
"tokenInformation",
"treeItemCheckbox",
"treeViewReveal",
"workspaceTrust",
"telemetry",
"localization"
]
}
{
"filename": "extensions/simple-browser/package.json",
"enabledApiProposals": [
"externalUriOpener"
]
}
{
"filename": "extensions/microsoft-authentication/package.json",
"enabledApiProposals": [
"idToken",
"extensionLog"
]
}