Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up CI system to create desktop installer of vanilla Theia #5481

Closed
thegecko opened this issue Jun 14, 2019 · 67 comments
Closed

Set up CI system to create desktop installer of vanilla Theia #5481

thegecko opened this issue Jun 14, 2019 · 67 comments
Assignees
Labels
electron issues related to the electron target enhancement issues that are enhancements to current functionality - nice to haves

Comments

@thegecko
Copy link
Member

Feature requirement to extend CI systems to build installers for the desktop flavour of Theia for:

  • Windows
  • MacOS
  • Linux

In order to correctly sign these, they will need to be built on Windows, Mac and Linux machines accordingly.

Options for systems to include:

  • Custom Jenkins setups (already proven at arm) available from the Eclipse foundation
  • Hosted build service for each OS (e.g. AppVeyor for Windows)

cc @arekzaluski @svenefftinge

@thegecko thegecko added the enhancement issues that are enhancements to current functionality - nice to haves label Jun 14, 2019
@thegecko thegecko self-assigned this Jun 14, 2019
@sr229
Copy link

sr229 commented Jun 15, 2019

@thegecko FWIW, Travis now has a Windows VM for you to use. You may also use Azure pipelines as well.

@thegecko
Copy link
Member Author

Thanks. The Mac ci may be tricky

@thegecko thegecko added this to To Do in Version 1.0 via automation Jun 18, 2019
@kittaakos
Copy link
Contributor

use Azure pipelines

+1

The Mac ci may be tricky

We have not had problems producing a dmg for macOS with Azure so far.

@thegecko
Copy link
Member Author

thegecko commented Jul 2, 2019

We have not had problems producing a dmg for macOS with Azure

Have you tried signing the image (if we care about that)?

@kittaakos
Copy link
Contributor

Have you tried signing the image

No.

@sr229
Copy link

sr229 commented Jul 3, 2019

At this point we can have our CI migrated to Azure Pipelines since it supports MacOS, Windows and Linux already @thegecko @kittaakos

@akosyakov
Copy link
Member

akosyakov commented Aug 26, 2019

@kittaakos
Copy link
Contributor

GitHub actions instead

Does it support downloadable artifacts without creating a new release? Azure does.

Otherwise, I do not have any objections.

@sr229
Copy link

sr229 commented Aug 26, 2019

GitHub CI is still in beta so we'll need to wait for their november GA.

@thegecko
Copy link
Member Author

thegecko commented Sep 3, 2019

How about using GitHub actions

I believe GH actions use docker containers behind the scenes, so are Linux only.

This would mean we couldn't sign installers for Mac / Windows :(

@akosyakov
Copy link
Member

It was just a suggestion, I would be happy with anything working :)

@thegecko
Copy link
Member Author

thegecko commented Sep 3, 2019

I would be happy with anything working

I get the hint :p

@chrisjj
Copy link

chrisjj commented Feb 2, 2020

Any progress on this?

@thegecko
Copy link
Member Author

thegecko commented Feb 3, 2020

No progress as yet, many options have been suggested (Azure, Travis), but as I only have experience building and signing using Jenkins servers more help will be needed if we take a different approach.

As I see it, there are 3 options:

  • Setup up Jenkins servers taking advantage of the eclipse foundation build systems. This approach builds on a known release strategy we use for Mbed Studio, but requires access to eclipse servers and may diverge our CI from what we already have

  • Create something from scratch using Travis. I would need some support from someone with Travis foo as I don't have time to research/learn it :). We can use existing Jenkins scripts as a starting point.

  • Create something from scratch using Azure. Again, I would need some support from someone with Azure foo as I don't have time to research/learn it :). We can use existing Jenkins scripts as a starting point.

Any solution we choose needs to support native MacOS, Windows and Linux build machines in order to sign any installers (if that's a requirement).

Thoughts/volunteers?

@akosyakov
Copy link
Member

I somehow like to reuse eclipse build systems. I think it is fine that it is a separate CI, bundling from monorepo does not work anyway, correct? It would be also close to what others have to do. Could scripts be open sourced?

@thegecko
Copy link
Member Author

thegecko commented Feb 3, 2020

We can sanitize the scripts and share them to use as a base line. Do we have a contact at eclipse to access their build system?

@benoitf
Copy link
Contributor

benoitf commented Feb 3, 2020

opening a bugzilla to get CI instance at https://ci.eclipse.org/theia ?

@chrisjj
Copy link

chrisjj commented Feb 3, 2020

Thanks all.

@thegecko
Copy link
Member Author

thegecko commented Feb 3, 2020

@marcdumais-work
Copy link
Contributor

Any solution we choose needs to support native MacOS, Windows and Linux build machines in order to sign any installers (if that's a requirement).

Just a quick mention that Travis CI meets this requirement of supporting CI for our 3 platforms. Not sure about The Foundation's CI infrastructure - I guess they'll let us know in the bugzilla above.

@thegecko
Copy link
Member Author

thegecko commented Feb 5, 2020

Travis CI meets this requirement of supporting CI for our 3 platforms

Excellent, I just need a volunteer with travis foo :)

@marcdumais-work
Copy link
Contributor

Excellent, I just need a volunteer with travis foo :)

A few of us have played with Travis and can help.

When we talk about "vanilla Theia", do we mean the Electron Example application from the main repo, e.g. built and packaged nightly?

We have this example electron app that can already generate native packages for our 3 platforms (manually for now, not included in repo's CI). The electron-builder config could be used as an example, e.g. used along with the Electron Example app's package.json.

@arekzaluski
Copy link
Contributor

Yes. https://github.com/theia-ide/theia-apps/blob/master/theia-electron/electron-builder.yml is a really good start. After 1.0.0 it may be worth adding more improvements. For example:

  • Enabling Asar (It will require few changes in Theia code. Quite important to reduce the disk size and installation time of Theia)
  • Updating electron-builder version (There are few known issues in newer versions that may break backward compatibility)
  • Adding electron-updater for auto-updates

@marcdumais-work
Copy link
Contributor

There will be some preliminary tasks before we get to the CI/Travis aspect:

  • for each of our supported platforms, chose which packaging(s) we want. Note: For Linux there are packagings that are distro-specific (e.g. .deb, .rpm) and one that's generic (AppImage).
  • for each wanted packaging, ideally find a public registry where we can push the Theia app, once packaged. e.g. for Docker images, we use dockerhub. If in some cases we can't find such a public registry, we can consider using a generic file hosting service (e.g. Eclipse Foundation infrastructure maybe).
    • create an account on each of these registries. Ideally they can provide a revocable authentication token that we can use to authenticate during CI, before pushing. We can add the tokens as Travis environment variables, like we did for dockerhub and npmjs

@thegecko
Copy link
Member Author

We have this example electron app that can already generate native packages for our 3 platforms

I think this is the ideal candidate

There will be some preliminary tasks before we get to the CI/Travis aspect:

IMO, getting the installers building and signing will be the hardest part here. The content and publish location doesn't necessarily block work in this area. I'm happy to share some (potentially redacted!) Jenkins scripts to support anyone trying this on Travis. Let me know where to put them.

for each of our supported platforms, chose which packaging(s) we want. Note: For Linux there are packagings that are distro-specific (e.g. .deb, .rpm) and one that's generic (AppImage).

After a lot of research and trial/error on our side, we recommend the following:

  • Linux .deb (and .sh) installer containing an AppImage as it's far easier to upgrade
  • Windows .msi (in an .exe)
  • MacOS .pkg

for each wanted packaging, ideally find a public registry where we can push the Theia app, once packaged. e.g. for Docker images, we use dockerhub. If in some cases we can't find such a public registry, we can consider using a generic file hosting service (e.g. Eclipse Foundation infrastructure maybe).

DockerHub makes sense for the docker images (shall we create one now?). For the installers, we could simply add them to the GitHub releases on Theia?

create an account on each of these registries. Ideally they can provide a revocable authentication token that we can use to authenticate during CI, before pushing. We can add the tokens as Travis environment variables, like we did for dockerhub and npmjs

Sensible.

@ride4sun
Copy link

Sorry, but it sounded a good alternative to VSCODE to me when I was reading the announcements:

https://www.globenewswire.com/news-release/2020/03/31/2009010/0/en/The-Eclipse-Foundation-Releases-Eclipse-Theia-1-0-a-True-Open-Source-Alternative-to-Visual-Studio-Code.html

Quote:
Theia is designed from the ground to run on both Desktop and Cloud... Eclipse Theia is designed to work as a native desktop application as well as in the ...

Native Desktop which oh sorry you can not install because it is not important enough?

Even the title says that:
The Eclipse Foundation Releases Eclipse Theia 1.0, a True Open Source Alternative to Visual Studio Code

So let me get the correct... it is an alternative I can just not install it????

Quote: (https://devclass.com/2020/03/31/eclipses-theia-sees-1-0-declared-not-your-parents-ide/)
Mike Milinkovich, executive director of the Eclipse Foundation, said in a statement: “Visual Studio Code is one of the world’s most popular development environments. Not only does Theia allow developers to install and reuse VS Code extensions, it provides an extensible and adaptable platform that can be tailored to specific use cases.”

So I can install an extension like in VSCODE but I can not install THEIA???

@chrisjj
Copy link

chrisjj commented May 26, 2020 via email

@chrisjj
Copy link

chrisjj commented May 26, 2020 via email

@vince-fugnitto
Copy link
Member

@ride4sun if you look at the quotes you referenced you'll see the following:

Not only does Theia allow developers to install and reuse VS Code extensions, it provides an extensible and adaptable platform that can be tailored to specific use cases.

In the EclipseFoundation announcement you'll see:

The Eclipse Foundation, one of the world’s largest open source foundations, today announced the release of Theia 1.0, a true open source alternative to Microsoft’s popular Visual Studio Code (VS Code) software. Eclipse Theia is an extensible platform to develop multi-language Cloud and Desktop Integrated Development Environments (IDEs) with state-of-the-art web technologies that enable developers, organizations, and vendors to create new, extensible developer experiences.

It clearly illustrates that the project is a framework which can build IDE-like applications which in fact rival and is an open-source alternative to VS Code. The main target audience for the framework are application-developers who are in need of a framework to build their own white label IDE-like applications.

The idea of providing an executable or installer to end-users is for the ability for more of an audience to get a feel of the default functionality and behavior of the framework but once again it will only be one example of many possible use-cases. Nothing prevents you today to build you own application using Theia.

In any case, the addition of desktop installers for an example Theia application, is being actively worked on by the main community and hopefully it will bring forth more visibility and users to the framework.

@ride4sun
Copy link

I worked on many software projects myself and I have to admit that the installer is something nobody wants to work on. I have the feeling that is the non-outspoken reason here.

That is really stupid because I really want to try the new layout engine and you guys should be eager to get a big install base to gain momentum and get feedback.

@ride4sun
Copy link

ride4sun commented May 26, 2020

@vince-fugnitto

Let make it clear than....

Theia 1.0, a true open source alternative to Microsoft’s popular Visual Studio Code

This is NOT a Vscode alternative because it addresses another use case. Totally misleading!

@vince-fugnitto
Copy link
Member

@ride4sun

I worked on many software projects myself and I have to admit that the installer is something nobody wants to work on. I have the feeling that is the non-outspoken reason here.

That is really stupid because I really want to try the new layout engine and you guys should be eager to get a big install base to gain momentum and get feedback.

@ride4sun it is being worked on :)
Providing end-users with installers is something the community wants to ensure is done right, and the necessary planning, background tasks, setting up the CI jobs and permissions.

Let make it clear than....
This is NOT a Vscode alternative because it addresses another use case. Totally misleading!

It can be argued that it is an open-source alternative since vscode is not truly open-source, we are providing vscode extension support, we have worked on providing an open-source alternative to the marketplace for downstream applications to use (open-vsx), and the default commands, implementation and user-interface are vscode inspired.

If you disagree then that is totally your opinion, I think the multiple comments (re-iterating the same point) are sufficient for this particular issue.

@chrisjj
Copy link

chrisjj commented May 26, 2020 via email

@ride4sun
Copy link

ride4sun commented May 26, 2020

I agree with that statement but that is quite a different statement than the press releases.

It can be argued that it is an open-source alternative since vscode is not truly open-source, 
we are providing vscode extension support, we have worked on providing an open-source
alternative to the marketplace for downstream applications to use (open-vsx), and the default commands, implementation and user-interface are vscode inspired.

Can you share the community efforts where people working on the installer?

@vince-fugnitto
Copy link
Member

Can you share the community efforts where people working on the installer?

The main community has discussed this feature and other features in the weekly dev-meeting.

@chrisjj
Copy link

chrisjj commented May 26, 2020 via email

@vince-fugnitto
Copy link
Member

@chrisjj the issue has been discussed and planned by the main community, and one of the main contributing companies are currently working on implementing the feature. In the dev-meeting, the community has discussed on what the best approach would be to handle the automation of the installers, the CI setup (for different operating systems), and other important factors.

@ride4sun
Copy link

Is there a ETA?

@vince-fugnitto
Copy link
Member

Is there a ETA?

@ride4sun no, there is none as of today. We need to make sure that the automation of the installers are done correctly and with quality as ultimately end-users will be the ones to consume them. If you are pressed for time, building you own IDE is quite easy, we have many examples and you can start using it today. For example, if you require a desktop Theia IDE, you can take a look at the theia-electron example which when built and packaged you can continue using like any standard application.

@eclipse-theia eclipse-theia locked and limited conversation to collaborators May 27, 2020
@marcdumais-work
Copy link
Contributor

Locked except for committers. Pestering the project for one's pet feature is not productive.

As @vince-fugnitto said, Theia is a framework. It's targeted at IDE Developers that want an alternative to forking VS Code and/or something VS Code-like that also runs well in the Cloud. This project was never meant to host an IDE product based on Theia - there will be other Eclipse Foundation projects for that eventually. Here are the details: https://projects.eclipse.org/projects/ecd.theia

For those that just want to use (vs make/design/maintain) an alternate desktop IDE, like VS Code but open-source, your best bet right now is VSCodium. I have no doubt we'll have installers here soon, but what it will install will not be a full-fledged IDE product.

@thegecko would you share your plans as per the dev-meeting? I can then link to here from the minutes :)

@thegecko
Copy link
Member Author

@thegecko would you share your plans as per the dev-meeting

Sure, as discussed in the dev meeting yesterday, the plan is to stand up a travis build system for the theia apps project.

For now we want to prove this will work so Initially we aren't going to worry about signing any installers and will also defer decisions around the channels to be released.

To set delivery expectations, this is something I'm doing in my spare time, so please offer help if you want this accelerated. I see Windows support on travis is experimental, so we may hit blockers quickly.

@marcdumais-work
Copy link
Contributor

Thanks Rob.

I see Windows support on travis is experimental, so we may hit blockers quickly.

True, but we've been using Travis on this repo here, for our Windows CI, since we moved to the Eclipse Foundation's org, and as far as I remember, we've had little issues caused by the service (windows-specific issues, e.g. related to the specificities of their file system, are another matter)

See here for the windows part of the Travis config we use here.

@kittaakos
Copy link
Contributor

In order to correctly sign these

@thegecko, you have some experience with creating and shipping a signed electron-based app. Can you please tell us how long does it roughly take to sign the Arm Mbed Studio for macOS? Have you managed to pack with asar? Thank you!

I see Windows support on travis is experimental, so we may hit blockers quickly.

Recently, I have managed to sign a dummy electron app with GH Actions so if Travis CI cannot do the Windows part, we can use GH Actions.

Unfortunately, I was hitting timeout issues (with GH Actions) when I wanted to sign a Theia-based app. Probably because it's a giant, almost 500MB (VS Code is ~240 MB).

@thegecko
Copy link
Member Author

Can you please tell us how long does it roughly take to sign

Builds are about 40 minutes in total for each platform

Have you managed to pack with asar

We don't use asar

...if Travis CI cannot do the Windows part, we can use GH Actions

I wasn't aware GH actions were available for Windows, I assumed they were all docker images

@kittaakos
Copy link
Contributor

Builds are about 40 minutes in total for each platform

Does this 40 min include the singing against the Apple server?

@thegecko
Copy link
Member Author

Does this 40 min include the singing against the Apple server

Yes, it includes macOS notarization

@marcdumais-work
Copy link
Contributor

@thegecko I noticed that the VSCodium project generates packages for many OS, and it looks like they sign the Mac ones using Travis. Maybe this can serve as inspiration.

@kittaakos
Copy link
Contributor

Yes, it includes macOS notarization

Thank you, @thegecko. It was a very useful input. Does this mean you do not have the "sign" : true explicitly set in the config, you run the notarization only but not an explicit singing, right?

also defer decisions around the channels to be released.

I propose not using channels (well, just the default, latest), but having a separate insider build which gets an update each evening. VS Code uses this pattern, it works perfectly.

If anyone is wondering what the channels are: https://www.electron.build/tutorials/release-using-channels

@thegecko
Copy link
Member Author

thegecko commented Jun 6, 2020

PR for a first pass of this here: theia-ide/theia-apps#370

@thegecko
Copy link
Member Author

First releases of theia-example v1.2.0 are now appearing here:

https://download.eclipse.org/theia/

Still unsigned and currently untested.

@thegecko
Copy link
Member Author

thegecko commented Aug 18, 2020

Signed versions now available:

https://download.eclipse.org/theia/1.2.0/

MacOS version still needs to be notarised.

@thegecko
Copy link
Member Author

Happy to announce we now have signed and notarized installers for MacOS.

This means the CI work for creating desktop installers is complete.

The latest installers for all platforms can be found here:

https://download.eclipse.org/theia/

@marcdumais-work
Copy link
Contributor

Awesome, thanks @thegecko

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
electron issues related to the electron target enhancement issues that are enhancements to current functionality - nice to haves
Projects
None yet
Development

No branches or pull requests