What to do on version incompatibilities between IDE and the project's specified engine

[hubertp]

We had a number of issues recently related to the fact that newer IDE was forced to use incompatible engine version.
Examples include:

• Opening searcher takes ages and finally ends up with an error breaking the whole project. #7095
• "Failed to open visualization" / "method not found", unless I remove the "edition" line from the project's package.yaml #7426
  and numerous discussions about the version of newly created projects having a sticky and old version of the engine, therefore causing immediate problems.

While it would be nice to be always compatible between IDE and engine I don't think it is realistic at this point in time.
Specifying which versions break the protocol and therefore should be upgraded is also non-trivial and rather error prone (easy to miss the changes). I've seen projects defining such logic but it is a massive can of worms.

Should we for now abandon the whole fixed version schema and always use latest?

[hubertp]

/cc @wdanilo your feedback would be appreciated.

[radeusgd]

Should we for now abandon the whole fixed version schema and always use latest?

I think indeed using latest seems like a good idea. Still IMHO it may be worth to keep the edition in package.yaml by default marking the version with which the project was first created.

Once the project is opened in a different IDE/engine version combination, we could allow the different version but display a warning note to the user saying that "the project was for a different Enso version so there may be some API incompatibilities" (similar to the current "Unsupported engine version" message).

[JaroslavTulach]

What is a difference between:

• "Unsupported engine version"?
• "the project was for a different Enso version so there may be some API incompatibilities"?

If the project is for different engine version, then we have an unsupported engine version, don't we?

[radeusgd]

The latter is about project being incompatible with Enso version.

The former - "Unsupported engine version" was a message that was supposed to indicate that the IDE and engine versions are not compatible with each other.

[JaroslavTulach]

I see. So we have three elements here:

• the version of the engine requested by project inside of its package.yaml
• the version of the engine started by project-manager
• the version of the engine the IDE wants to work with

[hubertp]

One problem with using "latest" is that "latest" is not really a version. And I would expect a different set of problems caused by that fact and it would also make it harder for us to debug.
Other thoughts:

• display engine version number next to the project name or in some project's description tab (we don't have such a thing yet)
• maybe IDE could keep track of the version of the engine, compare it to the version of the IDE and iff there are problems in communication/execution display a warning that user should consider bumping the version

I particularly like the second one.

[JaroslavTulach]

Checking whether the IDE can communicate with the engine/LS process is indeed important. Doesn't that imply we should version the protocol? Shouldn't connection management messages specify version?

Or how else the IDE and engine/LS negotiate their versions and display of "Unsupported engine version"?

I see bunch of capabilities whose could also be used for some form of versioning. Anyway its all about attitude:

If we care about compatibility, then let us say: we are not going to do any incompatible changes to the protocol since today (tomorrow or ...)!

[vitvakatu]

iff there are problems in communication/execution display a warning that the user should consider bumping the version

I believe detecting such issues is hard, and distinguishing between incompatibilities and bugs is even more complicated.

We should split this problem into two parts:

1. What behavior is better for us while developing the project?
2. What would be better for our users, especially considering the upcoming release?

I think the answer for 2 includes the current behavior when the user will use the engine the project was created with. Therefore, they won't need to rewrite the code each time to accommodate changes in the standard library. At the same time, it should be our job to keep track of breaking changes between the IDE and the engine and warn users about it.

Once the project is opened in a different IDE/engine version combination, we could allow the different version but display a warning note to the user saying that "the project was for a different Enso version so there may be some API incompatibilities" (similar to the current "Unsupported engine version" message).

I don't like this approach because there will be incompatibilities, and the user will be forced to rewrite the project instead of doing his job. If I open my project from last week and see that it is not working with the new IDE and I can't change that because the latest engine is always used, I will be infuriated.

[JaroslavTulach]

the user will use the engine the project was created with

In the context of this classification, this approach fixes the mismatch between project and engine. But it doesn't address the incompatibilities between the IDE and engine/LS.

There is no guarantee new IDE will understand the old protocol. Moreover there is one more problem...

[JaroslavTulach]

New visualizations (which are currently part of the IDE) and drop down support are not guaranteed to work with an old engine! We shall version also the visualizations. The JavaScript parts of visualization shall check for capabilities of the Standard.Visualization module, shall avoid using newer features when running with old engine & co. Complex, but anyway its all about attitude:

If we care about compatibility, then let us say: we are going to write our JavaScript visualizations defensively, ready to work with older Standard.Visualization version since today (tomorrow or ...)!

[4e6]

If I open my project from last week and see that it is not working with the new IDE and I can't change that because the latest engine is always used, I will be infuriated.

We're still in the early stage of the project, shipping many features and bug fixes with every release. And currently, the situation is the opposite. When you download a new IDE, you can't use it with the old (pinned) engine version because of the changes made since.

First, if we want to have versioning, we should ensure that the user has a clear understanding of it, i.e.

• Make the user aware that the project they created has a version (currently they are not)
• If the project version is outdated, make the user aware of it and allow them to change the version (at least to the version of a new IDE they have downloaded)

Then, we should distinguish between the engine version and the language server version.

• The engine version includes the compiler and the std-lib. This is the version that goes to the project.yaml and which the user can control.
• The language server API version should be bound to the IDE version as they are shipped together and work together as one system. The language server API should be always new as it is shipped together with the new IDE release.

Having separate engine and language server versions can cause a situation when a new language server version will not support older engine versions. But that's ok, and instead of failing to open the project or experiencing random issues in IDE (which is happening now), the user will be aware of it and will be forced to switch to a new engine version (or to use older IDE versions).

[JaroslavTulach]

Having separate engine and language server versions can cause a situation when a new language server version will not support older engine versions.

How am I supposed to interpret this sentence, @4e6? Our build creates following distribution:

enso$ ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/

that distribution contains engine as well as language server implementation. E.g. engine version and language server implementation are tight together.

language server API version should be bound to the IDE version as they are shipped together

Language server API version should be a number specified inside of the specification document.

• then there is language server implementation (which also may have a version) which resides inside of enso-engine distribution and implements some version of the language server API
• the IDE just uses the language server API and shall specify the lowest API version it requires to function properly - IDE shall reject to work with engine/LS older than the lowest API version.

However the IDE will also work with all newer versions of the language server API, while they are marked as compatible with the lowest API version.

[4e6]

To summarize discussions that happened since the last message.

• instead of versioning the language server API, we want to always use the latest IDE + the latest language server
• language server will keep track of breaking changes of the internal engine API, and maintain the oldest supported engine version (edition)
• I'm not sure if editions are comparable (they are definitely not semantic). I assume that we don't want to touch the edition, so instead we can introduce a private semantic version that comes together with edition. This private version will be used by the language server to determine if the engine is supported or not.

[JaroslavTulach]

How will you identify "latest language server" when there is no versioning?

[JaroslavTulach]

How can "language server will keep track of breaking changes" when there is no version to begin with?

[4e6]

We can use the one that is shipped with IDE.