Che ignores changes to devfile.yaml and che-editor.yaml after pushing to a git repository (because che-server URLFetcher sometimes caches its results)

[l0rd]

Is your enhancement related to a problem? Please describe

I’m always frustrated when…Che uses the old version of a devfile.yaml, not the one I have modified a few seconds ago on github.com.

That can be reproduced starting a workspace from a GitHub repository, pushing a devfile.yaml change, deleting the existing workspace and starting a new one with the same exact factory URL and within a 5 minutes time frame.

The problem can also be reproduced test che-server /scm/resolver API using swagger: for 5 minutes, any new request will return the old devfile.yaml or che-editor.yaml, even if you have pushed some changes in the meantime.

Describe the solution you'd like

Che-sever /scm/resolver endpoint should return the updated file, not a cached version.

Release Notes Text

Che ignores changes to devfile.yaml and che-editor.yaml after pushing to a git repository (because che-server URLFetcher sometimes caches its results)

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[l0rd]

/remove-lifecycle stale

[vitaliy-guliy]

I have a feeling, that the URL fetcher works pretty fine and caching is performed on another layer. The bug constantly reproduced only for che-code. When I specify che-editor=eclipse/che-theia/latest in the factory URL, I get che-theia with up-to-date workspace.

In my case, I forked java-spring-petclinic sample and created a workspace from the fork. I played a bit with the sample, opened few files and tried to run and debug the sample.
Then I switched to the dashboard and deleted the workspace.
Next steps were cloning the java-spring-petclinic locally, adding a test command to print greeting message to the terminal and pushing changes.
When I created a wokspace from the repository I got a workspace with saved state of che-code. My test command was not appeared in the list of che- commands.

When I tried to create a workspace with che-theia editor, everything was fine.

Now I deleted the workspace and created it again with che-code editor.
I got a workspace with saved state of che-code, but my test command is appeared in the command list.

I want to notice here, that from time to time I got an updated list of commands and it's not clear on what it depends.

[l0rd]

@vitaliy-guliy using a different factory URL (with a different parameter) invalidates the cache. I am afraid that your assumption, that caching happens on another layer, is wrong.

[l0rd]

@vitaliy-guliy using a different factory URL (with a different parameter) invalidates the cache. I am afraid that your assumption, that caching happens on another layer, is wrong.

@vitaliy-guliy @svor any comment on that?

[svor]

Today i can't reproduce the problem with new or modified task and che-code:
screencast-nimbus-capture-2022.08.22-15_40_35.webm
but i still see the history of task execution which doesn't work in the current session

[vitaliy-guliy]

@vitaliy-guliy using a different factory URL (with a different parameter) invalidates the cache. I am afraid that your assumption, that caching happens on another layer, is wrong.

Why did I decide, that caching is performed on another layer? Considering the following..

1. create a fork of che-sample https://github.com/che-samples/java-spring-petclinic/tree/devfilev2 -> https://github.com/vitaliy-guliy/java-spring-petclinic/tree/devfilev2
2. create a workspace from the fork https://github.com/vitaliy-guliy/java-spring-petclinic/tree/devfilev2, check the workspace name, test maven build command
3. clone the fork locally, change the workspace name and maven build command, push changes vitaliy-guliy/java-spring-petclinic@0d9ac30
4. delete the workspace, create it again using the same URI as in [ 2 ]
5. check the workspace name and test maven build command.

Workspace name is changed, that means the devfile was re-read when workspace creation.

Che-code still have a dozen of opened files. With them I played a week ago, but from that time I removed and then created fork again (with modifying the devfile) about five times.

To run maven-build command, it needs to delete the che:maven-build menu item, as it seems it contains some cached data from first workspace.

[vitaliy-guliy]

Today i can't reproduce the problem with new or modified task and che-code: screencast-nimbus-capture-2022.08.22-15_40_35.webm but i still see the history of task execution which doesn't work in the current session

It seems you reproduced it:

1. you used the same URI to create both workspaces (am I right?)
2. you still have the history of task execution, which does not work.

[svor]

@vitaliy-guliy

you used the same URI to create both workspaces (am I right?)

yes, I used the same URI

you still have the history of task execution, which does not work.

true, but what I didn't reproduce is that the new task was added and an existing one was modified when I create the workspace for the second time

[vitaliy-guliy]

@vitaliy-guliy

you used the same URI to create both workspaces (am I right?)

yes, I used the same URI

you still have the history of task execution, which does not work.

true, but what I didn't reproduce is that the new task was added and an existing one was modified when I create the workspace for the second time

What do we have then?
The devfile was fetched from git reposiotry, the factory URI is not cached.

You got another workspace, but it had some cached data from previous launch.

[l0rd]

I am not able to reproduce the caching problem anymore with latest Che (but I can still reproduce it with Dev Spaces 3.1 so it should have been fixed after the release of Che 7.50).

Closing

[l0rd]

I am re-opening this issue since I had reproduced the problem a few times using both Che (nigthly) and Dev Spaces (3.1).

From a user point of view the problem can be reproduced easily:

1. Start a workspace from a repo with a devfile (e.g. using https://github.com/l0rd/go-hello-world)
2. Edit the devfile (add a new command for example), commit the changes and pushes to origin main.
3. Delete the existing workspace, starts a new one using the exact same factory URL and within 5 minutes of the creation of the first workspace: the resulting DevWorkspace won't include the changes (oc get dw -o json | jq 'items[].spec.template') although the devfile.yaml` has the changes.

This is a problem with the che-server URL fetcher. It can be verified opening swagger and testing the /scm/resolve endpoint. Trying with a GitHub repository, the cache expires after 5 minutes, so the response doesn't change although the devfile has been modified.

Looking at the code we are creating a URLConnection without specifying setUseCache(false) and a github.com HTTP response will be considered fresh for 5 minutes (that's because github.com returns cache-control: max-age=300 in its responses header).

[vitaliy-guliy]

I would like to split this issue on two subtasks.

---

1. Changes in the devfile are not taken into account

Changes in the devfile are not being applied when re-create a workspace with exact factory URL (@l0rd described a steps to reproduce in the previous comment)

To get the content of the devfile che-server uses https://raw.githubusercontent.com. When the user push changes, the raw content is updated after 5 minutes. GitHub does that to serve large numbers of requests.

I tried to fetch raw content of the devfile with curl, tried to change / remove cache-control and if-none-match headers, but nothing worked.
The only solution that I found, is when referring the devfile with https://raw.githubusercontent.com, we need to use hash of the commit instead of the branch name.

Form URL like this

https://raw.githubusercontent.com/vitaliy-guliy/web-nodejs-sample/dbb0cf604550ddbf6f2e592f03bff01bb6ace207/devfile.yaml

Instead of

https://raw.githubusercontent.com/vitaliy-guliy/web-nodejs-sample/devfilev2-test/devfile.yaml

To get the last commit of the devfile che-server has to send a sequence of requests to GitHub API. And only then apply this commit ID to form the URL to get the raw content. I wonder here how it will work with oauth.

---

2. Keeping the editor state ( che-code only )

Che-code saves its state to the browser storage (che-theia stores to /home/theia on the file system).
Such unique identifiers as workspace ID or browser URI are not used to recognize for which workspace the state belong.
Che-code uses browser storage, represented as key-value database, to keep one state for all workspaces.

This leads to have a mess.

For instance, user launched a task, that he described in his devfile.
This task is added to the list of recently used tasks and always will be displayed on top when picking a task to run.
Then user removed this task from his devfile and re-create a workspace.
Now he can still run the task as the list of recently used tasks is persisted.

We have to decide what to do with this caching. As an option - clear the browser storage if the user is opened the workspace with different ID. But in this case it will not be possible to switch between workspaces with keeping the state.

Link with https://issues.redhat.com/browse/CRW-3421

[devstudio-release]

sync'd to Red Hat JIRA https://issues.redhat.com/browse/CRW-3562