Project status / update UI improvements

[vfried]

Description

The project status / update UI needs to be made more robust and better handle all the cases.

Implementation

Pre-req: Improve the information returned by the check_migrations endpoint. See SwissDataScienceCenter/renku-python#2048.

The new information should be structured like:

{
	"result": {
		"project_supported": true,
                 "core_renku_version": "0.14.2",
                "project_renku_version": "0.14.0", //latest version of renku used by a user on the project
		"core_compatibility_status": {
			"project_metadata_version": "7",
                        "current_metadata_version": "7"
			"migration_required": false,
		},
		"dockerfile_renku_status": {
			"newer_renku_available": true,      // docker_update_possible -> newer_renku_available
			"automated_dockerfile_update": true,  // new field
			"latest_renku_version": "0.14.2",
			"dockerfile_renku_version": "0.14.0",
		},
		"template_status": {
			"automated_template_update": false,
			"newer_template_available": true,    // template_update_possible -> newer_template_available
			"template_source": "https://github.com/SwissDataScienceCenter/renku-project-template",
			"project_template_version": "73db625bf5d9effa1d76a4e5e1fcbcb1a927f654",  // current_template_version -> project_template_version (ideally semver)
			"latest_template_version": "817b03c0f434a79eb54473688c1f897a66228b3d",		
			"template_ref": "master",
			"template_id": "python-minimal"
		}
	}
}

Given that information, the UI should be restructured a bit. Currently we have two sections: Renku Version and Template Version (ignoring KG version, which is fine as is). This should become 3 sections.

RenkuLab UI Compatibility

if (core_compatibility_status.migration_required) {
	show
		Datasets can be created and modified from the RenkuLab UI.
} else {
	show
		Datasets cannot be created and modified from the RenkuLab UI. Please upgrade the project to take advantage of these UI features.
}

Renku Version

if (!dockerfile_renku_status.newer_renku_available) {
	show 
		Project version ${dockerfile_renku_status.project_version}
		This project is using the latest version of renku.
} else {
	show 
		Project version ${dockerfile_renku_status.project_version}
		Latest version ${dockerfile_renku_status.latest_version}
		A newer version of Renku is available. See the [release notes] to learn more about improvements and new features.
		
	if (core_compatibility_status.migration_required) {
		An upgrade is required to create and modify datasets from the RenkuLab UI.
	}
	
	if (dockerfile_renku_status.automated_dockerfile_update) {
		[Update] [Do you prefer manual instructions?]
	} else {
		The Renku version cannot be automatically updated. (manual instructions)
	}
}

Project Template

if (template_status.template_id == null) {
	show
		This project does not use a versioned template.
} else {
	show 
		Template id ${template_status.template_id}
		Template ref ${template_status.template_ref}
		Template source <a href={template_status.template_source}>{template_status.template_source}</a>
		
	if (template_status.newer_template_available) {
		A newer version of this template is available. Please see the <a href={template_status.template_source}>template source</a> to 
		learn more about the changes to the template.
		
		if (template_status.automated_template_update) {
			[Update] [Do you prefer manual instructions?]
		} else {
			The template version cannot be automatically updated. (manual instructions)
		}		
	}
}

Table

		MIGRATION STATUS	ACTION/MESSAGE
1	migration_required T project_supported T newer_renku_available T automated_dockerfile_update T	NEEDED	AUTOMATIC MIGRATION (DOCKER + METADATA)
2	migration_required T project_supported T newer_renku_available T automated_dockerfile_update F	NEEDED	MANUAL UPGRADE
3	migration_required T project_supported T newer_renku_available F automated_dockerfile_update T	NEEDED	AUTOMATIC METADATA MIGRATION + DOCKERFILE UP TO DATE
4	migration_required T project_supported T newer_renku_available F automated_dockerfile_update F	NEEDED	AUTOMATIC METADATA MIGRATION + DOCKERFILE UP TO DATE
5	migration_required T project_supported F newer_renku_available T automated_dockerfile_update T	NOT POSSIBLE	MANUAL METADATA MIGRATION + AUTOMATIC DOCKERFILE UPGRADE
6	migration_required T project_supported F newer_renku_available T automated_dockerfile_update F	NOT POSSIBLE	MANUAL METADATA AND DOCKERFILE UPGRADE
7	migration_required T project_supported F newer_renku_available F automated_dockerfile_update T	NOT POSSIBLE	MANUAL METADATA MIGRATION + DOCKERFILE UP TO DATE
8	migration_required T project_supported F newer_renku_available F automated_dockerfile_update F	NEEDED	MANUAL METADATA MIGRATION + DOCKERFILE IN UNKNOWN STATE
9	migration_required F project_supported T newer_renku_available T automated_dockerfile_update T	OPTIONAL	METADATA UP TO DATE + AUTOMATIC DOCKERFILE UPDATE
10	migration_required F project_supported T newer_renku_available T automated_dockerfile_update F	OPTIONAL	METADATA UP TO DATE + MANUAL DOCKERFILE UPGRADE
11	migration_required F project_supported T newer_renku_available F automated_dockerfile_update T	NO NEED	METADATA UP TO DATE + DOCKERFILE UP TO DATE
12	migration_required F project_supported T newer_renku_available F automated_dockerfile_update F	NO NEED	METADATA UP TO DATE + DOCKERFILE UNKNOWN SINCE IT’S NOT COMPATIBLE
13	migration_required F project_supported F newer_renku_available T automated_dockerfile_update T	NOT POSSIBLE	METADATA IS AHEAD OF DEPLOYMENT + DOCKERFILE AUTOMATIC MIGRATION
14	migration_required F project_supported F newer_renku_available T automated_dockerfile_update F	NOT POSSIBLE	METADATA IS AHEAD OF DEPLOYMENT + MANUAL DOCKERFILE UPDATE
15	migration_required F project_supported F newer_renku_available F automated_dockerfile_update T	NOT POSSIBLE	METADATA AHEAD OF DEPLOYMENT + DOCKERFILE UP TO DATE
16	migration_required F project_supported F newer_renku_available F automated_dockerfile_update F	NOT POSSIBLE	METADATA AHEAD OF DEPLOYMENT + DOCKERFILE UINKNOWN SINCE IT’S NOT COMPATIBLE

Status

• Step 0: Check that the RP issue behaves as we expect, give feedback (sprint 2021-06-04)
• Step 1: Update the message without a link to the changelog
• Step 2: Design change to API to receive link to the changelog
• Step 3: Add a link to the changelog once the core-service returns it

Historical

Two situations can happen:

• There is a newer version of renku-python, but the metadata is compatible
• There is a newer version of renku-python, but the Dockerfile does not support automatic updating

The first is explained below, the second still needs a design.

Details

In a project like this:
https://virginia.dev.renku.ch/projects/virginiafriedrich/my-test-project-vir/overview/status

We see a message like this:

Mockup

2021-03-30 Update

From @Panaetius:

So you now get back

automated_template_update = fields.Boolean()
template_source = fields.String()
template_ref = fields.String()
template_id = fields.String()

on the migrations_check endpoint. automated_template_update is set on a template and determines if that template supports updating or not. if it's false, the service will only update if passed the "force_template_update" flag.

template_source = repository or "renku" if it was created with the CLI
template_ref = branch/tag used in the repo or empty
template_id = name of the template, like "python-minimal"

[vfried]

TO CONSIDER WHEN WORKING...

[rokroskar]

In the case outlined above, the result of clicking "update" will be that the line

ARG RENKU_VERSION=<version>

in the Dockerfile will be changed to whatever version of renku the core service is running. in this case, the Dockerfile itself is actually not being updated, as the proposed message implies - with this message ("A new version of the Dockerfile is available") I would expect that the structure of the Dockerfile itself would change - that would happen as a result of a template update only. So maybe a better message would be "The version of Renku installed in your interactive sessions can be automatically updated." or something along those lines?

[ciyer]

@rokroskar

I do not completely understand the information being returned from the core service and how that squares with the requested message.

Here is the response from the core service:

{
  project_version: "0.12.2",
  latest_version: "0.12.2",
  project_supported: true,
  migration_required: false,
  docker_update_possible: true
}

I do not see how that leads to a message: "The version of Renku installed in your interactive sessions can be automatically updated." Given the information in the response, it looks like the version of renku is current. Or am I misunderstanding the response?

[vfried]

@ciyer This is related to the conversation on top, what needs to be updated, or can be updated here is the renku version inside the dockerfile. This is why docker_update_possible is true

migration_required is related to the metadata i think.

[rokroskar]

@ciyer there are two renku versions - one which was used to commit (meta)data to git and another which is installed in the docker image. These are usually the same but not always, because the user may have run renku commands outside of the docker container that was built for the project. In the case of the response above, my guess is that they are not the same, i.e. the Dockerfile has something like

ARG RENKU_VERSION=0.12.0

My point is just that in this case, the update will only change this single line and not the Dockerfile itself and the message should reflect that.

[pameladelgado]

for the case where the Dockerfile does not support automatic updating, the ideal would be to inform the user about this (regardless of the Renku version) and give/point to clear instructions about how to enable this manually.

[Panaetius]

Feedback for the table of different combinations above:

project_supported means, well, the project is not supported by the service if it's false. I.e. the project was made/migrated with a newer version of renku-python than is on the core-service and there is nothing we can do with it. So all those rows in the table with project_supported F can be conflated into one: "The Service can't work with this project and there's nothing you can do about it other than wait for the deployment to catch up". We don't support downgrades, and there's no realistic way for users to manually downgrade metadata, MANUAL METADATA MIGRATION isn't a thing. I doubt even a renku-python developer could manually migrate metadata except for in the most trivial cases.

Case 2 + case 10: This can't happen, if the Dockerfile doesn't support automated updating (i.e. it lacks the ARG RENKU_VERSION=... line, then we can't know what the version of renku in the Dockerfile is, and, by extension, can't know whether a newer one is available.

Case 4 + case 12: In a similar vein, we don't know if the Dockerfile is up to date, just that it can't be updated

I think a flowchart would be more readable:
https://drive.google.com/file/d/1yue3Znz6f42lkzgiL0wwYtR4RXkXCeVM/view?usp=sharing

Green = required migration
Blue = optional
Orange = Manual step to be done by user
Red = Nothing can be done

[ciyer]

I think in the UI, we will have 3 sections. Here are the mock-ups for how the 3 sections should look in the different situations.

RenkuLab UI Compatability

Renku Version

Template Version

[Panaetius]

That looks great. I'm still not sure if a manual template update is possible. We keep checksums of the files as they were in the template when the project was created/last updated. Then we can do a three-way check of the checksums per file, to see if a file changed in the template and/or if a user changed a template file, and then act accordingly (overwriting with new version or leaving it as is if it was edited, with some additional logic to handle files that are marked as "User should not change these" on the template).

Now a user could set the checksums to:

• checksums of what he has currently in the repo --> template update would overwrite all his files
• checksums of what is currently in the template repo --> nothing would get updated, only if there's new changes in the template repo
• all 0's -->nothing would get updated
• decide for each file which checksum to set, either what's in the template repo now or 000 --> I guess this could work? But sounds really tedious.

A user might be able to checkout the original renku-init commit and get checksums for files in that state, but that wouldn't account for the case where renku-init was run on top of an existing repository.
We could also try to add a command that does this automatically but there's so many edge cases/repos in different states when templates worked differently that I'm not sure we could make that robust

Plus a user would have to know how to calculate sha256 hashes of files.

[ciyer]

I see. So if there is a newer template, but automated template update is not possible, we will show the message:

Automated update is not possible; [contact the template maintainer](link to template repo) to find out how to upgrade.

[Panaetius]

I've updated the original post a bit.

• removed service from the name of return values. We also return this information in the CLI so naming a field core-service doesn't make sense there
• core_compatibility_status entries return the metadata version, not the renku-python version, as that's the only version relevant for migrations. This is a plain integer (currently 8).
• latest version of renku-python that was used in a command on the project is returned in the toplevel project_renku_version
• project_renku_version renamed to dockerfile_renku_version in the dockerfile response values, to be less ambiguous
  Implemented in feat(service): improve formatting for migrationscheck response renku-python#2122

[ciyer]

Fixed in #1422