refactor: move projects list into its own yaml file, fixes #5639

[GuySartorelli]

The Issue

• Project list should not be in global_config.yaml, move to ~/.ddev/projects.yaml #5639

How This PR Solves The Issue

Moves the projects list into its own ~/.ddev/project_list.yaml file.

For now this is a fairly naive approach - I'd probably lean towards calling the file ~/.ddev/project_list.yaml, and renaming all of the GlobalProject* that I've added to ProjectList, i.e.

• GetGlobalProjectsPath => GetProjectListPath
• DdevGlobalProjects => DdevProjectList
• DdevGlobalProjectName => DdevProjectListFileName
  etc. If maintainers agree, I'll make those changes.

I'll admit I don't fully know what that list is used for - but running this locally everything seems to work as expected. I'll look at updating tests once maintainers have let me know if this is the right direction, and any general implementation changes they want.

Manual Testing Instructions

I'm honestly not sure what this list is for so I'm not sure what manual testing will actually show if it's working as expected.

Creating a new project did add it to the new file, .gotmp/bin/linux_amd64/ddev list worked as expected, and deleting the project removed it from the list.

Automated Testing Overview

To be done once I know if this is the right direction.

Related Issue Link(s)

#5639

Release/Deployment Notes

The source of truth for the project list remains in ~/.ddev/global_config.yaml.
This way, we ensure that this change is not breaking and people will be able to move from one version of DDEV to another without losing the project list.

Some time after a few new releases, when the vast majority of people have this new project file, project_list can be easily removed from ~/.ddev/global_config.yaml, leaving ~/.ddev/project_list.yaml as the only source.

[github-actions]

Download the artifacts for this pull request:

• all-ddev-executables.zip
• ddev-linux-amd64.zip
• ddev-linux-arm64.zip
• ddev-macos-amd64.zip
• ddev-macos-arm64.zip
• ddev-windows-amd64-installer.zip

See Testing a PR

[stasadev]

This is a tricky task because we also need to allow people to downgrade without losing the project list.

What if DDEV keeps the project list in two files for now and reads it from projects.yaml, and in some future version we remove project_info from global_config.yaml 🤔

As for the function and variable names, I agree that Global should be removed.

I'm honestly not sure what this list is for so I'm not sure what manual testing will actually show if it's working as expected.

It is used in many places, for example, it allows you to execute ddev start project-name from any folder - so it knows that you have a project-name (from the global configuration file).

[GuySartorelli]

What if DDEV keeps the project list in two files for now and reads it from projects.yaml, and in some future version we remove project_info from global_config.yaml 🤔

Sounds good to me, that should be pretty straight forward. The fact that most of the test runs are passing already is a good sign that we won't end up missing any reads.

I am a little worried about the two (non-doc) failures though - I'm not sure what is causing those failures, and I'm hesitant to make the swap to keeping the list in both places until we know everything will be using the new file for reading without causing problems. Could you please help me understand what's causing those failures?

[stasadev]

I am a little worried about the two (non-doc) failures though

Colima fails all the time, restarted the test.

The panic in ddev-macos-amd64-mutagen looks scary, we had some issues with Docker Desktop in this runner, it may not be related to your changes, restarted the test.

[GuySartorelli]

Awesome, I've updated the variables and file name and changed the logic to write to both global config and the new file.

Here's a scenario we don't currently handle:

1. When upgrading, if there are any projects in project_info in global_config.yaml they will not be added to the new file, because we're not reading the project info from that file anymore.
2. As soon as the project list is written to the new file for the first time, the data from project_info will be overwritten.

That's probably not ideal behaviour - my current thinking is in (or after) ReadGlobalProjects() we should check DdevGlobalConfig.ProjectList - if that list does not match what we just read from the new projects list file, we can assume that we've just upgraded from an earlier version and that the global config content is correct.

Does that sound about right?

Also, do you want any new tests for this? If so, it'll be my first time properly writing new tests (until now I've just copied existing ones that were very similar to what I needed) so I'll need some guidance for where to put them and what sort of thing you want to be tested.

[stasadev]

That's probably not ideal behaviour - my current thinking is in (or after) ReadGlobalProjects() we should check DdevGlobalConfig.ProjectList - if that list does not match what we just read from the new projects list file, we can assume that we've just upgraded from an earlier version and that the global config content is correct.

Sound right for me.

Also, do you want any new tests for this? If so, it'll be my first time properly writing new tests (until now I've just copied existing ones that were very similar to what I needed) so I'll need some guidance for where to put them and what sort of thing you want to be tested.

I've only written a few tests myself 🙂, looking for advice from @rfay here.

[rfay]

What I see in tests seems like it will be fine, but I won't be able to look at this carefully until probably February.

[GuySartorelli]

That's probably not ideal behaviour - my current thinking is in (or after) ReadGlobalProjects() we should check DdevGlobalConfig.ProjectList - if that list does not match what we just read from the new projects list file, we can assume that we've just upgraded from an earlier version and that the global config content is correct.

Sound right for me.

Sweet, I've done that. There might be a better way to compare maps - if so let me know and I'll make the change.

What I see in tests seems like it will be fine, but I won't be able to look at this carefully until probably February.

Sweet, no worries

[rfay]

I wasn't implying that it needs to wait for my attention. Stas is fully authorized to move forward on anything when he's comfortable with it.

[stasadev]

Rebased because the artifact bot comment links were invalid (see #5662).

[stasadev]

Tested, and confirmed that ddev config writes data to both files.

One more important issue, when you switch the binary and run ddev list and ddev start everything looks good.

But when you don't have ~/.ddev/project_list.yaml yet, replace the binary, and run, for example, ddev version, project_info is lost in ~/.ddev/global_config.yaml.

[GuySartorelli]

Good catch - I didn't notice that 'cause the ddev list command still shows those existing projects even though they're not in the file anymore (not sure where it's getting the info from...)

Should be fixed now.

[stasadev]

It looks great!

I've updated the Release/Deployment Notes in the first post.

[rfay]

DDEV inspects existing container labels for ddev list in addition to using the global config for this. So that's where things are coming from.

[stasadev]

Thank you for all the hard work on this, merging.

[rpkoller]

awesome thank you for all the effort! already tested and i can confirm that right after the update ~/.ddev/project_list.yaml was created. with the identical number of projects listed in the global_config.yaml. at first i wanted to raise a question because of the redundancy between the two yaml files, but note to myself i should read the Release/Deployment Notes first properly ;) therefore instead of a question this confirmation that everything works and a thank you ;)