Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apis: west.manifest.Manifest.projects should not include the manifest itself #327

Open
mbolivar opened this issue Oct 21, 2019 · 8 comments
Labels
API west's library APIs

Comments

@mbolivar
Copy link
Contributor

Trying to treat the manifest repository as an ordinary Project was doomed to failure; they're simply too different. We should instead move the valid state that's in ManifestProject today to Manifest attributes, removing MANIFEST_PROJECT_INDEX and the fact that projects[MANIFEST_PROJECT_INDEX] contains the manifest.

This will require a bit of thought into how west list should behave, though. Maybe we need to resurrect west list --all to include manifest project data like {path} and {west_commands}? And perhaps that should abort if you west list --all -f {url} or try to access some other attribute that's not defined for the manifest repository.

@mbolivar mbolivar added the API west's library APIs label Oct 21, 2019
@tejlmand
Copy link
Collaborator

tejlmand commented Oct 23, 2019

Trying to treat the manifest repository as an ordinary Project was doomed to failure

what kind of issues are observed with the current behavior.
Could you try to list the issues you experience when the manifest repo is treated as an ordinary project.

Cause I remember there were long discussion back and forth on how to treat the manifest project.

Note, I'm not advocating against changing it, just would like to see a list of issues experienced so that everyone looking at this issue know why to change API.

@mbolivar
Copy link
Contributor Author

Welcome back :).

The manifest repository is a repository for sure, and its Python representation should have some Project attributes (like west_commands, etc.), but it's not truly a Project:

  • it doesn't have a revision
  • or a url
  • or, in some cases (when parsing a manifest from data without self: path: foo), a path

So by "not truly a Project", I mean this in a Liskov substitution principle sense.

Right now we hack it by setting these fields to None, but I've noticed again and again we've messed up updating the docstrings or code in ManifestProject following changes to Project, leading to errors, which is also a bad sign.

From personal experience, I also find myself writing downstream code (nrf/scripts/ncs_west_commands/ncs_commands.py) that wants to iterate over projects and ending up special-casing the manifest repository as a result of crashes on code that naively looped over them. The manifest repository doesn't really have a 'name' in the project sense, and trying to treat it as if it were a project has led to weird edge cases.

I think it was a worthy attempt and what we did made sense at the time, but experience has shown it was not a good idea in retrospect.

@mbolivar
Copy link
Contributor Author

mbolivar commented Oct 23, 2019

leading to errors, which is also a bad sign.

For example, west forall -h still says that the projects list defaults to "all projects in the manifest", which is wrong: it's actually all projects in the manifest.... plus the manifest repository, because that's not filtered out.

@tejlmand
Copy link
Collaborator

tejlmand commented Oct 30, 2019

Welcome back :).

Thanks, hope I can still keep a little up to date on west ;-)

The manifest repository is a repository for sure, and its Python representation should have some Project attributes (like west_commands, etc.), but it's not truly a Project:

I agree that it doesn't contain all the project attributes.
But thinking further ahead in time, the question is whether we should treat manifest project separately, or if it should be considered that a project may not have all those attributes.

Take a look at: #243

Will those kinds of projects be "true projects" in current sense of true project.
Those projects may not have all attributes set, just as the manifest project.

We should avoid ending in a situation where changing this API results in treating other projects types individually, such as those proposed in #243 .

that wants to iterate over projects and ending up special-casing the manifest repository as a result of crashes on code that naively looped over them

Should we consider to be able to return an iterator for projects meeting a special criteria.

  • revision != none
  • project type == git | svn | tarball other (in future)
  • other

@tejlmand
Copy link
Collaborator

leading to errors, which is also a bad sign.

For example, west forall -h still says that the projects list defaults to "all projects in the manifest", which is wrong: it's actually all projects in the manifest.... plus the manifest repository, because that's not filtered out.

and such issues can happen regardless of what we do - and should have been fixed a long time ago :(

@mbolivar
Copy link
Contributor Author

mbolivar commented Nov 1, 2019

Thanks, hope I can still keep a little up to date on west ;-)

I'm very glad!

Take a look at: #243

Good point. In this case I think as long as the non-git projects show up in manifest: projects: in the manifest file we should be able to keep the model consistent.

In fact, making the change here (#327) may lead us to add another section besides projects: for fetching some non-git "things", e.g. large files, if they aren't cleanly "repositories of some kind with a name, URL, revision, path, (maybe) clone depth, and optional extension commands".

That definition for what a "project" is was how I modeled a project in the multimanifest work I am doing and I think it's nice and general for hg, svn, p5, etc.

Clone depth we copied from google repo, and various issues (e.g. #307, #319) show it's not thought through properly.

and such issues can happen regardless of what we do - and should have been fixed a long time ago :(

Right, but by requiring explicit inclusion of the manifest repository by having to say manifest.foo instead of getting it out of manifest.projects automatically we make them less likely in the future.

So I believe making this change means the API is easier to use correctly if manifest.projects corresponds exactly to, well, manifest: projects: in west.yml, instead of mixing in manifest: self: content.

@tejlmand
Copy link
Collaborator

⚠️ In case you start doing some work on this, remember that Zephyr modules uses west list to obtain a list of ALL projects, as those are candidates for being a Zephyr module:
https://github.com/zephyrproject-rtos/zephyr/blob/51fc680ba225c2df5f3fca9064839b825a9c0511/scripts/zephyr_module.py#L177

and in some cases the manifest project is ALSO a Zephyr module, and it would be bad to have west break backward compatibility to projects based on older Zephyr revisions.
I would prefer a new west to also work if i checkout an old version of NCS, as example.

@mbolivar
Copy link
Contributor Author

For sure, west list should continue to contain the manifest repository. It's just the internal API I want to change.

mbolivar-nordic added a commit to mbolivar-nordic/west that referenced this issue Jun 10, 2020
Change the url, revision, and clone_depth properties to plain old
attributes, changing url from None to the empty string. This will be
used to make it possible to type annotate Project in a subsequent
commit.

This seems like even more evidence that ManifestProject is not a
Project in a subtyping sense, should not have been treated like one,
and should be removed (zephyrproject-rtos#327).

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
mbolivar-nordic added a commit that referenced this issue Jun 24, 2020
Change the url, revision, and clone_depth properties to plain old
attributes, changing url from None to the empty string. This will be
used to make it possible to type annotate Project in a subsequent
commit.

This seems like even more evidence that ManifestProject is not a
Project in a subtyping sense, should not have been treated like one,
and should be removed (#327).

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
mbolivar-nordic added a commit to mbolivar-nordic/west that referenced this issue Aug 14, 2020
Clean up the file based on what zephyr's pylintrc would have
complained about. There's one remaining issue, which is that
ManifestProject doesn't call Project.__init__(), but that should be
fixed with zephyrproject-rtos#327 instead.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
mbolivar-nordic added a commit that referenced this issue Aug 14, 2020
Clean up the file based on what zephyr's pylintrc would have
complained about. There's one remaining issue, which is that
ManifestProject doesn't call Project.__init__(), but that should be
fixed with #327 instead.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
mbolivar-nordic added a commit to mbolivar-nordic/west that referenced this issue Mar 19, 2022
This is an API break. We are still allowed to do that until west 1.0.

The west.manifest.Manifest class's path handling has become
unmaintainable and we need to overhaul it. The from_data() and
from_file() constructors are too clever and try to allow too many use
cases, which results in the guts of the manifest construction code
often not being able to tell whether it's in a real workspace or not.

This makes it difficult to know when we can safely read configuration
files or not. That in turn makes it difficult to accurately
distinguish between the 'manifest.path' value in the local configuration
file and the 'self: path:' value in the manifest data, especially when
we can take keyword arguments that say "hey, pretend this is your
path".

Additionally, there are a few assumptions in the code that pretend
that the manifest file itself always lives in the top level directory
of the manifest repository. That was a valid assumption up to the
point the 'manifest.file' configuration variable was introduced, but
it's no longer valid anymore, since 'manifest.file' can point to a
subdirectory of the manifest repository. This leaves us with some
hacks to find the git repository we live in that shouldn't be
necessary and which we can now remove with this overhaul.

Rework the whole thing so that we can correctly handle the following
situations:

 - when running in a workspace, you should now use from_file() or
   a newly introduced from_topdir() to make your Manifest

 - when running outside of any workspace, use from_data()

From now on, we forbid creating a manifest from data "as if" it were
in a real workspace. If all you have is data, the abspath attributes
will all be None, reflecting reality.

Accordingly, rework the Manifest.__init__() method to either take a
topdir or a bit of data, but not both. Additionally, now that we have
both manifest.path and manifest.file configuration options, it's not
usually the right thing to ask for a Manifest from a *file*: what you
really usually want is a Manifest from a *workspace*, and for that you
just need a topdir. From the topdir, you can create a
west.configuration.Configuration, and from there you can read
manifest.path and manifest.file to get the complete path to the top
level manifest within its repository. For that reason, we remove the
source_file keyword argument from __init__() in favor of topdir and a
new 'config' argument.

For backwards compatibility, it's still allowed to use from_file() to
load another manifest file than the one pointed to by
topdir/manifest.path/manifest.file. However, this handling code is now
just a bit of special case trickery in from_file(), and it also
introduces a restriction that the other file is still in a git
repository in the workspace. This moves potential sources for error or
confusion out of Manifest.__init__() and the functions it calls. (This
has proven to be a useful feature in practice; I am aware of west
extensions that use it to compare the parsed contents of two different
manifest files within the same workspace.)

To migrate away from the now-outdated from_file(), introduce a new
from_topdir() factory method and use it throughout the tree. This is
clearer and more accurate code, which always avoids the hacks left
behind for compatibility in from_file().

Finally, add various new instance attributes to the Manifest class
that will let us tell the YAML path from the actual path, and the path
to the manifest file from the path to the manifest repository in the
workspace. These in turn let us remove some hacky code from the 'west
list' implementation, and allow us to rework our internals so that
ManifestProject is just a legacy remnant at this point, further
helping along with zephyrproject-rtos#327.

Keep tests up to date, removing obsolete code as needed and updating
cases to reflect API breakage.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
mbolivar-nordic added a commit to mbolivar-nordic/west that referenced this issue Mar 19, 2022
This is an API break. We are still allowed to do that until west 1.0.

The west.manifest.Manifest class's path handling has become
unmaintainable and we need to overhaul it. The from_data() and
from_file() constructors are too clever and try to allow too many use
cases, which results in the guts of the manifest construction code
often not being able to tell whether it's in a real workspace or not.

This makes it difficult to know when we can safely read configuration
files or not. That in turn makes it difficult to accurately
distinguish between the 'manifest.path' value in the local configuration
file and the 'self: path:' value in the manifest data, especially when
we can take keyword arguments that say "hey, pretend this is your
path".

Additionally, there are a few assumptions in the code that pretend
that the manifest file itself always lives in the top level directory
of the manifest repository. That was a valid assumption up to the
point the 'manifest.file' configuration variable was introduced, but
it's no longer valid anymore, since 'manifest.file' can point to a
subdirectory of the manifest repository. This leaves us with some
hacks to find the git repository we live in that shouldn't be
necessary and which we can now remove with this overhaul.

Rework the whole thing so that we can correctly handle the following
situations:

 - when running in a workspace, you should now use from_file() or
   a newly introduced from_topdir() to make your Manifest

 - when running outside of any workspace, use from_data()

From now on, we forbid creating a manifest from data "as if" it were
in a real workspace. If all you have is data, the abspath attributes
will all be None, reflecting reality.

Accordingly, rework the Manifest.__init__() method to either take a
topdir or a bit of data, but not both. Additionally, now that we have
both manifest.path and manifest.file configuration options, it's not
usually the right thing to ask for a Manifest from a *file*: what you
really usually want is a Manifest from a *workspace*, and for that you
just need a topdir. From the topdir, you can create a
west.configuration.Configuration, and from there you can read
manifest.path and manifest.file to get the complete path to the top
level manifest within its repository. For that reason, we remove the
source_file keyword argument from __init__() in favor of topdir and a
new 'config' argument.

For backwards compatibility, it's still allowed to use from_file() to
load another manifest file than the one pointed to by
topdir/manifest.path/manifest.file. However, this handling code is now
just a bit of special case trickery in from_file(), and it also
introduces a restriction that the other file is still in a git
repository in the workspace. This moves potential sources for error or
confusion out of Manifest.__init__() and the functions it calls. (This
has proven to be a useful feature in practice; I am aware of west
extensions that use it to compare the parsed contents of two different
manifest files within the same workspace.)

To migrate away from the now-outdated from_file(), introduce a new
from_topdir() factory method and use it throughout the tree. This is
clearer and more accurate code, which always avoids the hacks left
behind for compatibility in from_file().

Finally, add various new instance attributes to the Manifest class
that will let us tell the YAML path from the actual path, and the path
to the manifest file from the path to the manifest repository in the
workspace. These in turn let us remove some hacky code from the 'west
list' implementation, and allow us to rework our internals so that
ManifestProject is just a legacy remnant at this point, further
helping along with zephyrproject-rtos#327.

Keep tests up to date, removing obsolete code as needed and updating
cases to reflect API breakage.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
mbolivar-nordic added a commit to mbolivar-nordic/west that referenced this issue Mar 19, 2022
This is an API break. We are still allowed to do that until west 1.0.

The west.manifest.Manifest class's path handling has become
unmaintainable and we need to overhaul it. The from_data() and
from_file() constructors are too clever and try to allow too many use
cases, which results in the guts of the manifest construction code
often not being able to tell whether it's in a real workspace or not.

This makes it difficult to know when we can safely read configuration
files or not. That in turn makes it difficult to accurately
distinguish between the 'manifest.path' value in the local configuration
file and the 'self: path:' value in the manifest data, especially when
we can take keyword arguments that say "hey, pretend this is your
path".

Additionally, there are a few assumptions in the code that pretend
that the manifest file itself always lives in the top level directory
of the manifest repository. That was a valid assumption up to the
point the 'manifest.file' configuration variable was introduced, but
it's no longer valid anymore, since 'manifest.file' can point to a
subdirectory of the manifest repository. This leaves us with some
hacks to find the git repository we live in that shouldn't be
necessary and which we can now remove with this overhaul.

Rework the whole thing so that we can correctly handle the following
situations:

 - when running in a workspace, you should now use from_file() or
   a newly introduced from_topdir() to make your Manifest

 - when running outside of any workspace, use from_data()

From now on, we forbid creating a manifest from data "as if" it were
in a real workspace. If all you have is data, the abspath attributes
will all be None, reflecting reality.

Accordingly, rework the Manifest.__init__() method to either take a
topdir or a bit of data, but not both. Additionally, now that we have
both manifest.path and manifest.file configuration options, it's not
usually the right thing to ask for a Manifest from a *file*: what you
really usually want is a Manifest from a *workspace*, and for that you
just need a topdir. From the topdir, you can create a
west.configuration.Configuration, and from there you can read
manifest.path and manifest.file to get the complete path to the top
level manifest within its repository. For that reason, we remove the
source_file keyword argument from __init__() in favor of topdir and a
new 'config' argument.

For backwards compatibility, it's still allowed to use from_file() to
load another manifest file than the one pointed to by
topdir/manifest.path/manifest.file. However, this handling code is now
just a bit of special case trickery in from_file(), and it also
introduces a restriction that the other file is still in a git
repository in the workspace. This moves potential sources for error or
confusion out of Manifest.__init__() and the functions it calls. (This
has proven to be a useful feature in practice; I am aware of west
extensions that use it to compare the parsed contents of two different
manifest files within the same workspace.)

To migrate away from the now-outdated from_file(), introduce a new
from_topdir() factory method and use it throughout the tree. This is
clearer and more accurate code, which always avoids the hacks left
behind for compatibility in from_file().

Finally, add various new instance attributes to the Manifest class
that will let us tell the YAML path from the actual path, and the path
to the manifest file from the path to the manifest repository in the
workspace. These in turn let us remove some hacky code from the 'west
list' implementation, and allow us to rework our internals so that
ManifestProject is just a legacy remnant at this point, further
helping along with zephyrproject-rtos#327.

Keep tests up to date, removing obsolete code as needed and updating
cases to reflect API breakage.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
mbolivar-nordic added a commit to mbolivar-nordic/west that referenced this issue Mar 19, 2022
This is an API break. We are still allowed to do that until west 1.0.

The west.manifest.Manifest class's path handling has become
unmaintainable and we need to overhaul it. The from_data() and
from_file() constructors are too clever and try to allow too many use
cases, which results in the guts of the manifest construction code
often not being able to tell whether it's in a real workspace or not.

This makes it difficult to know when we can safely read configuration
files or not. That in turn makes it difficult to accurately
distinguish between the 'manifest.path' value in the local configuration
file and the 'self: path:' value in the manifest data, especially when
we can take keyword arguments that say "hey, pretend this is your
path".

Additionally, there are a few assumptions in the code that pretend
that the manifest file itself always lives in the top level directory
of the manifest repository. That was a valid assumption up to the
point the 'manifest.file' configuration variable was introduced, but
it's no longer valid anymore, since 'manifest.file' can point to a
subdirectory of the manifest repository. This leaves us with some
hacks to find the git repository we live in that shouldn't be
necessary and which we can now remove with this overhaul.

Rework the whole thing so that we can correctly handle the following
situations:

 - when running in a workspace, you should now use from_file() or
   a newly introduced from_topdir() to make your Manifest

 - when running outside of any workspace, use from_data()

From now on, we forbid creating a manifest from data "as if" it were
in a real workspace. If all you have is data, the abspath attributes
will all be None, reflecting reality.

Accordingly, rework the Manifest.__init__() method to either take a
topdir or a bit of data, but not both. Additionally, now that we have
both manifest.path and manifest.file configuration options, it's not
usually the right thing to ask for a Manifest from a *file*: what you
really usually want is a Manifest from a *workspace*, and for that you
just need a topdir. From the topdir, you can create a
west.configuration.Configuration, and from there you can read
manifest.path and manifest.file to get the complete path to the top
level manifest within its repository. For that reason, we remove the
source_file keyword argument from __init__() in favor of topdir and a
new 'config' argument.

For backwards compatibility, it's still allowed to use from_file() to
load another manifest file than the one pointed to by
topdir/manifest.path/manifest.file. However, this handling code is now
just a bit of special case trickery in from_file(), and it also
introduces a restriction that the other file is still in a git
repository in the workspace. This moves potential sources for error or
confusion out of Manifest.__init__() and the functions it calls. (This
has proven to be a useful feature in practice; I am aware of west
extensions that use it to compare the parsed contents of two different
manifest files within the same workspace.)

To migrate away from the now-outdated from_file(), introduce a new
from_topdir() factory method and use it throughout the tree. This is
clearer and more accurate code, which always avoids the hacks left
behind for compatibility in from_file().

Finally, add various new instance attributes to the Manifest class
that will let us tell the YAML path from the actual path, and the path
to the manifest file from the path to the manifest repository in the
workspace. These in turn let us remove some hacky code from the 'west
list' implementation, and allow us to rework our internals so that
ManifestProject is just a legacy remnant at this point, further
helping along with zephyrproject-rtos#327.

Keep tests up to date, removing obsolete code as needed and updating
cases to reflect API breakage.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
mbolivar-nordic added a commit to mbolivar-nordic/west that referenced this issue Mar 19, 2022
This is an API break. We are still allowed to do that until west 1.0.

The west.manifest.Manifest class's path handling has become
unmaintainable and we need to overhaul it. The from_data() and
from_file() constructors are too clever and try to allow too many use
cases, which results in the guts of the manifest construction code
often not being able to tell whether it's in a real workspace or not.

This makes it difficult to know when we can safely read configuration
files or not. That in turn makes it difficult to accurately
distinguish between the 'manifest.path' value in the local configuration
file and the 'self: path:' value in the manifest data, especially when
we can take keyword arguments that say "hey, pretend this is your
path".

Additionally, there are a few assumptions in the code that pretend
that the manifest file itself always lives in the top level directory
of the manifest repository. That was a valid assumption up to the
point the 'manifest.file' configuration variable was introduced, but
it's no longer valid anymore, since 'manifest.file' can point to a
subdirectory of the manifest repository. This leaves us with some
hacks to find the git repository we live in that shouldn't be
necessary and which we can now remove with this overhaul.

Rework the whole thing so that we can correctly handle the following
situations:

 - when running in a workspace, you should now use from_file() or
   a newly introduced from_topdir() to make your Manifest

 - when running outside of any workspace, use from_data()

From now on, we forbid creating a manifest from data "as if" it were
in a real workspace. If all you have is data, the abspath attributes
will all be None, reflecting reality.

Accordingly, rework the Manifest.__init__() method to either take a
topdir or a bit of data, but not both. Additionally, now that we have
both manifest.path and manifest.file configuration options, it's not
usually the right thing to ask for a Manifest from a *file*: what you
really usually want is a Manifest from a *workspace*, and for that you
just need a topdir. From the topdir, you can create a
west.configuration.Configuration, and from there you can read
manifest.path and manifest.file to get the complete path to the top
level manifest within its repository. For that reason, we remove the
source_file keyword argument from __init__() in favor of topdir and a
new 'config' argument.

For backwards compatibility, it's still allowed to use from_file() to
load another manifest file than the one pointed to by
topdir/manifest.path/manifest.file. However, this handling code is now
just a bit of special case trickery in from_file(), and it also
introduces a restriction that the other file is still in a git
repository in the workspace. This moves potential sources for error or
confusion out of Manifest.__init__() and the functions it calls. (This
has proven to be a useful feature in practice; I am aware of west
extensions that use it to compare the parsed contents of two different
manifest files within the same workspace.)

To migrate away from the now-outdated from_file(), introduce a new
from_topdir() factory method and use it throughout the tree. This is
clearer and more accurate code, which always avoids the hacks left
behind for compatibility in from_file().

Finally, add various new instance attributes to the Manifest class
that will let us tell the YAML path from the actual path, and the path
to the manifest file from the path to the manifest repository in the
workspace. These in turn let us remove some hacky code from the 'west
list' implementation, and allow us to rework our internals so that
ManifestProject is just a legacy remnant at this point, further
helping along with zephyrproject-rtos#327.

Keep tests up to date, removing obsolete code as needed and updating
cases to reflect API breakage.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
mbolivar-nordic added a commit to mbolivar-nordic/west that referenced this issue Mar 23, 2022
This is an API break. We are still allowed to do that until west 1.0.

The west.manifest.Manifest class's path handling has become
unmaintainable and we need to overhaul it. The from_data() and
from_file() constructors are too clever and try to allow too many use
cases, which results in the guts of the manifest construction code
often not being able to tell whether it's in a real workspace or not.

This makes it difficult to know when we can safely read configuration
files or not. That in turn makes it difficult to accurately
distinguish between the 'manifest.path' value in the local configuration
file and the 'self: path:' value in the manifest data, especially when
we can take keyword arguments that say "hey, pretend this is your
path".

Additionally, there are a few assumptions in the code that pretend
that the manifest file itself always lives in the top level directory
of the manifest repository. That was a valid assumption up to the
point the 'manifest.file' configuration variable was introduced, but
it's no longer valid anymore, since 'manifest.file' can point to a
subdirectory of the manifest repository. This leaves us with some
hacks to find the git repository we live in that shouldn't be
necessary and which we can now remove with this overhaul.

Rework the whole thing so that we can correctly handle the following
situations:

 - when running in a workspace, you should now use from_file() or
   a newly introduced from_topdir() to make your Manifest

 - when running outside of any workspace, use from_data()

From now on, we forbid creating a manifest from data "as if" it were
in a real workspace. If all you have is data, the abspath attributes
will all be None, reflecting reality.

Accordingly, rework the Manifest.__init__() method to either take a
topdir or a bit of data, but not both. Additionally, now that we have
both manifest.path and manifest.file configuration options, it's not
usually the right thing to ask for a Manifest from a *file*: what you
really usually want is a Manifest from a *workspace*, and for that you
just need a topdir. From the topdir, you can create a
west.configuration.Configuration, and from there you can read
manifest.path and manifest.file to get the complete path to the top
level manifest within its repository. For that reason, we remove the
source_file keyword argument from __init__() in favor of topdir and a
new 'config' argument.

For backwards compatibility, it's still allowed to use from_file() to
load another manifest file than the one pointed to by
topdir/manifest.path/manifest.file. However, this handling code is now
just a bit of special case trickery in from_file(), and it also
introduces a restriction that the other file is still in a git
repository in the workspace. This moves potential sources for error or
confusion out of Manifest.__init__() and the functions it calls. (This
has proven to be a useful feature in practice; I am aware of west
extensions that use it to compare the parsed contents of two different
manifest files within the same workspace.)

To migrate away from the now-outdated from_file(), introduce a new
from_topdir() factory method and use it throughout the tree. This is
clearer and more accurate code, which always avoids the hacks left
behind for compatibility in from_file().

Finally, add various new instance attributes to the Manifest class
that will let us tell the YAML path from the actual path, and the path
to the manifest file from the path to the manifest repository in the
workspace. These in turn let us remove some hacky code from the 'west
list' implementation, and allow us to rework our internals so that
ManifestProject is just a legacy remnant at this point, further
helping along with zephyrproject-rtos#327.

Keep tests up to date, removing obsolete code as needed and updating
cases to reflect API breakage.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
mbolivar-nordic added a commit that referenced this issue Mar 25, 2022
This is an API break. We are still allowed to do that until west 1.0.

The west.manifest.Manifest class's path handling has become
unmaintainable and we need to overhaul it. The from_data() and
from_file() constructors are too clever and try to allow too many use
cases, which results in the guts of the manifest construction code
often not being able to tell whether it's in a real workspace or not.

This makes it difficult to know when we can safely read configuration
files or not. That in turn makes it difficult to accurately
distinguish between the 'manifest.path' value in the local configuration
file and the 'self: path:' value in the manifest data, especially when
we can take keyword arguments that say "hey, pretend this is your
path".

Additionally, there are a few assumptions in the code that pretend
that the manifest file itself always lives in the top level directory
of the manifest repository. That was a valid assumption up to the
point the 'manifest.file' configuration variable was introduced, but
it's no longer valid anymore, since 'manifest.file' can point to a
subdirectory of the manifest repository. This leaves us with some
hacks to find the git repository we live in that shouldn't be
necessary and which we can now remove with this overhaul.

Rework the whole thing so that we can correctly handle the following
situations:

 - when running in a workspace, you should now use from_file() or
   a newly introduced from_topdir() to make your Manifest

 - when running outside of any workspace, use from_data()

From now on, we forbid creating a manifest from data "as if" it were
in a real workspace. If all you have is data, the abspath attributes
will all be None, reflecting reality.

Accordingly, rework the Manifest.__init__() method to either take a
topdir or a bit of data, but not both. Additionally, now that we have
both manifest.path and manifest.file configuration options, it's not
usually the right thing to ask for a Manifest from a *file*: what you
really usually want is a Manifest from a *workspace*, and for that you
just need a topdir. From the topdir, you can create a
west.configuration.Configuration, and from there you can read
manifest.path and manifest.file to get the complete path to the top
level manifest within its repository. For that reason, we remove the
source_file keyword argument from __init__() in favor of topdir and a
new 'config' argument.

For backwards compatibility, it's still allowed to use from_file() to
load another manifest file than the one pointed to by
topdir/manifest.path/manifest.file. However, this handling code is now
just a bit of special case trickery in from_file(), and it also
introduces a restriction that the other file is still in a git
repository in the workspace. This moves potential sources for error or
confusion out of Manifest.__init__() and the functions it calls. (This
has proven to be a useful feature in practice; I am aware of west
extensions that use it to compare the parsed contents of two different
manifest files within the same workspace.)

To migrate away from the now-outdated from_file(), introduce a new
from_topdir() factory method and use it throughout the tree. This is
clearer and more accurate code, which always avoids the hacks left
behind for compatibility in from_file().

Finally, add various new instance attributes to the Manifest class
that will let us tell the YAML path from the actual path, and the path
to the manifest file from the path to the manifest repository in the
workspace. These in turn let us remove some hacky code from the 'west
list' implementation, and allow us to rework our internals so that
ManifestProject is just a legacy remnant at this point, further
helping along with #327.

Keep tests up to date, removing obsolete code as needed and updating
cases to reflect API breakage.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API west's library APIs
Projects
None yet
Development

No branches or pull requests

2 participants