Make os packages mutually exclusive

[chadrik]

Last week I started experimenting with using rez on linux and osx, and quickly discovered that OS packages are not mutually exclusive (I think I may be the first person actually using rez on two operating systems at the same site). There is nothing stopping a resolve on osx from pulling in packages that require Linux. Making Linux and Darwin anti-packages of each other only protects against bad resolves by aborting the resolution.

My plan is to create an "os" package, with versions per os: os-darwin, os-linux, os-windows. Versions are inherently mutually exclusive, so it will prevent os-darwin and os-linux from being loaded simultaneously, and thus allow packages that require or vary on these to resolve properly.

To achieve this, we will first need to implement a new type of Version subclass that supports a string label.

name: os
version: linux

name: os
version: darwin

name: os
version: windows

similarly, architectures can be implemented with arch-i386 and arch-x86_64 packages.

name: arch
version: x86_64

name: arch
version: i386

lastly, specific versions of an operating system can be implemented as their own packages. e.g. osx-10.8.1, fedora-19, etc. Here are some complete examples from windows, with numeric versions taken from wikipedia.

name : windows
version: 7  # technically, this is version 6.1
requires:
- os-windows
variants:
- [arch-x86_64]
- [arch-i386]

name : windows
version: 6.0
version_alias: Vista
requires:
- os-windows
variants:
- [arch-x86_64]
- [arch-i386]

name : windows
version: 5.1
version_alias: XP
requires:
- os-windows
variants:
- [arch-x86_64]
- [arch-i386]

name : windows
version: 5.0
version_alias: 2000
requires:
- os-windows
- arch-i386

[chadrik]

A prototype of this is here: https://github.com/LumaPictures/rez/tree/versions

[nerdvegas]

Hey Chad,

Wrt the pkgs and dependencies you're going with, that looks right to me.

I'll be starting work on a new version submodule soon. In the meantime
though, you could use Rez's support for single lowercase letters as a temp
solution:

os-w for windows
os-l for Linux, etc

arch-i for i386
arch-x for x86

On Friday, November 15, 2013, Chad Dombrova wrote:

Last week I started experimenting with using rez on linux and osx, and
quickly discovered that OS packages are not mutually exclusive (I think I
may be the first person actually using rez on two operating systems at the
same site). There is nothing stopping a resolve on osx from pulling in
packages that require Linux. Making Linux and Darwin anti-packages of each
other only protects against bad resolves by aborting the resolution.

My plan is to create an "os" package, with versions per os: os-darwin,
os-linux, os-windows. Versions are inherently mutually exclusive, so it
will prevent os-darwin and os-linux from being loaded simultaneously, and
thus allow packages that require or vary on these to resolve properly.

To achieve this, we will first need to implement a new type of Versionsubclass that supports a string label.

similarly, architectures can be implemented with arch-i386 and arch-x86_64packages.

name: os

version: linux

name: os

version: darwin

name: os
version: windows

name: arch

version: x86_64

name: arch
version: i386

name : windows
version: 7 # technically, this is version 6.1
requires:

• os-windows variants:
• [arch-x86_64]
• [arch-i386]

---

name : windows
version: 6.0
version_alias: Vista
requires:

• os-windows variants:
• [arch-x86_64]
• [arch-i386]

---

name : windows
version: 5.1
version_alias: XP
requires:

• os-windows variants:
• [arch-x86_64]
• [arch-i386]

---

name : windows
version: 5.0
version_alias: 2000
requires:

• os-windows
• arch-i386

name : windows
version: 4.10
version_alias: 98
requires:

• os-windows
• arch-i386

---

name : windows
version: 4.0
version_alias: 95
requires:

• os-windows
• arch-i386

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40
.

[chadrik]

I posted before I was actually ready. check github for the final post.

all of the work is done. i added label versions and added the os packages.

On Fri, Nov 15, 2013 at 5:02 PM, allan johns notifications@github.comwrote:

Hey Chad,

Wrt the pkgs and dependencies you're going with, that looks right to me.

I'll be starting work on a new version submodule soon. In the meantime
though, you could use Rez's support for single lowercase letters as a temp
solution:

os-w for windows
os-l for Linux, etc

arch-i for i386
arch-x for x86

On Friday, November 15, 2013, Chad Dombrova wrote:

Last week I started experimenting with using rez on linux and osx, and
quickly discovered that OS packages are not mutually exclusive (I think
I
may be the first person actually using rez on two operating systems at
the
same site). There is nothing stopping a resolve on osx from pulling in
packages that require Linux. Making Linux and Darwin anti-packages of
each
other only protects against bad resolves by aborting the resolution.

My plan is to create an "os" package, with versions per os: os-darwin,
os-linux, os-windows. Versions are inherently mutually exclusive, so it
will prevent os-darwin and os-linux from being loaded simultaneously,
and
thus allow packages that require or vary on these to resolve properly.

To achieve this, we will first need to implement a new type of
Versionsubclass that supports a string label.

similarly, architectures can be implemented with arch-i386 and
arch-x86_64packages.

name: os

version: linux

name: os

version: darwin

name: os
version: windows

name: arch

version: x86_64

name: arch
version: i386

name : windows
version: 7 # technically, this is version 6.1
requires:

• os-windows variants:
• [arch-x86_64]
• [arch-i386]

---

name : windows
version: 6.0
version_alias: Vista
requires:

• os-windows variants:
• [arch-x86_64]
• [arch-i386]

---

name : windows
version: 5.1
version_alias: XP
requires:

• os-windows variants:
• [arch-x86_64]
• [arch-i386]

---

name : windows
version: 5.0
version_alias: 2000
requires:

• os-windows
• arch-i386

name : windows
version: 4.10
version_alias: 98
requires:

• os-windows
• arch-i386

---

name : windows
version: 4.0
version_alias: 95
requires:

• os-windows
• arch-i386

—
Reply to this email directly or view it on GitHub<
https://github.com/nerdvegas/rez/issues/40>
.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-28615776
.

[chadrik]

to accomplish label versions i had to make a change to the heart of the resolving algo, but i wrote a bunch of tests beforehand to be sure that the results were the same after the refactor.

rez was iterating through available packages in a somewhat inefficient way. it would ask for a version in range "0+<5", then if the resolved failed, it would move to "0+<4", then to "0+<3", etc, always setting the new upper bound to the value of the last failed version:

ver_range_valid = ver_range_valid.get_intersection(VersionRange("0+<" + pkg_req_.version))

each time that newly formatted range was passed to memcache.find_package2, which performed a new search: iterating through all of the values that we previously tried and failed and re-filtering them until it came to a version below our new boundary. the directory access was cached, so it wasn't terribly slow, but still, it was not very efficient, especially when you consider that the net effect of this behavior is simply to iterate through available versions in descending order. after i had that realization i replaced this logic with a python generator that iterates through versions (also, i should not that earliest mode is currently broken and is now fixed in my branch).

the reason i tackled this problem is because replacing the "next valid version" logic with an iterator made adding "label" versions possible, because it removed the need for labels to work with the numeric version range syntax, which would not have made sense.

have a look and let me know. i'm just about done with the changes i need to get rez up and running here. i just need to write some more tests and make certain that everything is working cross-platform.

[nerdvegas]

Hey Chad,

Yeah that rings a bell. It's funny, some inefficiencies I didn't bother
doing anything about because I was working on the resolving algorithm so
much of the time, and that by far affects resolve speed the most - in fact
it's still possible to expose Rez to worst case resolve scenarios that can
theoretically never resolve because it will take too long (this is very
rare, but I have seen it). This is one of the reasons I want to revisit the
resolve algorithm and maybe use something a bit fancier, such as SAT.
Anyway, I'd actually forgotten about the case you've just pointed out - the
fix sounds correct to me.

Version aliases - I get what this is, but how do we avoid the problem where
a user tab-completes when using rez-env for eg, sees two options
(windows-6, windows-vista), and doesn't know that they're the same thing? I
think this is a problem. One thing we could do is visually indicate
aliases, perhaps with a trailing symbol:

(windows-6, windows-vista@)

If the trailing @ gets pulled in as part of the arg (which will be often
because of completion) Rez would just strip it. This would be a cli feature
only.

A

On Fri, Nov 15, 2013 at 7:03 PM, Chad Dombrova notifications@github.comwrote:

to accomplish label versions i had to make a change to the heart of the
resolving algo, but i wrote a bunch of tests beforehand to be sure that the
results were the same after the refactor.

rez was iterating through available packages in a somewhat inefficient
way. it would ask for a version in range "0+<5", then if the resolved
failed, it would move to "0+<4", then to "0+<3", etc, always setting the
new upper bound to the value of the last failed version:

ver_range_valid = ver_range_valid.get_intersection(VersionRange("0+<" + pkg_req_.version))

each time that newly formatted range was passed to memcache.find_package2,
which performed a new search: iterating through all of the values that we
previously tried and failed and re-filtering them until it came to a
version below our new boundary. the directory access was cached, so it
wasn't terribly slow, but still, it was not very efficient, especially when
you consider that the net effect of this behavior is simply to iterate
through available versions in descending order. after i had that
realization i replaced this logic with a python generator that iterates
through versions (also, i should not that earliest mode is currently broken
and is now fixed in my branch).

the reason i tackled this problem is because replacing the "next valid
version" logic with an iterator made adding "label" versions possible,
because it removed the need for labels to work with the numeric version
range syntax, which would not have made sense.

have a look and let me know. i'm just about done with the changes i need
to get rez up and running here. i just need to write some more tests and
make certain that everything is working cross-platform.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-28618429
.

[chadrik]

Version aliases - I get what this is, but how do we avoid the problem where a user tab-completes when using rez-env for eg, sees two options (windows-6, windows-vista), and doesn't know that they're the same thing? I think this is a problem. One thing we could do is visually indicate aliases, perhaps with a trailing symbol:

(windows-6, windows-vista@)

what i'm calling "label" versions are different from aliases. a version alias must have an underlying numeric version, but a label version is a valid version in its own right: like a numeric version it has its own sub-directory, and is supplied as the version in the package.yaml.

i don't think that it should be legal for a package to mix label and numeric versions: either a package has a standardized, hierarchical version convention or it doesn't. attempting to mix the two makes for some pretty nasty situations in the Version and VersionRange code, since, for example, it makes no sense to intersect Foo with 1.2+.

so, in the example that you've given, the version windows-vista must logically be an alias because windows-6 exists. i'm not entirely against the @ sign, but i think it's a little overprotective and we might need the symbol for something else later (the number of shell-safe symbols is dwindling). after all, a user will have to read the manual to even know what the @ means.

two alternate solutions that will help people learn about aliases: a more fully featured rez-info command (one that basically does a print of the package.yaml with colorization) and a feedback statement printed to the terminal during resolution saying windows-vista --(alias)--> windows-6.

version alias tab-completion has a tricky technical problem lurking, which is that many package.yaml files must be read in order to know them. it might make sense to put the mapping in a family-level windows/package.yaml so that only one file needs to be read per package:

name :  windows
version_aliases :
  vista : 6.0
  2000 : 5.0

lastly, I thought I should point out that I won't actually be implementing this now, since I don't need it to get rez rolled out here. we should probably split this discussion into a separate ticket.

[chadrik]

i have a question for everyone regarding the os package: do you prefer the name platform or os?

e.g. do you like this better:

os-darwin
os-linux
os-windows

or this:

platform-darwin
platform-linux
platform-windows

(keeping in mind that you won't actually need to type these at the command prompt)

i can sense that there will be some confusion over the name os, especially when users implement windows, osx, fedora, and centos packages, which must require the appropriate os package version to gain mutual exclusion. presenting darwin, linux, and windows as platforms might be more intuitive. what do you think?

[nerdvegas]

On Monday, November 18, 2013, Chad Dombrova wrote:

Version aliases - I get what this is, but how do we avoid the problem
where a user tab-completes when using rez-env for eg, sees two options
(windows-6, windows-vista), and doesn't know that they're the same thing? I
think this is a problem. One thing we could do is visually indicate
aliases, perhaps with a trailing symbol:

(windows-6, windows-vista@)

what i'm calling "label" versions are different from aliases. a version
alias must have an underlying numeric version, but a label version is a
valid version in its own right: like a numeric version it has its own
sub-directory, and is supplied as the version in the package.yaml.

i don't think that it should be legal for a package to mix label and
numeric versions: either a package has a standardized, hierarchical version
convention or it doesn't. attempting to mix the two makes for some pretty
nasty situations in the Version and VersionRange code, since, for
example, it makes no sense to intersect Foo with 1.2+.

I'll forgo all the details here, but I have a versioning scheme that allows
numerical and non numerical version parts to coexist nicely, where in fact
the intersection you have above is valid (although it would be empty in
this example). Really, you need to have full alphanumeric support, because
you need to compare versions such as foo.3.bah and foo.eek.7. This work has
been done in Certus and I'll be porting it into Rez.

so, in the example that you've given, the version windows-vista must
logically be an alias because windows-6 exists. i'm not entirely against
the @ sign, but i think it's a little overprotective and we might need
the symbol for something else later (the number of shell-safe symbols is
dwindling). after all, a user will have to read the manual to even know
what the @ means.

two alternate solutions that will help people learn about aliases: a more
fully featured rez-info command (one that basically does a print of the
package.yaml with colorization) and a feedback statement printed to the
terminal during resolution saying windows-vista --(alias)--> windows-6.

Colorisation is a nice touch but I'd say one we shouldn't rely on, we
should make sure that any print out still gives enough context sans color.
Re the alias info, for sure. This should be indicated in the dot graph
also.

version alias tab-completion has a tricky technical problem lurking, which
is that many package.yaml files must be read in order to know them. it
might make sense to put the mapping in a family-level windows/package.yamlso that only one file needs to be read per package:

name : windowsversion_aliases :
vista : 6.0
2000 : 5.0

It might have to be done this way, because another problem is alias
conflicts, what's to stop two different pkg versions from claiming to own
the vista alias?

lastly, I thought I should point out that I won't actually be implementing
this now, since I don't need it to get rez rolled out here. we should
probably split this discussion into a separate ticket.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-28723370
.

[chadrik]

I'll forgo all the details here, but I have a versioning scheme that allows numerical and non numerical version parts to coexist nicely, where in fact the intersection you have above is valid (although it would be empty in this example). Really, you need to have full alphanumeric support, because you need to compare versions such as foo.3.bah and foo.eek.7. This work has been done in Certus and I'll be porting it into Rez.

sounds awesome.

i would beg a favor from you: can you start from my changes? i'm still pretty anxious that i'm going to end up with my own unmergable version of rez.

what i've done in my versions branch:

• code cleanup on the Version and VersionRange classes
• added two new subclasses for ExactVersion and ExactVersionSet. ExactVersion instances are only created when rez is certain the version requested is exact (e.g. in rez_filesys.get_versions_in_directory()). it's a nice to have the distinction because they have slightly different properties than regular versions, since they do not represent a range.
• merged in my iter_resolve branch with the fix mentioned above
• added a testsuite for versions and resolve/config

Colorisation is a nice touch but I'd say one we shouldn't rely on, we should make sure that any print out still gives enough context sans color.

agree.

It might have to be done this way, because another problem is alias conflicts, what's to stop two different pkg versions from claiming to own the vista alias?

good point.

[chadrik]

oh, forgot to mention that the versions branch also changes the os packages from e.g. Linux to os-linux (possibly platform-linux pending feedback on that topic).

[instinct-vfx]

Am not really sure i guess. Are we still supporting Architecture? I always referred to a platform as OS+HW (e.g. linux-x86). So i guess i'd vote for OS.

[nerdvegas]

On Monday, November 18, 2013, Chad Dombrova wrote:

I'll forgo all the details here, but I have a versioning scheme that
allows numerical and non numerical version parts to coexist nicely, where
in fact the intersection you have above is valid (although it would be
empty in this example). Really, you need to have full alphanumeric support,
because you need to compare versions such as foo.3.bah and foo.eek.7. This
work has been done in Certus and I'll be porting it into Rez.

sounds awesome.

i would beg a favor from you: can you start from my changes? i'm still
pretty anxious that i'm going to end up with my own unmergable version of
rez.

what i've done in my versions branch:

• code cleanup on the Version and VersionRange classes
• added two new subclasses for ExactVersion and ExactVersionSet.
  ExactVersion instances are only created when rez is certain the
  version requested is exact (e.g. in
  rez_filesys.get_versions_in_directory()). it's a nice to have the
  distinction because they have slightly different properties than regular
  versions, since they do not represent a range.
• merged in my iter_resolve branch with the fix mentioned above
• added a testsuite for versions and resolve/config

Yup will do. I'm a bit confused about ExactVersion though... In my new
version code, Version is an exact version. VersionRange contains Versions,
which define the lower and upper bounds of the range. Pretty much analogous
to normal numbers and number ranges, in fact you can think of it as a kind
of number system really.

Speaking of merges, before I branch off your version branch, would you be
able to merge in the changes from merge_osx? They're fairly minimal but it
fixes some bugs in the osx port.

Colorisation is a nice touch but I'd say one we shouldn't rely on, we
should make sure that any print out still gives enough context sans color.

agree.

It might have to be done this way, because another problem is alias
conflicts, what's to stop two different pkg versions from claiming to own
the vista alias?

good point.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-28727076
.

[chadrik]

I'm a bit confused about ExactVersion though... In my new version code, Version is an exact version. VersionRange contains Versions, which define the lower and upper bounds of the range. Pretty much analogous to normal numbers and number ranges, in fact you can think of it as a kind of number system really.

the current Version class is really a version range, while the VersionRange is really a set of Version instances. i left that code mostly alone and introduced a class that represents a single exact version. If your new Version is exact -- i.e. represents precisely a single version -- then all the better.

also, i made Version immutable, for a few reasons:

• it can be used as the key in a dictionary
• no more need to call copy()
• there were not any cases in the code where Versions were being used in a way that mutability would be advantageous

Speaking of merges, before I branch off your version branch, would you be able to merge in the changes from merge_osx? They're fairly minimal but it fixes some bugs in the osx port.

already done. i have a few minor things to cleanup on the version branch, but it's mostly ready to go.

[nerdvegas]

On Monday, November 18, 2013, Chad Dombrova wrote:

I'm a bit confused about ExactVersion though... In my new version code,
Version is an exact version. VersionRange contains Versions, which define
the lower and upper bounds of the range. Pretty much analogous to normal
numbers and number ranges, in fact you can think of it as a kind of number
system really.

the current Version class is really a version range, while the
VersionRange is really a set of Version instances. i left that code
mostly alone and introduced a class that represents a single exact version.
If your new Version is exact -- i.e. represents precisely a single
version -- then all the better.

also, i made Version immutable, for a few reasons:

• it can be used as the key in a dictionary
• no more need to call copy()
• there were not any cases in the code where Versions were being used
  in a way that mutability would be advantageous

Cool. Good idea re version immutability, I think this is advantageous also.

Speaking of merges, before I branch off your version branch, would you be
able to merge in the changes from merge_osx? They're fairly minimal but it
fixes some bugs in the osx port.

already done. i have a few minor things to cleanup on the version branch,
but it's mostly ready to go.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-28734291
.

[mstreatfield]

i have a question for everyone regarding the os package: do you prefer the name platform or os?

Personally I prefer platform. On the Linux side, we'd most likely have a CentOS package (which would be the os) which would depend on a platform-linux package. Although as someone else commented, where would the architecture fit in this hierarchy?

Would CentOS-6.3 depend on plaform-linux and arch-x64?

[chadrik]

Would CentOS-6.3 depend on plaform-linux and arch-x64?

yup.

[chadrik]

btw, I also prefer platform over os because I think it is more conceptually accurate and will thus reduce confusion.

[nerdvegas]

I'll go with that.

On Monday, November 18, 2013, Chad Dombrova wrote:

btw, I also prefer platform over os because I think it is more
conceptually accurate and will thus reduce confusion.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-28750553
.

[chadrik]

ok, one more opinion call: platform or plat? plat matches arch nicely, but may be too obscure?

platform-linux
platform-darwin
platform-windows
arch-x86_64
arch-i386

plat-linux
plat-darwin
plat-windows
arch-x86_64
arch-i386

-chad

[instinct-vfx]

I say plat for no particular reason besides liking the alignment more than disliking the obscurity ;)

[nerdvegas]

testing (going thru work firewall - please ignore)

[nerdvegas]

Ok all, I'm starting on Rez work here at my work. One of our goals is to use Rez to share software across all of our worldwide studio locations. In order to do this, we have to have agreement over every site that a package with a given name and version, is the same thing at one studio as it is at another. For example, the package representing the linux platform would have to be called, say, "platform-linux" at every site.

So from my POV we need to get a standard naming convention nailed down very soon, and have agreement across the board on it. I feel like we've gotten close on this thread, but not quite there - for instance, 'plat' or 'platform'? What are the exact platform names, and how are they chosen? Can we use python's platform module as a yardstick for some of this? Etc etc.

Thx all
A

[nerdvegas]

To kick off the discussion again, I have two points:

plat vs platform. Either way I agree with this name, it is more correct than 'os'. I vote for 'platform' though... whilst 'arch' is also a shortening, it's a well known one, and 'plat' is not, for that reason only I think we should avoid it.

Chad in your example above where you showed all the various 'windows' packages, isn't the problem here that various os-packages are not mutually exclusive? For example, you could resolve both a fedora and an ubuntu package into the same env, but this will more than likely cause problems. Shouldn't we take the same approach to os as we do to platform? Consider:

name: platform
version: linux


name: os
version: fedora.19

name: os
version: ubuntu.12.4


name: arch
version: x86_64

name: arch
version: i386


# now we can package them up into something nicer to use
name : ubuntu
version: 12.4
requires:
- platform-linux
- os-ubuntu.12.4
variants:
- [ arch-i386 ]
- [ arch-x86_64 ]

[mstreatfield]

1. plat vs platform.

I prefer platform.

In the thread about making Rez pure python we discussed a machine() class/function which would provide information about the current platform for access in the commands section. Should that also become platform() to be consistent, or is it providing sufficiently different/specialised information that this is ok? Should the platform package be a source for some of the detail that machine provides?

2. Shouldn't we take the same approach to os as we do to platform?

That does seem like quite a neat solution. Would the features feature we've discussed before help at all here?

[mottosso]

Is this implemented? :)

[nerdvegas]

Yes, we've standardized on platform/arch/os packages for a long time now. That's juts default behavior though, you're free to do this as you'd like. Note this entry in rezconfig.py: implicit_packages = [ "~platform=={system.platform}", "~arch=={system.arch}", "~os=={system.os}", ] https://github.com/nerdvegas/rez/wiki/Basic-Concepts#weak-references Some packages may not be platform dependent, hence the weak references. Hth A

…

On Mon, Apr 22, 2019 at 11:41 PM Marcus Ottosson ***@***.***> wrote: Is this implemented? :) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#40 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMOUSQD33XHAKAK3ZJXRE3PRW6BTANCNFSM4AJW7BSQ> .