Promulgating with charm un-promulgates previous charm

[arosales]

We noticed in the case with postgresql and ghost that when we promulgate new layer'ed charm for a given series, say Xenial previous series of the promulgated charm are un-promulgated.

For example, when we promulgated xenial/ghost, the store un-promulgated precise/ghost.

[urosj]

This seems like regression. We'll look into it.

[urosj]

@arosales we do not support promulgation based only on series only, but both on username and series. When promulgation of a new multi series charm for ghost was done here (not just xenial, also trusty), it was done to a new user.

The only way to get precise charm promulgated again is to put it under the same user as the new ghost charm.

[johnsca]

I don't understand why we wouldn't support different maintainers for different series. If the charms are completely different, as they are in the case of ghost, it seems perfectly reasonable to say that Adam will be the owner of the trusty and xenial series charm while Jeff will continue to own the precise charm, as they are completely different code-bases.

[arosales]

Reopening as this is a change from when the charm store was tied to LP. When the charm store was tied to LP each series was a new LP branch and thus a whole new source of code. I am not saying that is the correct way to manage this, but we do need to consider the change and how it affects the community of charmers.

@johnsca has a good point when a charm author just wants to work on the later version (ie Xenial) and not have to work on and maintain previous versions (ie Precise)

[johnsca]

I should also point out that, since Python3 is not available on Precise, no reactive charms will work on Precise.

[arosales]

The only way to get precise charm promulgated again is to put it under the same user as the new ghost charm.

@urosj Could we have a Xenial version of Zookeeper promulgated under say ~bigdata-dev and have a completely different Trusty/Precise promulgated version of Zookeeper if it published under the ~bigdata-dev name space?

[kwmonroe]

I just tried @arosales suggestion with hive. Currently, hive is a precise charm owned by ~charmers:

https://jujucharms.com/hive/

I wanted to test putting this precise charm into the ~bigdata-dev namespace, where we currently have a trusty hive charm:

https://jujucharms.com/u/bigdata-dev/hive

I did a charm pull cs:hive to get the precise version, followed by:

ubuntu@0eb88ebb43ef:~/charms/precise/hive$ charm push . cs:~bigdata-dev/precise/hive

Great! I can see a precise version of the charm at:

https://jujucharms.com/u/bigdata-dev/hive/precise

I then made the ~bigdata-dev/hive charm multi-series by adding series: [trusty, xenial] in metadata.yaml. The problem now is that I cannot make any updates to the precise version of this charm.. As a test, I changed the readme and tried to push a precise-only change and got this:

ubuntu@0eb88ebb43ef:~/charms/precise/hive$ charm push . cs:~bigdata-dev/hive
ERROR cannot post archive: series not specified in url or charm metadata

So the tl;dr is that we can push series-specific charms to a namespace in order to satisfy the requirement that all series be in the same namespace. However, it seems we cannot push updates to the non-multi-series charms.

[johnsca]

@kwmonroe You copypasta'd the wrong error message.

Note there is currently both a precise-only charm, as linked by Kevin, at https://jujucharms.com/u/bigdata-dev/hive/precise as well as the multi-series charm with both trusty and xenial (but notably not precise) support at https://jujucharms.com/u/bigdata-dev/hive/

Attempting to update the precise charm (presumably this would also affect attempting to push the initial precise charm after the multi-series one has been pushed, which is critical for fixing issues after re-promulgating charms into a new namespace), we see the following:

[johnsca@murdoch precise] $ charm pull cs:~bigdata-dev/precise/hive
cs:~bigdata-dev/precise/hive-0
[johnsca@murdoch precise] $ cd hive
[johnsca@murdoch hive] $ echo '<!-- test -->' >> README.md  # ensure the charm has a new hash
[johnsca@murdoch hive] $ charm push . cs:~bigdata-dev/precise/hive
ERROR cannot post archive: charm name duplicates multi-series charm name cs:~bigdata-dev/hive-7

The problem here is that the multi-series charm in no way overlaps with the precise charm, yet still blocks pushing the precise charm. Thus, even if the charms are promulgated from the same namespace (which is still less than ideal, but we can probably deal with it), having any multi-series charm means that it has to cover all series, ever, which is untenable.

[arosales]

@urosj any thoughts on how we should move forward on this issue?

[johnsca]

I should also note that during our testing, when there was an overlap between the multi-series and single-series charm, the single-series won out. That is, the multi-series charm only listed the non-overlapping series, and the single-series charm code was used for the series it supported, even when the multi-series charm was pushed after the single-series charm. So that's another bug in how this is handled.

Edit: I attempted to replicate this with a new charm and couldn't. Perhaps we did something wrong with the push in our initial tests, or maybe the single-series charm had been ingested automatically instead of being pushed with the new tools. I'll follow up with @kwmonroe when he gets back, but for now, please disregard this.

[ryan-beisner]

A newly-published or promulgated multi-series charm should only affect the list of series declared in that charm's metadata.yaml. All other series of the charm, whether past, future, existing or not-yet existing, should remain intact and unaffected for pushes and deploys.

Ex: A series:[xenial, yakkety] charm should not interfere with single-series pushes of the same-named charm to series:precise, series:trusty, or series:zzz.

I believe this is the expected and needed behavior.

[urosj]

Although I understand that there might be cases where current scenario is not ideal, the fix is not a simple "2 day hack". As said, we have to find a solution, but a proper one is not trivial. You all talk about nice cases, where there is no intersection between multiseries charms. Reality is not like that. Bob has promulgated charm with xenial and trusty. Alice has same char with precise. Now, why can't Alice create a multiseries charm with xenial and precise. And if she does, who's charm for xenial is promulgated? And if Alice can't push xenial multiseries charm, why not? Why is she required to keep multiple charms?

I agree that some cases are not ideal at the moment, but if we did what you suggest carelessly, there will be others with just the same amount of issues.

[urosj]

@johnsca if same user pushes multiseries charm and then after that single series, the one that automatically resolves is the last one (always), so it will be just single series. That's the only scenario in the logic of charm store, that would do what you have described.

[urosj]

@ryan-beisner, @arosales, @johnsca the charm is not resolved just by name and series. It is also resolved by user namespace and revision.. I keep repeating there are many edge cases (see previous post). Those edge cases get weirder if you take into account many revisions. Just because there is exactly one case where intersection of series is an empty set, does not mean that that case is something we should just jump on.

bob/charm/0 -> [xenial, trusty] promulgated as charm
alice/charm/0 -> [precise] promulgated as charm
alice/charm/1 -> [precise, xenial] ? what to do with xenial?
bob/charm/1 -> [xenial, trusty, precise] ? what to do with alice precise? and user upgrades?
Now bob makes a mistake (really common in charm store):
bob/charm/2 -> [tusty, precise] ? All of a sudden, all xenial charms are upgraded to Alice?

And it's easy to say: "We don't allow clashes in multiseries namespace, Alice should maintain xenial charm separately!" But then, why is Alice being "punished" for having promulgated charm for precise and now has to maintain that one and all the other in at least two repos?

And say we have a new charm.
bob/newcharm/0 -> [xenial, trusty]
alice/newcharm/0 -> [trusty, precise]
We want to promulgate both? Which trusty do you want to use?

There could be a case, that we change the promulgation logic and you have to explicitly state which series from which user you want to promulgate. But
a) that defeats the purpose of multiseries charms
b) my deployment becomes totally nondeterministic.

"juju deploy charm" (no series specified) will sometimes deploy bob/charm and sometimes alice/charm, just based on the series of the machine it's being deployed to. And based on order of API calls in controller, sometimes
juju deploy precise/othercharm --to 1
juju deploy promcharm --to 1
will work
but
juju deploy promcharm --to 1 (selecting xenial)
juju deploy precise/othercharm --to 1 (FAIL)
will not.

We really don't want such nondeterministic behaviour.

[johnsca]

Being the maintainer of a promulgated charm comes with responsibility to be aware of and resolve any conflicts, and it's reasonable for the store to reject anything that would result in an overlap in a promulgated charm. In your examples, Bob and Alice (possibly mediated by a charmer) would have to come to an agreement on who is going to maintain the charm for what series and update their charms to resolve any overlaps. The only use-case for having promulgated charms from two namespaces is that ownership and the code-base has changed at a series boundary, with the new charm not supporting the older series, in which case there is no overlap or ambiguity.

Currently, the store just silently unpromulgates the other charm, which is worse than an error message telling you that there is a conflict. And it does it even if there's no conflict.

As for your deploy scenario, that applies any time you rely on Juju to pick a series for you, regardless of whether the charm is multi-series or promulgated. It's also not non-deterministic because there are definite rules for how Juju picks a series if you ask it to. And you can always use the --series option or URL fragment to disambiguate if you know you're going to be doing something unusual like that.

Within a namespace, it should just be highest rev for charm + series wins, just like it is with a single series, because there's no way to remove a charm rev from the namespace. Currently, a multi-series charm blocks all updates to any single-series of that charm in the same namespace, even if there's no overlap. Because of this, we can't even work with the single-namespace restriction.

[johnsca]

Would it make sense to open a separate bug for the rejecting of non-overlapping updates in a single namespace?