hab pkg download file format is insufficiently expressive.

[markan]

The current file format (package idents separated by newlines) is isn't expressive enough to easily treat as an independent artifact.

Many origins have different channel names and promotion policies than core. When building a 'starter list' of packages, such as we are doing in #6902, we end up wanting to include multiple origins and channels. For example we pretty much always want stable packages from core, but might accept unstable from effortless or other third party origins.

If the wrong channel is used, many packages may not be at the right versions, or even found, and so a list of package idents isn't really complete without a channel specified. Unfortunately, the channel can only be specified on the command line, and applies to all the files provided. There isn't a way to specify multiple channels in the same input file, or designate the channel as part of the file.

A similar problem exists for target architecture, again packages might not exist for all architectures, and so the input file is in practice architecture specific as well.

So sharing 'starter list' files divorced from the context of channel and target is likely to be error prone, and in some cases tedious, as the download command will need to be run multiple times.

One possibility would be to stay with the simple text file format but expand the ident syntax to allow some sort of qualifier. Something like /ORIGIN/NAME?target=TARGET&channel=CHANNEL might work, but there's almost certainly a better UX there. We would have a kludgy workaround if we were able to find a fully qualified ident without needing the channel ident but currently we only look in the current channel even if the package is otherwise fully specified (see #7039)

It might make more sense to switch to a human friendly structured text file format. Most likely you would want to represent a list of tuples {target, channel, [package_idents]} or a list of hashes {target: TARGET, channel: CHANNEL, package_idents: [ALL_THE_IDENTS]}.

JSON is quite expressive, but the mainstream varieties don't have comments.
TOML is less expressive (my initial attempt was pretty clunky), but is commonly used in habitat, and allows comments.
YAML is popular, expressive and allows comments, but currently isn't used inside habitat, and might be too much.
ASN.1 ... just kidding.

[markan]

We want a format that is

1. expressive enough to allow us to fully specify packages by channel and target
2. extensible, so that we don't have to change the format yet again
3. self documenting

Probably the simplest option would be to use TOML; we could do something like:

# simple form (in many cases all we need)
target = "x86_64-linux"
channel = "unstable"
packages = [ "core/foo", "core/bar" ]

# more complex form, can be used alongside or instead of the simple form above.
[[section]]
target = "x86_64-linux"
channel = "stable"
packages = [ "core/foos", "core/bars" ]

[[section]]
target = "x86_64-windows"
channel = "stable"
packages = [ "core/foosw", "core/barsw" ]

The '[[section]]' is a little clunky, but necessary (I think) to get a list of hash entries. The actual keyword 'section' is arbitrary, and perhaps a better one could be chosen. It still seems likely to cause occasional user confusion.

The advantages of this is we already use TOML files as part of habitat plans, so users likely to be familiar with that format.

The downside is that TOML isn't as mainstream as YAML, and the structure we want (a list of tables) is a little clunky to express in YAML, because the 'root' of a YAML file is a map, and we probably want a list. A similar file in YAML would look like a list of maps

- target: "x86_64-linux"
  channel: unstable
  packages:
    - "core/foo"
    - "core/bar"
- target: "x86_64-linux"
  channel: stable
  packages: [ "core/foos", "core/bars" ]
- target: "x86_64-windows"
  channel: stable
  packages: [ "core/foosw", "core/barsw" ]

A simple file could be just a map

target: "x86_64-linux"
channel: unstable
packages:
    - "core/foo"
    - "core/bar"

YAML provides a more flexible format, and at least appears more concise and lacks the somewhat arbitrary ```[[section]]`` constructs. However it has us introducing a new file format into habitat, with some usability costs of its own.

Alternately we could come up with our own DSL to express this. I have trouble seeing this being worth the effort, unless there was a huge usability gain, and we have great confidence that we were going to want this workflow long term.

[mwrock]

I think there is probably a use case that I am not considering but I am having difficulty understanding why a download file would want multiple targets. I would think it safe to assume that the target should match the architecture calling the command. If that machine downloaded harts for other architectures, what could it do with them.

Perhaps the use case is to be able to download a "bundle" for a particular target that will be deployed to other supervisors of that target architecture, but one may perform the doenload from any target. Is that correct?

[markan]

There is a use case around grabbing packages for on-prem-builders, which may be behind an air-gap or firewall. The architecture(s) running hab behind the firewall may be different than the architecture that has access for download. For example, a user may have a mixed linux and windows environment, and will want to be able to grab packages for both in one go.

[mwrock]

Got it! Thanks.

I personally prefer TOML since that is more habitat ubiquitous and so far we don't use yaml but I don't feel incredibly strong about this either. If going with TOML, we could use the target as the table name.

[apriofrost]

Discussion note: had a conversation with @markan about the idea. We prefer to stick to using TOML for the reason that both Automate 1 and 2 use it as well, besides Habitat. The ideas below are not valid TOML and we will think more about how to make the naming of the table both human- and machine-readable.

Throw some ideas for discussion. I don't have strong opinions on the file format, my idea is mostly around how the information is structured. The comments could use more work.

Here is a template example if they are in the TOML format:

# A set of packages to seed for the x86_64-linux platform.
[[x86_64-linux packages]]
  target = "x86_64-linux" 

  # Specify a channel name to seed the latest version of x86_64-linux packages in the channel.
  [["x86_64-linux packages"."in channel"]] 
    channel = "CHANNEL_NAME_1" 
    packages = [ "ORIGIN/NAME", "ORIGIN/NAME" ]


  # Specify a channel name to seed the latest version of x86_64-linux packages in the channel.
  [["x86_64-linux packages"."in channel"]] 
    channel = "CHANNEL_NAME_2" 
    packages = [ "ORIGIN/NAME", "ORIGIN/NAME" ]


  # Seed specific releases of x86_64-linux packages, no matter which channel they are.
  [["x86_64-linux packages"."in channel"]] 
    packages = [ "ORIGIN/NAME/VERSION/RELEASE", "ORIGIN/NAME/VERSION/RELEASE" ]


# A set of packages to seed for the x86_64-windows platform.
[[x86_64-windows packages]]
  target = "x86_64-windows" 

  # Specify a channel name to seed the latest version of x86_64-windows packages in the channel.
  [[latest packages in CHANNEL_NAME_1]] 
    channel = "CHANNEL_NAME_1" 
    packages = [ "ORIGIN/NAME", "ORIGIN/NAME" ]


  # Specify a channel name to seed the latest version of x86_64-windows packages in the channel.
  [[latest packages in CHANNEL_NAME_2]] 
    channel = "CHANNEL_NAME_2" 
    packages = [ "ORIGIN/NAME", "ORIGIN/NAME" ]


  # Seed specific releases of x86_64-windows packages, no matter which channel they are.
  [[packages matching the fully qualified identifiers]] 
    packages = [ "ORIGIN/NAME/VERSION/RELEASE", "ORIGIN/NAME/VERSION/RELEASE" ]

If the list is fully defined, it could look like this:

[[x86_64-linux packages]]
  target = "x86_64-linux" 

  [[latest packages in unstable]] 
    channel = "unstable" 
    packages = [ "core/foo", "core/bar" ]

  [[latest packages in stable]]
    channel = "stable"
    packages = [ "core/foos", "core/bars" ]

  [[packages matching the fully qualified identifiers]] 
    packages = [ "core/foos/1.0/20191011132470", "core/bars/1.0/20191011132470" ]

[[x86_64-windows packages]]
  target = "x86_64-windows" 

  [[latest packages in unstable]] 
    channel = "unstable"
    packages = [ "core/foo", "core/bar" ]

  [[latest packages in stable]]
    channel = "stable"
    packages = [ "core/foos", "core/bars" ]

  [[packages matching the fully qualified identifiers]] 
    packages = [ "core/foos/1.0/20191011132470", "core/bars/1.0/20191011132470" ]

[markan]

A few things from the discussion with @apriofrost

It seems like it might be nice to have a more hierarchical structure, probably some nesting of architecture, channel, package. The TOML example I provided is relatively flat. Perhaps we want something with a nesting structure more like"

architecture_1
    channel_1
         packages
    channel_2
         packages

architecture_2
....

Another issue is whether we want to make default assumptions as to channel; should we assume stable unless it is specified?

Do we want to version this file format? Probably would have something like version = 1 at the top level.

[markan]

One possibility would be to make the nesting implicit:

[x86_64-linux.unstable]
packages = [...]
[x86_64-windows.stable]
packages = [...]

The chief advantage is that the file format is very simple.

I see a few downsides:

• The TOML nesting is implicit, not explicit; the user has to know the hierarchy of architecture, channel, package
• Many users find the TOML nesting via '.' unintuitive, and I expect a common error path is to put [arch] and [channel] as separate entries.

[mwrock]

While I can see some benefits of hierarchy, it seems like a flatter structure does not lose much but gains back more in clarity:

[x86_64-linux]
terget = "unstable"
packages = [...]

[x86_64-windows]
target = "stable"
packages = [...]

Also I like the idea of making stable the default and versioning the format.

[kagarmoe]

I'm wondering how an end-user would really think about this information. Would the sysadmin or developer ask "What architectures have which packages installed" or "What packages are installed for which architectures"? I don't have the answer.

Individual lists for each architecture potentially has significant redundancy in the file, which means a lot of room for typos, bad copy-pasta, and the package lists could get hard to read, which means that users could easily scan ["core/bar", "core/bars/", "core/bat"] as ok when they actually wanted ["core/bar", "core/barz", "core/bat"]

[x86_64-linux]
target = "unstable"
packages = [core/foo, core/bar]

[x86_64-linux]
target = "stable"
packages = [core/foo, core/bar]

[x86_64-windows]
target = "stable"
packages = [core/foo, core/bar]

Or is an end-user more likely to think of the information:

# one package, two targets, two channels
[core/foo]
architecture = [x86_64-linux, x86_64-windows]
target = ["stable", "unstable"]

# one package, two targets, two channels (uses two blocks)
[core/bar]
architecture = [x86_64-linux]
target = ["stable"]
 
[core/bar]
architecture = [x86_64-windows]
target = ["unstable"]

# one package, one target, one channel
[core/baz]
architecture = [x86_64-windows]
target = ["stable"]

[apriofrost]

@kagarmoe good call!

There is currently a download file, which is user-defined and serves as an instruction for Habitat to decide which package exactly to download.

There will be another manifest file (temporary name), which is Habitat-generated and as an output to show users which package exactly were downloaded.

In my current understanding, the download file format serves these major functions:

How could we help users easily define the packages they want to download from the target Builder?

It has to be flexible to cover the use cases where:

1. users know which exact version or release they want to download
2. users just want the latest version for a package defined by the origin, package, and channel names
3. it's likely that users will make changes to these files as their needs for the packages change over time, so they need to know which file exactly is the latest and probably even which file was used during which download.

---

For the manifest file, it could reuse the same format as the download file, or it could have its own format. We probably need to leverage the effort for creating two separate file formats v.s. satisfying different needs from different use cases. The manifest file format services these major functions:

How could we help users easily understand which packages they have downloaded from the target Builder?

I think this question is more related to the example you demonstrated. I don't have a good answer to this question now without a good understanding of the users' perspective.

---

For the download file format, it reminded me of a time when I was trying to redesign the versions tab under the package details page. I also feel this correlates with how users would like to search and filter on the packages they want.

Currently the versions tab groups packages by version number, and for each version, show a mix of platforms each release under the version has. It can be confusing when each version has a different set of platforms.

From talking with @smacfarlane and others, my understanding is that when looking for a package, users are most interested in the platform and the release. For example, they want to look for the "latest Linux packages" for x package.

So I'd support using the platform name as the primary grouping mechanism for the download file since it resembles the use case of "looking for a package".

Food for thought! 😸

[smacfarlane]

Another thing to note, is the channel probably belongs on an origin level, rather than target. You typically want to ensure that, within an origin, you're pulling from the same channel. If not, you're likely to end up with graph conflicts when you try to build.

I would probably express this something like:

[origins.channels]
"core" = "stable" # This would also be the default
"stuartpreston" = "development"
"chef" = "current"

Though that makes the format more complex, but describing channel at the target level really makes my spideysense tingle.

---

To echo @apriofrost, in my experience users are typically thinking of target as the top level construct. "For linux, I need core/foo and core/bar, for windows I need core/bar, etc."

In addition, I realize the above are just examples, but we want to be careful not to change our terminology. I see target above being used in place of channel and architecture in place of target which we want to make sure doesn't end up in the final format.

# 'target' could be more accurately called 'package_target',
# though is often shortened because humans like shorter names :smile: 
target = [ x86_64-linux, x86_64-linux-kernel2, x86_64-windows, x86_64-darwin ]
channel = [ 'unstable', 'stable', 'etc' ] 

[markan]

Thinking through the comments above, a few thoughts

1. origin and channel are indeed likely linked, and probably should be taken together

2. There's likely enough overlap between packages in different target architectures to make it worthwhile to allow multiple targets.
   That could be done via the 'target' key either always taking an array, or optionally taking an array, or a separate 'targets' that takes an array. (I'd lean towards optionally taking an array myself)

3. We should give some thought to this being usable as the manifest output format; I think we should be able to take a manifest output and use that as input for pkg download, as well as pkg upload

4. There's been a lot of discussion around what part of the hierarchy people start with; it sounds like there's some case for target architecture first. But I wonder if maybe we should start with origin. That's closer to how we present things in the builder GUI.

5. Listing target architecture and channel for each package might be a bit too verbose, as there is likely going to be a bunch of packages with the same channel and architecture.

Here's a slightly different format proposal based on the above:

# simple
version = 1
target = "x86_64-linux"
channel = "stable"
packages = [ "core/foo", "core/bar" ]

# complex
[[core]] # choosing array syntax here, so you can have multiple entries for the same origin
target =  "x86_64-linux"
# omitting channel implies stable
packages = ["foo", "bar/3.2", "baz/2.71/20190910181446"]

[[core]]
target =  ["x86_64-linux", "x86_64-windows"] 
channel = "unstable"
packages = ["crazypants", "cliffs_of_insanity/0.0.1/20191010181446"]

[[effortless]]
target = "x86_64-windows"
channel = "stable"
packages = ["audit-baseline"]

[smacfarlane]

Some off the cuff thoughts to the above @markan

1. Strong agree 😺
2. This feels it could be overly complex, and lead to some unexpected results. Though that may be my bias from build where you want to be very explicit about what you're doing.
3. I like this idea, perhaps approaching this from the upload angle of "how do we put humpty-dumpty together again" might allow us to tease out what we're trying to describe.
4. Maybe the Builder UI has it backwards, depending on your persona? If I'm managing packages, I would start with my origin. If I'm searching for something though, I'd probably start with where I intend to deploy it. I may not care about Linux, but I am probably interested in a variety of origins.
5. I agree with channel, that's an origin level concept that describes part of the "where". Target is a package level construct that is a "what", ideally it would be part of our fully qualified ident, but could also be a grouper, i.e. "I want the following packages for my Linux systems","and then these for my Windows systems"

My preference would also to not decouple origin from name, as that goes against how we typically describe idents*, and (depending on implementation of course) make 3 harder to implement. I'd think that if we go that route, the only difference between the input and output of download would be the output would contain exclusively fully qualified idents.

• It occurs to me that perhaps we're trying to describe different things with what goes into packages. My assumption is that it should be a partially or fully qualified ident, i.e. something you could drop straight into hab pkg install.

[apriofrost]

@smacfarlane I'm not quite understanding why the packages from the same origin have to come from the same channel, could you explain a bit more? :)

In the hab pkg bulkupload command, users can use the --channel option to specify the channel to upload the artifacts to (default: unstable).

That means the channels where the artifacts are downloaded from the target Builder specified in hab pkg download are not relevant in the target Builder specified in hab pkg bulkupload.

For your point about the graph conflicts when build, does it only matter in the target builder specified in the hab pkg bulkupload command?

[markan]

Had a bunch of discussions online and by zoom today, trying to sum up where we left off.

Versioning:

This file format may well need to change in the future, and we should
have a format_version field to help us track that.

Traceability:

It would eventually be helpful to produce some information on the
origin of the inputs. This is especially helpful if we are generating
manifests, where some logging of the inputs would be useful. Two
possibilities occur to me. First is to have a version key in the file,
that the user can update (or the VCS can set automatically) and
include that in various outputs. Second is to produce a normalized
form of the file and compute and record a checksum of that. This
probably should be saved for version 2 of the format.

Expressiveness:

The full combinatorial space is required right now. Some seed lists
may draw from multiple channels; some use cases (like chef and
effortless packages) may require fetches from both stable, current and
unstable to complete. So mapping origins to channels isn't sufficient
to cover our use case, we have to allow arbitrary origin, channel,
target combinations.

An important nuance of the current implementation of hab pkg download is that we don't try to 'resolve' the packages into a
consistient set. Instead we resolve the ident to a fully qualified
ident, and then add the set of TDEPS to the download. Multiple
versions of the same package may be downloaded, depending on what the
top level packages resolve to.

This provides completeness, at the cost of excess downloads, and
possible conflicts when building local packages. However, those
conflicts should be the exact same as if the build happened against a
snapshot of builder taken when the download occured.

After some discussion, allowing combinatorial specifiers (this is the
list of packages for this list of targets) seems unlikely to be as
useful as hoped, and unfortunately quite fragile. Linux and windows
diverge at a high frequency, and so a 'common' subsection as well as
individual 'packages for windows' and 'packages for linux' subsection
will be required. Even the correct channel to use might be different.

Structure and hierarchy:

Much of this discussion boils down to whether we have a flat
structure, or a hierachical structure, and if so, how many layers. If
we choose a hierarchical structure, we have to decide which entity
goes first, and how deep we go.

The flat structure ends up having an array of tables specifing target,
channel, and package list groups. For the sake of the format, there's
some arbitrary keyword we use to denote each one. I've used 'section'
here, but open to suggestions

For hierarchical structures, so far we've had target, channel, origin,
and package suggested as the place to start.

@smacfarlane as convinced me that origin makes little sense. It
goes against the way most people conceptialize things, and it breaks
the standard identifier syntax a bit, so I'm thinking I should drop
that idea.

A package centric approach would tend to be verbose; we usually have
lists where there are many packages for a (target, channel) pair. And
many things would switch as a unit; deciding to rename or restructure
channels (as is happening with the introduction of 'current' in our
workflow) would require large numbers of lines to change.

Target would work fine; there's not much overlap between package names
across targets. For example the counts for core_full are:

685 core_full_x86_64-linux_stable 
102 core_full_x86_64-windows_stable
 61 in common

Channel could also work as a starting point. The disadvantage is that
the list of channels is arbitrary, and would difficult to validate
during parsing. We control the list of targets, so there's a natural
method of checking for valid target keys in the parsed TOML.

That issue also arises if we choose some hierarchical scheme where we
have TARGET.CHANNEL (e.g. x86_64-linux.stable) sections. In general,
it would be better to have a format that allows for up front
verification.

Use of the file format for package manifests:

None of the formats would preclude using them as package manifest
formats, at least in the simplest possible way. We'd emit clauses for
each target with the expanded list of TDEPS, and the channel
'unstable'. It's hard to choose any other channel. While the bulk of
the transitive dependencies will likely have come from 'stable' there
isn't really a good way to determine that post-hoc, as the package
dependency resolution has happened elsewhere. Some TDEPS may have been
from 'unstable' depending on things like build groups and channel
overrides.

That would work to direct an upload; the bulk upload command could
choose another channel if it wanted, but would default to unstable in
any event.

That would also work to allow a manifest to be used as a seed
list. Not sure about the exact use case for that though.

One interesting subcase is using a list of manifests as a 'filter', to
indicate what fully quallified idents have already been fetched by a prior run, so
they won't be downloaded again.

One could imagine a manifest format that provide more detail than the
simplest list of fully qualified idents; for example you could try and provide
'tracability'; ident x in channel c for target t was resolved to fully
qualified ident y.

While this is worth thinking on, it's probably best to defer this for
a bit, and rely on a format_version tag to help us sort out what is
going on.

Potential formats

In the end I think this reduces us two possible formats for v1, with
a possible simplified format. I'm going to post them as separate
posts, please give +1/-1 on the choices.

Simple format: a flat file with no groupings, just a channel, target,
and ident list. Should we allow this in the interest of making the least complex case brief? Or does
that add too much complexity for the user.

One of the following:
Target grouping: Each target would have one or more groupings, and each group
would specify channel and a list of idents

'flat' grouping: A grouping tag 'section' would separate a set of channel, target,
ident list groupings. The actual tag chosen is arbitrary.

Universal to all formats:

# if not specified, defaults to latest version understood by habitat.
format_version = 1
# to be rendered during execution and in potential manifests
file_descriptor = "user specified data" 

[markan]

# Optional simple section
# target and packages are mandatory; if one appears, both
# should. Channel will have default of 'stable'
target = "x86_64-linux"
channel = "unstable"
packages = [ "core/foo", "core/bar" ]

Thumbs up/down if you think this should be included or not.

[markan]

# Target grouping. Channel defaults to stable
[["x86_64-linux"]]
channel = "stable"
packages = ["core/foo", "core/bar/3.2", "core/baz/2.71/20190910181446"]

[["x86_64-linux"]]
channel = "unstable"
packages = ["core/crazypants", "core/cliffs_of_insanity/0.0.1/20191010181446"]

[["x86_64-windows"]]
channel = "stable"
packages = ["effortless/audit-baseline"]

Thumbs up/down if you think this should be the format or not.

[markan]

# flat grouping. Channel defaults to stable
[[section]]
channel = "stable"
target = "x86_64-linux"
packages = ["core/foo", "core/bar/3.2", "core/baz/2.71/20190910181446"]

[[section]]
channel = "unstable"
target = "x86_64-linux"
packages = ["core/crazypants", "core/cliffs_of_insanity/0.0.1/20191010181446"]

[[section]]
channel = "stable"
target = "x86_64-windows"
packages = ["effortless/audit-baseline"]

Thumbs up/down if you think this should be the format or not.

[jeremymv2]

From my perspective, I think of artifacts in the following top down hierarchy:

• ARCHITECTURE -> CHANNEL -> ORIGIN/PKG_NAME

Therefore, I think your format suggestion here is 🚀

Considering that format, it will be important for the user to understand that if they specify multiple tables of the same architecture, double brackets [["architecture_name"]] are needed and not single ["architecture_name"] so that toml parsing yields an array of tables and not one single table.

[smacfarlane]

@smacfarlane I'm not quite understanding why the packages from the same origin have to come from the same channel, could you explain a bit more? :)

In the hab pkg bulkupload command, users can use the --channel option to specify the channel to upload the artifacts to (default: unstable).

That means the channels where the artifacts are downloaded from the target Builder specified in hab pkg download are not relevant in the target Builder specified in hab pkg bulkupload.

For your point about the graph conflicts when build, does it only matter in the target builder specified in the hab pkg bulkupload command?

@apriofrost Coming from the build side, consistency in the source channel is important to avoid dependency graph errors on builds. In the context of this flow, I still think mixing channels is a bad idea, and I would also advise users on upload to ensure that the groups they download move together on the consuming end.

After talking to @markan, though I do see the use for mixing channels. When packages in the channel are "leaf" packages, i.e. nothing depends on them and they are, at most, going to be wrapped once, then it's a relatively safe operation, though still makes me a bit nervous.

Specifically with the core origin on bldr.habitat.sh though, it does feel like we're setting our users up for a bad time if they download anything but the stable. The exception I can think of would be mirroring for our Biome friends, but it would be important to be able to replicate package->channel mapping.