Community feedback process (e.g. ZEP)

[joshmoore]

With recent discussion around:

• How widely is column-major supported? community#41
• Add Sharding Support zarr-python#877
• Multiscale convention zarr-specs#125

and even older discussions like:

• v2: Standardizing .zmetadata zarr-specs#113

there is a clear need to have a process for moving specification modifications, extensions, and even conventions forward in a timely fashion.

The goal of this issue is to identify an appropriate process for the Zarr community and to encode it in this repository and potentially elsewhere (for example in zarr-specs or one or more new, dedicated repos). Existing processes that we would like to evaluate include:

• PEP
• NEP
• STAC
• ...

In addition to existing processes, a proposal that has currently been floated is to focus less on a voting mechanism and more "statements-of-intent" from the maintainers of Zarr implementations. If a specification, extension or convention clearly won't be adopted by a large number of implementors (or they are just skeptical that the developer capacity exists to implement ir) then the community should be wary of saving data with such changes.

Other potential stakeholders include:

• User communities (Pangeo, OME, ...)
• ...

[rabernat]

I think this is a great idea! Adding some links which define the various processes mentioned above so people can read further:

• PEP Purpose and Guidelines
• NEP Purpose and Process
• STAC development and release process / STAC principles

It would not be a bad idea for @zarr-developers/steering-council and anyone else interested in this to spend 30 minutes reading the above documents so we can get on the same page.

One important distinction we should make is enhancements to the Zarr spec vs enhancements to the various Zarr implementations. We have not always separated these two cleanly, and that has led to some challenges.

Of all of the above, I personally think the STAC process is most directly applicable to Zarr.

Whatever we choose, the hard part will be getting sufficient community engagement for the process to move forward. Hopeful that @MSanKeys963 can help with this!

[joshmoore]

Chatting today @rabernat (rightly) assigned the rest of the @zarr-developers/steering-council the homework of having read the linked STAC resources for the next discussion. After a first read, a few things occurred to me:

• The current semver-based STAC version is 1.0.0 which I understand to mean that the process of a major breaking change hasn't been tested yet.
• STAC applies if I'm reading correctly only to JSON documents. I have some concern that a binary protocol may have more difficult dealing with changes.
• The process requires that any change to the spec "must have two positive reviews and no negative reviews." which matches roughly zarr-python's API change process, and still need to deal with communicating the adoption of the changes.

None of that is to say that it doesn't look to be incredibly well-done nor that it couldn't serve as a great basis. Just looking to identify potential issues.

[MSanKeys963]

I think we can safely close this issue.
CC: @joshmoore