MessagePack should be developed in an open process

[cabo]

MessagePack has become highly visible on the Internet approximately
three years ago. These three years were exciting, with many new
implementations coming up. In these three years, there never was any
serious attempt to evolve the specification.

When I made my original BinaryPack specification proposal ("-00") in
October 2012, I tried to engage @frsyuki. After a little interaction
it seemed to me the MessagePack community wasn't prepared for any
change at all, and that wasn't going to change either.

So I thought, to address my requirements, I had to go for a fork, and
I submitted a "BinaryPack" specification that is not fully compatible
with MessagePack. For another three months, nothing happened. But
now that the MessagePack community has indeed woken up to some of the
new requirements, it may be time to think abut the best process to
address them.

I'm pretty sure that "design by twitter" (and now slashdot) is not the
best process. It is much better to write up complete draft
specifications so people exactly know what they are talking about.
Much of the discussion on the github issue fora has been by people who
haven't read what was written up so far (maybe because they didn't
even know where to look). And much of the thinking appears to be
formed outside the fora, because people come in with preconceived
notions that they don't bother to explain.

I think this is a symptom of a lack of openness. Before people
come back and claim that everything here is open, let me clarify this
term by pointing to http://open-stand.org/principles/.

The MessagePack community can work to evolve into an open community by
itself. Or it can try to work with an existing community that has the
process development already done. It should be no surprise that I'm
partial to choosing the IETF, because they are already doing some work
in this space, and it seems to work pretty well.

Akimichi: Your fears are unfounded. In the IETF, we know how to work
with other communities. Yes, it takes some time to adapt (on both
sides), and mistakes are always made. But in the end, it tends to
work. "Us vs. them" thinking is unnecessary. You become part of the
IETF by joining a mailing list, and the process is indeed completely
open.

It is by no means assured that the IETF is interested in picking up
this work. This depends on a willingness to come up with a clear
program of work, and to be open in earnest. I think the first point
is relatively easy, because the scope of the specification work
required is very narrow — this is not about a multi-year "MessagePack
2.0" effort.

Re the role of people in a potential IETF effort:

• Independent of any official roles, @frsyuki will be to MessagePack
  what Douglas Crockford is for JSON or Roy Fielding for HTTP. That
  doesn't mean he has to chair a WG or take on any other official role.
  His voice will be heard.
• Somebody has to do the editing of a specification. That goes faster
  if that person knows about the IETF process and how protocol
  specifications are done. If this becomes an IETF specification, that
  person becomes an "editor" and is bound by the decisions of the WG.
  As is the usual approach in the IETF, that person can make use of what
  is written up so far and develop the final specification making use in
  any way (including completely rewriting) any of it. It is not unusual
  that the originator of a technology also plays an editor role (usually
  as the first author on the list).
• I'm not sure this should be handled in a WG. I'd like to talk to
  other IETFers about this in two weeks Orlando ("Hallway Time"). If we
  want to do this in a WG, we'll need a WG chair. That doesn't need to
  be @frsyuki, and as it involves getting more familiar with IETF
  processes it may or may not be a good use of his time. All of the
  people questions can (and should) be discussed offline.

One personal comment: I don't care about leading any of this, I just
want the work done so I can use it for my real work. Right now I'm
just trying to push some people out of their cozy lethargy. Sorry,
pushing sometimes hurts. Some people really have to get over the "us
vs. them" thinking. @kazuho's approach of depersonalizing me as "an
IETF chair person" shows a level of disrespect I'm just not used to.
If you stopped thinking about defending an imaginary fort and focused
on finding a good way forward, it would be better for everyone. My
"-01" updated specification is mostly an attempt to rescind the idea
of a fork that was in "-00". But now "MessagePack is in danger!"?
(https://www.facebook.com/okukazuho/posts/10200154358053473) Indeed,
but certainly not from me or from the IETF. End of personal comment.

Process discussion must be out in the open, too, that's why I made
this ticket. At the moment, there doesn't seem to be any other place
to put down serious discussion, anyway. (The ticket may also help to
keep any process discussion out of the other tickets, before they get
overwhelmed like #121.)

[kazuho]

@cabo

First of all, it is was very disappointing to me that you submitted an early draft of MessagePack to IETF, even though others requested not to, including @frsyuki - the author of the protocol, and me.

It is also very sad for me that you opened this issue, I have explained to you many times why interoperability is so important for MessagePack (it is since it is used for storing data). But since I do understand the requirements people including you have (that is, to add new types), I wrote the proposal here #128 (comment) which guarantees forward compatibility while giving people to add new types.

I do not understand why a person (me) who actually makes such proposals should be referred to as

Right now I'm just trying to push some people out of their cozy lethargy.

My proposal might not be the approach you prefer, but if that is the case, please propose an alternative. Should such an approach seem better than mine, I am willing to hear, and I am also sure that @frsyuki and others will.

[najeira]

@cabo

We discussing on GitHub. See threads of 121 and 128.

Authors of msgpack libraries talks on Twitter is fact.
I guess, because they can use Japanese more than English very well.

After that, they posted their proposals to GitHub in English.
They replying to comments on GitHub too.

I think this is open.

How do you think this is not open?

[cabo]

@kazuho here we go again.

I submitted a draft of BinaryPack to the IETF ("-00") in October 2012 before the Atlanta IETF. I tried to engage @frsyuki. Nobody cared about the problem that was being solved.

Then suddenly github issue #121 exploded. Several proposals were made, and I used what made sense of these to fix my draft just in time for the Orlando IETF. I believe the resulting "-01" is closer to what we will end up with in the end than what "-00" is. Yes, -01 is more compatible to MessagePack than -00, but it is still an evolution of my October proposal. I used ample language in the abstract, in section 2.5 and other places to indicate there is potential for further change. Now what exactly were you disappointed about?

Re your other point, I indeed wrote up -01 to propose a specific solution (yes, you have to read it up to the Appendix to find it). Now how are you disappointed in that I did what you are asking for?

I think this whole thing is a great demonstration of the lack of a culture of openness I was talking about. Making my proposal visible in the IETF is not going to distract from the discussion here. In the best case, it is creating additional awareness that feeds into the discussion. In the worst case, it achieves nothing. But there is no danger of the IETF running away with your brain child. That's just not the IETF's style.

Another personal comment: I got a lot of hilarity out of the various side discussions about how to use IPR law to stop me from making my proposal available to the IETF. Wonderful demonstration about the lack of openness. (And it doesn't work: I'm not using the term MessagePack except in some factual statements, so there is no trademark violation. All text is mine, so there is no copyright violation. And MessagePack is simple enough that it isn't patentable, and there also wasn't an attempt to patent it at the time when such an attempt would have been possible.) But that is a side show.

The danger that MessagePack is exposed to right now is the "us vs. them" thinking, the thinking it is a crown jewel that needs to be protected from the unwashed masses in the outside world. The actual crown jewel is the wide consensus around it. Trying to protect the consensus by excluding/alienating dissenters doesn't work.

I'm still in the process of understanding and adapting to your way of working, and I'm making mistakes in the process. I'm asking for some forgiveness and some endurance while I learn. The fact that I'm learning doesn't make me "an IETF chair person"; I write code, too, you know. It also doesn't mean I have to adapt my style of working completely to yours.

(Final comment back on the technical level: Your proposal (if we are talking about the one written up in https://gist.github.com/frsyuki/5028082) is pretty good. My proposal may be an inch better, at least I believe so. In the end, either will work, and I'll be happy to support what the community agrees to do. After a good technical discussion. Let's do the technical work over in #128. Let's talk about process here.)

[repeatedly]

@cabo

I submitted a draft of BinaryPack to the IETF ("-00") in October 2012 before the Atlanta IETF. I tried to engage @frsyuki. Nobody cared about the problem that was being solved.

I didn't know your draft until you posted on github issue.
What the action did you take?

JFYI, there are no design discussion on slashdot :)

[cabo]

@najeira that's all great (and the language barrier is a fact that we just have to work with; I wish it would create less of an "us vs. them" syndrome).

I have experienced the lack of openness quite vividly. You know, I'm "the IETF chair person", not that guy with some thirty years of experience about protocol encoding and an open github issue on the msgpack-ruby implementation. And when I write up my version of the spec and make it available in a form that is more useful than a github gist (i.e., an Internet-Draft), that's putting "MessagePack in Danger". You can only get openness by thinking open. You need to embrace new members of the community openly instead of making fun of them or portraying them as the devil in the Japanese language media.

("You" is the figurative you here, of course, not you personally.
And yes, there have been some quite positive interactions here, too.)

I'll need to take a break from writing all this stuff here now — I don't have the time, but more importantly I don't want to dominate the discussion. So please go ahead, if needed I'll return tomorrow.

[cabo]

@repeatedly I only pointed the -00 out to @frsyuki.
I only found out last week that this repo is the place for the discussion. You live, you learn...

Now off for real...

[kuenishi]

TL;DR

[ugorji]

@cabo I appreciate that you are trying to help msgpack get standardized by the IETF. And I like that you are trying to make this whole process more open and engaging.

However, I think the approach you are using is lacking some tact.

There is a legitimate concern with the IETF getting pushed down the community's throat. To put it bluntly, it feels like the community is getting threatened that they better get on your bandwagon or you would submit a forked specification with your own name on it. From our point of view, there shouldn't be this pressure while we're trying to make the right decision on such a serious matter i.e. making an potentially incompatible change in an codec which has been out in the wild for a few years. No one knows all the libraries in use, or the heterogeneous environment people are using it in.

Given these very real concerns with your approach, the leaders in the community have been very polite in asking you to slow down on the IETF issue till the community is ready to take it on. That will come not during, but after a concrete proposal for this change has happened.

For your part, what you could do since you know some others in the IETF and spec/encoding communities is to circulate that proposal to them for their feedback, and let them know that you evaluated msgpack, and it is quite promising, and the community is finalizing on some extensions to handle strings and other future extensions in a compatible way.

My personal view is that the IETF is a very good and necessary thing for msgpack. However, the time to push that message across cannot be now, right in the middle of hashing out this. Let's get a proposal that we're all comfortable with, and initially engage the IETF indirectly till some major players in the community are comfortable.

Also, the personal attack on @kazuho was not in good taste. You're the new guy, and these guys have lived and bled msgpack for a few years.

[kumagi]

I think, twitter is open process platform too.
And issues like #128 are running open process.
They are already discussing in existing open process. This issue is waste of time.

[frsyuki]

@cabo said: #121 (comment)

Maybe I should add that we are looking at this space in the IETF. If we actually choose to standardize something, IETF will need change control, and change is likely. In other words, complete backwards compatibility is somewhat unlikely. I'd prefer to shoot for the best standard for a wider audience (i.e., not just focused at RPC, but for just about everything JSON is used for today).
If that is not possible within the msgpack community, it seems that forking is the best choice.

#121 (comment)

BTW, I don't want to "promote msgpack" either, I want to solve a problem, and msgpack seemed to be 95 % of the solution. IETF would need to have change control (and thus the power to break compatibility) in any case, if they want to pick it up.

#121 (comment)

If (big if!) this ever becomes an IETF WG document, then the WG chair (likely not me; I'm chairing other WGs) will select a document editor

#121 (comment)

I'm sure if you want to be the editor of the msgpack draft this will be highly welcome, because this is your work.
Remember, however, that such a position comes with the responsibility to carry out changes as the WG decided. It has happened in the past that an editor and the WG disagreed to the extent that the editor stopped editing or even made unwanted changes; in such cases a new editor is appointed

• OAuth 2.0 and the Road to Hell http://hueniverse.com/2012/07/oauth-2-0-and-the-road-to-hell/

#121 (comment)

Due to the timing in the runup to the Orlando IETF, I will have to submit the final version of the Internet-Draft on Monday. So I would be happy if we could get a bit of a closure on the string representation issue here until then.

@kiyoto said> Is there any extrinsic reason you need to get this done this year?

@cabo said: #121 (comment)

the standardization of Smart Object Networking (Internet of Things, IP Smart Objects, whatever you want to call it) is happening now.
I believe adding a msgpack-like component to it would create an important basis for other standards to build on

I said: #121 (comment)

Finding dependable person who kindly spends time for an open source project seems hard.
I think I have following 3 options but none of them seems excellent idea:

1. I'll be the editor (hard to spend time)
2. I try to found a dependable person for an editor (takes time)
3. Just say "don't propose msgpack as a draft" (result in confusing two specs)

@rasky said: #128 (comment)

it's the new msgpack specification that mandates that "raw" will become "string". This means that, in @frsyuki 's view, there is no big backward compatibility problem to be handled with this transition, as most msgpack users will have utf-8 strings stored in existing messagepacks. What I can't understand is why we should first draft a specification that breaks compatibility (by deciding that "raw" becomes "string" for existing msgpack), and then we need to implement a workaround at the bindings levels, thus making all bindings make the wrong thing by default (that is, by default they would implement the msgpack specification wrong). It looks a completely wrong plan to me.

I said: #121 (comment)

What I want to do from this time regarding this string type issue is following process:

1. I'm about to propose an idea to change the spec
2. Active authors of implementation projects implement the idea
3. They release that implementation as an experimental release (could be an internal release for their use case)
4. Active users try the implementation and validate how it works
5. If the proposal needs fix, fix it. This fix may include changes of the proposed format
6. Iterate 1 to 4 again until there're enough knowledge
7. Release the already implemented experimental release officially and the proposal as the official release and official spec

• Interview with Ian Hickson, HTML editor http://html5doctor.com/interview-with-ian-hickson-html-editor/

[rasky]

FWIW, I'm finding that the current development process of adding a new "string" type is being fully made in public. Not only @frsyuki did listen to our cry to add such a type and changed his mind, but he proposed a process that is totally transparent, iterative, and open to feedbacks.

I understand that previously the process wasn't fully public, but it looks like it has now changed.

[lestrrat]

@cabo There was hesitation to put up a draft so early (repeatedly stated), and you did it anyway. And you complain when people express their discontent with your actions, and further blames the culture and people doing the hard work?! ....And now you think that people will want to try to work with you? That is certainly and interesting train of thought.

[hayamiz]

I'm pretty sure that "design by twitter" (and now slashdot) is not the
best process. It is much better to write up complete draft
specifications so people exactly know what they are talking about.
Much of the discussion on the github issue fora has been by people who
haven't read what was written up so far (maybe because they didn't
even know where to look). And much of the thinking appears to be
formed outside the fora, because people come in with preconceived
notions that they don't bother to explain.

I think this is a symptom of a lack of openness. Before people
come back and claim that everything here is open, let me clarify this
term by pointing to http://open-stand.org/principles/.

I'm versy surprised that "openness" forbids casual conversations with friends.

[ugorji]

@cabo @frsyuki and all others.

My previous comment only touched on the pressure to go the IETF route at this current time.

Having said that, @cabo fundamental points are solid:

• using twitter, slashdot, bug tracker (ie github) do not constitute an open process.
  For example, I only found out about this discussion because I did a search for
  msgpack string support as I was struggling with how to handle it better in my codec,
  and saw it had exploded over the last 7 days (thanks to Google).
• we need a way that all interested parties can be involved, and not have to find out things when they change.

Let's get back to these fundamental points he raised.

I think the folks here are understandably cautious about using the IETF. However, we still need a not-so-informal process (semi-formal) that we can commit to and which is scalable. This semi-formal process could simply leverage:

• a discussion forum (e.g. google groups), maybe called msgpack-devs.
  This supports real threaded conversations. This would allow the core maintainers, library authors, and other users with vested interests (e.g. Pinterest, redis folks) to hash out issues before exposing to all casual users (members of the msgpack users forum).
• A place to store and view documents (e.g. proposals, etc). I propose using a Google Drive shared folder. Write access granted to members of msgpack-devs, Read access public.
• bug tracking, source code, etc stays on github.

Thoughts?

[kazuho]

@ugorji
I totally agree with you that it is hard to find issue #128, and I think there should at least be a direct link from msgpack.org considering the impact changes to the specification would cause (and not rushing to finalize the new spec. would also be a good idea IMO).

Regarding where the discussion should be made, I prefer staying on the github issues (although I would not mind a lot if we moved to Google groups). The reason is because we can use markdown. Examples / code snippets are powerful tools to express one's ideas regarding discussion of software development, and I would like to continue using markdown.

Regarding where the documents should be stored, I do not care where it would be, as long as there is a permanent link from the official discussion (i.e. github or Google groups). IMO it would be preferable that such documents, should be version-controlled so that others could track the changes, and to that aspect Gist is my preferred location.

PS. @ugorji I think for these specific issues regarding which tool we should use to make discussions it would be better to open a separate issue, since this issue started as a discussion point if the discussion should be done in IETF. How do you think?

[cabo]

I now have to agree with @najeira that the process has become much more open since the "#121 explosion". Organizations often grow as a result of significant events. (The IETF became as open as it is today only after what is known as the "Kobe revolution".)

But let's not close this ticket just yet, but instead use it as a place for fine-tuning the process.

Two issues have come up:

• outreach. We must engage both today's and potential future communities. I think the actual venue of discussion is less important, as long as this discussion is openly advertised. It must be allowed to pull in people to the discussion. This is not the place to ask for some additional "smoke-filled backrooms" that aren't open — we may not be able to (or even want to) prevent them happening, but the results of these backroom discussions have to undergo the same technical merit check that every other input undergoes. With respect to potential future communities, we may in the end decide there is no way to address their requirements while keeping the current community happy, but this should be a conscious decisions after the input was processed.
• communication and document handling. The github issue forum format works for me, with the possible exception of not having threading. We may want to have multiple threads as multiple issues, and may even want to open a new repo to facilitate that. I also agree with @frsyuki that documents need to be version-controlled, and that markdown is exactly the right technology to foster real collaboration. As long as we do enough outreach, it doesn't matter whether the venue is obscure. But there is a dark cloud. Because I didn't want to give people bad ideas, I haven't disclosed what really pushed me to submit the -01 to the IETF after the -00: getting a timestamped archival record. The significance may not be abvious to most people here — not many will have fought patent issues in their professional life. Without going into all the details: This is an essential element of a standards process. We might simply hijack the IETF process for this single purpose and submit a new Internet-draft each time there is technical progress (that is the normal thing to do in the IETF anyway). Trying to set up our own subpoena-proof time-stamped archival record wastes too much time and money.

[kuenishi]

TL;DR

[ugorji]

@kazuho

Regarding opening a different issue on which tools to use for open discussions, I'm fine either way. I think @cabo is partial to continuing in this issue, so unless others disagree, we should honor that.

@cabo

It's good to see you back. I think I speak for more that just myself that your insight, experience and expertise in this matters is highly highly valued.

In response to tools to use, let me try to make a case for benefits of groups and online document drive.

Google Groups vs GitHub Issue

• Exposure to more than just developers.
• Easy way to look at just discussions (github has discussions interspersed with bugs and changelists. You need to know the)
• Threading: Biggest value. Easier to follow divergent discussions, replies, etc.
• More natural for discussions

Online Drive (e.g. Google Drive) vs GitHub

• Richer formatting options (with possible in-line reviews)
• Automatic micro and macro versioning
• Public, Restricted or Private support. It allows 3 folks collaborate on an idea, and then release it to more people when it's good enough to share.
• One link (shared drive) allows you see all the documents. No need to update a website to include the links whenever documents are created.

I've been involved (using the word loosely) with Go Language development since release in late 2009, and have seen this combination work extremely well.

My choice for tools is:

• Google Groups for discussions. Keep msgpack-users for global users, and msgpack-devs for those that want to be involved while things are getting hashed out.
• Google Drive (for documents)
• GitHub Issue Tracker
• GitHub Gist (for code snippets, etc)

Having said this, the best model is always one that the group is familiar with and still gives the same returns. I'd be fine with whatever the group chooses (as long as it's not twitter ;))

[frsyuki]

Thank you for taking time for msgpack.
Yes, I'll comment on the tools later.

@cabo

I can't find obvious reasons that I need to require existent devloppers to switch the actual venue of discussion to IETF:

a) The new specification we're now discussing on issue #128 is not implemented yet, or not validated in the real production environments as I mentioned

b) The standardization in IETF requires one or more of MessagePack developers to work hard on creating the standard itself as you mentioned

I'm sure if you want to be the editor of the msgpack draft this will be highly welcome, because this is your work.
Remember, however, that such a position comes with the responsibility to carry out changes as the WG decided.

c) We're having open discussion on the github issue as @najeira mentioned

d) Changes which are incompatible with current spec is not acceptable while IETF likely does as you mentioned

Maybe I should add that we are looking at this space in the IETF. If we actually choose to standardize something, IETF will need change control, and change is likely. In other words, complete backwards compatibility is somewhat unlikely. I'd prefer to shoot for the best standard for a wider audience (i.e., not just focused at RPC, but for just about everything JSON is used for today).

I think we have a consensus that there're no obvious reasons to switch the actual venue of discussion to IETF, and msgpack community should keep disucussion.
On the other hand, I would need to make my attitude clear about this standardization issue, before Orland IETF.

Thus I propose as following about this issue:

1. msgpack community keeps discussion, implementation, and validation of the specification and implementations
2. repeat 1. and work out the new specification documents
3. we validate whether the implementation work well and achieve knowledge
4. msgpack community keeps improving the documents based on the validation and knowledge
5. msgpack community feedback the result to the standardization process working on IETF if it's happening
6. and IETF feedbacks their opinions to msgpack community
7. repeat 5. and 6.
8. we finally can have a RFC

Please don't get me wrong. This my proposal doesn't actively prevent you from creating a standard.
My points are that the msgpack community should keep discussion, and discussions on IETF and msgpack community don't conflict if they're consistent.

Therefore, I suggest as following to @cabo:

• you don't add changes which are inconsistent from the spec achieved by a result of consensus of MessagePack developers including me
• you don't propose the draft into RFC/ISEG at the moment when MessagePack developers including me don't reach a consensus to do that
• you don't use the name "MessagePack" or something suggesting similarities with MessagePack unless MessagePack developers including me agree that

Even writing a specification of MessagePack and submit it as an individual Internet-Draft under my name is one of my options.
Sooner or later, we need to have a clear document about MessagePack spec (which includes the String/Binary types).

I hope that you add positive comment on my proposal, @cabo. Thank you.

[kazuho]

@frsyuki
Kudos to you for the wise proposal. I totally support this.

Personally I have no objection to @cabo's action if he accepts all of the three requests. I do respect his expertise in the area (for both the technical knowledge and the experience in standardization, it would be great to work with him), and am more than willing to help his efforts on standardization if he agrees to work unanimously with @frsyuki and the community.

[tokuhirom]

@frsyuki +1 (from the commiter of the perl binding)

[cabo]

@frsyuki
Thank you, that is a clear message I can take to the upcoming discussions in Orlando.

There certainly will be no premature action at the IETF, so there is enough time to do the technical work for consensus on a solid solution here.

In this ticket (or any others that we create from it), we can fine-tune procedural aspects of that work (such as: how do we write up the proposal in its various stages) in parallel to the actual technical work, as required.

[yappo]

@frsyuki +1 (from the author of the O/R Mapper binding)

[xaicron]

@frsyuki +1 (from a heavy user of the perl library)

[kzk]

@frsyuki +1 (as Treasure Data CTO, where we store customers' hundreds of billions of records in MessagePack)

thanks for your 'crystal clear' write-up.

[cabo]

I promised that I would make the sources to my Internet-Draft available for anyone to pick up the work if desired. I finally got around to doing this. Enjoy at: https://github.com/cabo/I-D

[frsyuki]

the topic about tools should be done on another ticket.

[maisiliym]

Therefore, I suggest as following to @cabo:
[...]
you don't use the name "MessagePack" or something suggesting similarities with MessagePack unless MessagePack developers including me agree that

This is bad strategy; lost opportunity for a big boost in community size. Instead, the project became fragmented. If CBOR was seen as an extension to msgpack (which it is, essentially), it could help the communities come together, and avoid a lot of duplication. Most of the CBOR/msgpack implementations are very similar; CBOR adds more types, which can be fitted into 'lower' msgpack types. Suntzu advised against fighting battles and making ennemies; the opposite strategy yields superior results. Musashi gave similar advice as well.

If some people are interested, they could come forward and a strategy could be formulated to unify as many cbor/msgpack projects as possible. Those who are not interested will be tuned out of the commenter's thinking as thoroughly as possible.