[Format/C++/Java] Removing field layout from Schema.fbs breaks backward compatibility

[asfimport]

Change in ARROW-1785

Breaks backward compatibility with schema generated in prior versions of Arrow, as VectorLayout field was removed from Field table.

This is definitely not supported behavior in FlatBuffers.

I suggest to put back following field

layout: [ VectorLayout ] (deprecated);

 into 

table Field 

and mark it "deprecated" - as shown above

It will also require putting back 

table VectorLayout

which can have all the deprecated fields as well.

code generation footprint will be minimal in this case, as FlatBuffers won't generate getters for deprecated fields, but it will really solve backward compatibility issues.

Also may be a good idea to introduce "version" into Schema.

 

 

 

Reporter: Yuliya Feldman / @yufeldman
Assignee: Yuliya Feldman / @yufeldman

PRs and other links:

• GitHub Pull Request #2263

_{Note: This issue was originally created as ARROW-2843. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
I'm -0 to -1 on this change. We do already have a metadata version number.

We've made no guarantees of backwards compatibility in the columnar format metadata yet, so we have broken no contracts with users.

To be honest, I'm not pleased that there are still a number of underspecified parts of the columnar format and metadata. As an example, I proposed a more complete Interval / Timedelta type about a year ago and these changes have still not been implemented.

I would very much appreciate if Dremio and other parties depending on Arrow in production would work more proactively with me and others in the Arrow community to reach a point where we can declare a stable 1.0 version of the columnar format and its metadata. In particular, if any more breaking changes are required (I hope that there will not be), that we make them as soon as possible to minimize disruption to applications.

[asfimport]

Yuliya Feldman / @yufeldman:
Thank you @wesm  for very quick reply and thoughts.

I do understand about "no guarantees" part, but why not do our best to maintain backwards compatibility, especially when it is possible.

 

 

[asfimport]

Wes McKinney / @wesm:
In my opinion, we haven't worked hard enough as a community to stabilize the metadata. We have failed to make progress since mid-2017, why is that? I have been trying consistently to advance to a 1.0 release; can we not work together to finalize the binary protocol?

[asfimport]

Yuliya Feldman / @yufeldman:
I truly appreciate your opinion on general direction and collaboration. I just don't feel it should be part of this JIRA.

My main concern is backward compatibility when we can do it. 

[asfimport]

Yuliya Feldman / @yufeldman:
I am going to resolve this Jira as "won't fix", as it will create issues with 0.8, 0.9 versions around backward compatibility (though we did not promise it yet).

Just for the future we have to be much more strict regarding changes in "fbs" and do not allow fields removal - make them deprecated at least.