drop the hard dependency on duckdb-substrait-extension?

[benbellick]

In working on #95, I have encountered some test failures due to the duckdb extension not recognizing the field "extensionUrns" in substrait plans. This means that to accomplish #95, a change must be made to the dependency https://github.com/substrait-io/duckdb-substrait-extension (i.e. substrait-io/duckdb-substrait-extension#164).

Rather than tightlyncouple the two, we could consider removing the dependency on the extension here to make development in substrait-python easier. That being said, I'm not sure this is the right decision, but wanted to open up the discussion.

[tokoko]

it's currently used for sql conversion testing only (along with datafusion). Those specific tests aren't that essential, but tbh I feel like any change that produces an output that messes up duckdb and datafusion integration are probably not the best to make. I'm a little puzzled why duckdb fails on urn changes though.. Wasn't that specifically designed to be backwards-compatible? Is there something specific in duckdb extension that does unnecessary extra checks?

[benbellick]

I haven't done a completely thorough analysis, so it may be possible to work around. The case that this came up in was when submiting substrait json plans via from_substrait_json which contained only extensionUrns rather than extensionUris.

The error I received was about the field extensionUrns not being recognized. Perhaps the error would go away if I submit plans containing both extensionUris and extensionUrns, or it may be the case that the JSON deserialization in the duckdb extension is too restrictive and fails on deserialization of JSON which contains unrecognized fields.

I will investigate and report back.