Snowflake support does not include list features #2280
Comments
|
@adchia @woop here is the docs related to semi-structured data in Snowflake: https://docs.snowflake.com/en/sql-reference/data-types-semistructured.html#array Arrays for list feature types are possible in Snowflake, but there is no enforcement of data type within the array, its stored as type VARIANT, where VARIANT, can store values of any other type, including OBJECT and ARRAY, up to a maximum size of 16 MB. Snowflake does not currently support fixed-size arrays or arrays of elements of a specific non-VARIANT type. Im curious to your initial thoughts on this. We can certainly test an array if it complies with a certain data type that is supported or throw an error. If it does then execellent, if not we throw an error to remove the feature |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
I think we can take the approach described by @sfc-gh-madkins |
sfc-gh-madkins
added
kind/feature
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
bump |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
bump
…On Tue, Apr 25, 2023 at 8:30 PM stale[bot] ***@***.***> wrote:
This issue has been automatically marked as stale because it has not had
recent activity. It will be closed if no further activity occurs. Thank you
for your contributions.
—
Reply to this email directly, view it on GitHub
<#2280 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ATSRCU5DJDXD5YTTP23ET2LXDB3FFANCNFSM5NPKRUQQ>
.
You are receiving this because you were assigned.Message ID:
***@***.***>
|
|
Looks like we'll need this for our implementation so I'll be taking a look at implementing this in our sprint starting tomorrow. |
|
@JohnLemmonMedely I am afraid snowflake does not support structured types on the server side. All semistructured types are currently returns as strings. We are working to support this in the next couple of quarters. |
|
@sfc-gh-madkins I was planning on using the method you suggested earlier and attempt to cast them to what they need to be and throwing errors when we can't. It'll be better to guarantee data types via Snowflake but until that's possible I can enforce those guarantees myself via the data pipeline.
|
|
will look forward to the PR :)
…On Tue, Sep 5, 2023 at 4:51 PM John Lemmon ***@***.***> wrote:
@sfc-gh-madkins <https://github.com/sfc-gh-madkins> I was planning on
using the method you suggested earlier and attempt to cast them to what
they need to be and throwing errors when we can't. It'll be better to
guarantee data types via Snowflake but until that's possible I can enforce
those guarantees myself via the data pipeline.
We can certainly test an array if it complies with a certain data type
that is supported or throw an error. If it does then execellent, if not we
throw an error to remove the feature
—
Reply to this email directly, view it on GitHub
<#2280 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ATSRCU3WGHHQXHAVBAE467LXY6NH7ANCNFSM5NPKRUQQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@sfc-gh-madkins I had to pause my work on the PR but we were able to use my fork in production. I'm trying to see if it makes sense at this point to get the PR ready to merge or if I should just wait on Snowflake adding structured types instead. Is that still on the Roadmap to add early next year? |
|
let me ping to see where this is at
…On Fri, Dec 8, 2023 at 11:41 AM John Lemmon ***@***.***> wrote:
@sfc-gh-madkins <https://github.com/sfc-gh-madkins> I had to pause my
work on the PR but we were able to use my fork in production. I'm trying to
see if it makes sense at this point to get the PR ready to merge or if I
should just wait on Snowflake adding structured types instead. Is that
still on the Roadmap to add early next year?
—
Reply to this email directly, view it on GitHub
<#2280 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ATSRCU4FFUY62RAO4MRN3XTYINGNJAVCNFSM5NPKRUQ2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBUG42TOOBYGEYQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@sfc-gh-madkins just following up on if Snowflake still plans on supporting structured types soon. I just need to know if work will start on it over the next quarter or not. |
|
negative. this is continuing to be pushed out due to lack of customer
demand. however, we do now have vector support capabilities for both fixed
and floating point 32 bit numbers, however, the client arent ready yet
…On Fri, Dec 15, 2023 at 10:09 AM John Lemmon ***@***.***> wrote:
@sfc-gh-madkins <https://github.com/sfc-gh-madkins> just following up on
if Snowflake still plans on supporting structured types soon. I just need
to know if work will start on it over the next quarter or not.
—
Reply to this email directly, view it on GitHub
<#2280 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ATSRCUZRMMJRFCX2LJLWSOLYJRY5JAVCNFSM5NPKRUQ2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBVHAYTEMRVHAZA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Expected Behavior
Snowflake offline store works with list features
Current Behavior
Right now Snowflake produces errors when using list features. This isn't caught by tests because of this skip with the aws provider: https://github.com/feast-dev/feast/blob/master/sdk/python/tests/integration/registration/test_universal_types.py#L39
The text was updated successfully, but these errors were encountered: