-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix snowflake sometimes does not find tables #37620
Conversation
|
In theory this should fix the bug, but doesn't this imply we should eliminate usage of
|
(first (jdbc/query {:connection conn} | ||
(show-command-sql | ||
"DYNAMIC TABLES" | ||
:database db-name-or-nil :schema schema-name :like table-name)))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Based on the implemenation of show-command-sql
, providing :database
and :schema
should be mutually exclusive with each other. If a :schema
is provided, then :database
should not be provided and vice versa. If both are provided it would compile to SHOW DYNAMIC TABLES IN DATABASE <database> SCHEMA <schema>
which is invalid. I think that should be in the docstring or preconditions of show-command-sql
too, to prevent misuse.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've updatd to make it works for both cases
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good
On second thought, let's see if this actually solve people's problem in https://discourse.metabase.com/t/snowflake-broken-in-0-48-1/63849 @paoliniluis would you bind sending them the latest build? |
sure! thanks @qnkhuat! |
@qnkhuat when I test with the release branch I get schemas and tables as "show objects..." returns fine and then the "select true as ..." query also returns fine. When I test this branch, the "show objects..." returns fine but the "select true as..." queries fail with "Cannot perform SELECT. This session does not have a current database. Call 'USE DATABASE', or use a qualified name." same user, same permissions. The user and password is saved on our password manager as metabasesuccess |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just tested and does not return active tables
@@ -406,28 +405,62 @@ | |||
(sql-jdbc/query driver database {:select [:*] | |||
:from [[(qp.store/with-metadata-provider (u/the-id database) | |||
(sql.qp/->honeysql driver table))]]})) | |||
(defn- show-command-sql | |||
;; IMPORTANT: prefer using this command over using the JDBC `.getTables` method because the jdbc is buggy | |||
;; it works sometimes but other times randomly. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
;; it works sometimes but other times randomly. | |
;; it works sometimes but other times randomly does not. |
;; we have tried to use it but people reported that some schemas disappear on their instances. | ||
;; See https://discourse.metabase.com/t/snowflake-broken-in-0-48-1/63849 | ||
[object-type & {:keys [like database schema]}] | ||
;; in case the name is escaped for schema using escape-name-for-metadata, we undo that just to be sure |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm wondering if you really need to unescape stuff here, since you're doing a custom implementation maybe you can just not escape it in the first place? It's always better to not escape stuff in the first place than to escape it and unescape it, less moving parts and less ways to accidentally do something wrong
;; it works sometimes but other times randomly. | ||
;; we have tried to use it but people reported that some schemas disappear on their instances. | ||
;; See https://discourse.metabase.com/t/snowflake-broken-in-0-48-1/63849 | ||
[object-type & {:keys [like database schema]}] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Making this an mu/defn
and giving object-type
and :enum
schema would make it a lot clearer for people using this what kinds of things it supports. A docstring with an example usage would be helpful too
@@ -420,3 +420,15 @@ | |||
result-comment (second (re-find #"-- (\{.*\})" result-query)) | |||
result-map (json/read-str result-comment)] | |||
(is (= expected-map result-map)))))) | |||
|
|||
(deftest show-command-sql-test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(deftest show-command-sql-test | |
(deftest ^:parallel show-command-sql-test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall this looks really good to me, but I think we need to be super careful around escaping stuff and keep in mind that Snowflake identifiers are allowed to contain wacky characters https://docs.snowflake.com/en/sql-reference/identifiers-syntax#double-quoted-identifiers
I would be more comfortable if we just made sure we never called escape-name-for-metadata
in the first place (which should be easy enough since the one place that uses it is in the default implementation of describe-database
that we're replacing, and then made sure we're properly escaping quotes in that show-command-sql
function.
;; we have tried to use it but people reported that some schemas disappear on their instances. | ||
;; See https://discourse.metabase.com/t/snowflake-broken-in-0-48-1/63849 | ||
[object-type & {:keys [like database schema]}] | ||
;; in case the name is escaped for schema using escape-name-for-metadata, we undo that just to be sure |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems dangerous because it seems like technically a Snowflake identifier can contain slashes and underscores, like it's totally valid to have a table named My table\_2
. It seems like things are going to break in that situation.
https://docs.snowflake.com/en/sql-reference/identifiers-syntax
" " | ||
(cond-> [(format "SHOW %s" object-type)] | ||
(some? like) | ||
(conj (format "LIKE '%s'" like)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We probably need to escape like
, because AFAIK Snowflake table names are allowed to contain single quotes https://docs.snowflake.com/en/sql-reference/identifiers-syntax#double-quoted-identifiers
If you name your table something with a quote mark in it you're asking for trouble, but we should try to be careful and not explode even if people do dumb things.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually for that matter basically all of the identifiers here... are we sure that Snowflake doesn't allow double quotes in identifier names? Do we need to be escaping those?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh actually the dox do say you can create a Table with double quotes in its name
To use the double quote character inside a quoted identifier, use two quotes. For example:
create table "quote""andunquote""" ...creates a table named:
quote"andunquote"
(and database schema) | ||
(conj (format "%s" (str/join "." (remove nil? [(quote-and-unescape database) (quote-and-unescape schema)])))) | ||
|
||
(and (some? database) (nil? schema)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(and (some? database) (nil? schema)) | |
database |
You're just checking basic truthiness in the previous condition, not (and (some? database) (some? schema))
, so it's weird and risky to do the related checks using different logic. No need to make this more complicated than it needs to be
(conj "IN") | ||
|
||
(and database schema) | ||
(conj (format "%s" (str/join "." (remove nil? [(quote-and-unescape database) (quote-and-unescape schema)])))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(conj (format "%s" (str/join "." (remove nil? [(quote-and-unescape database) (quote-and-unescape schema)])))) | |
(conj (str/join "." (remove nil? [(quote-and-unescape database) (quote-and-unescape schema)]))) |
(format "%s")
doesn't really do anything
(and (some? database) (nil? schema)) | ||
(conj (format "DATABASE %s" (quote-and-unescape database))) | ||
|
||
(and (nil? database) (some? schema)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(and (nil? database) (some? schema)) | |
schema |
@camsaul what do you think about sending schema.table instead of database.schema.table when we do the simple-select-probe-query on sync? |
Thanks for the review, but I'll wait to see if this will solve people's issues before proceeding further. |
Closing in favor of https://github.com/metabase/metabase/pull/38160/files |
Fixes #38135
Fix https://discourse.metabase.com/t/snowflake-broken-in-0-48-1/63849
We changed to use getTables JDBC method to sync snowflake in a recent PR
because I thought using JDBC will be more reliable and it'll be consistent across the drivers.
But it turns out it's not; Snowflake driver has a history of being buggy like this, and it's biting us again. In fact, we have this comment from 3 years ago,
metabase/modules/drivers/snowflake/src/metabase/driver/snowflake.clj
Line 222 in ea22d29
I added back a comment with the word "IMPORTANT" to it, hopefully it won't get deleted this time.
I can't reproduce the bug and so is Luiggi.