Fix hive.hive-views.run-as-invoker handling with legacy view translation #14077

skrzypo987 · 2022-09-09T12:49:24Z

This is #12403 + product tests that should cover #12221 as well

Description

Non-technical explanation

Fix a bug when querying hive views with hive.hive-views.run-as-invoker set to true

Release notes

I am not sure release notes are required since this feature was not working in the first place

( ) This is not user-visible and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

...-product-tests-launcher/src/main/java/io/trino/tests/product/launcher/env/common/Hadoop.java

...resources/docker/presto-product-tests/common/hadoop/hive_with_run_view_as_invoker.properties

findepi · 2022-09-09T14:50:27Z

testing/trino-product-tests/src/main/java/io/trino/tests/product/hive/TestHiveViewsLegacy.java

@@ -150,6 +150,26 @@ public void testCurrentUser()
                .hasMessageContaining("Failed parsing stored view 'hive.default.current_user_hive_view': line 1:20: mismatched input '('");
    }

+    @Test(groups = HIVE_VIEWS)
+    public void testRunAsInvoker()


should this go into AbstractTestHiveViews ?

If I understand correctly, it makes sense only if legacy hive views are used.
I am not an expert here though so I may be missing something.

Is this relevant?

trino/plugin/trino-hive/src/main/java/io/trino/plugin/hive/ViewReaderUtil.java

Lines 88 to 95 in 6ceec25

if (isHiveViewsLegacyTranslation(session)) {

return new LegacyHiveViewReader(runHiveViewRunAsInvoker);

}

return new HiveViewReader(

new CoralSemiTransactionalHiveMSCAdapter(metastore, coralTableRedirectionResolver(session, tableRedirectionResolver, metadataProvider)),

typeManager,

runHiveViewRunAsInvoker);

I added a siimlar test in TestHiveViews, since the one from legacy would not work.

Why wouldn't it? what's the difference?

Views created in hive won't work. We need to create one using Trino.
BTW your comment reaction time is absurdly good.

It appears that you are correct. The error I got was something completely unrelated.
Moved the test to AbstractTestHiveViews

testing/trino-product-tests/src/main/java/io/trino/tests/product/hive/TestHiveViews.java

findepi · 2022-09-14T07:37:34Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/LegacyHiveViewReader.java

@@ -49,7 +49,7 @@ public ConnectorViewDefinition decodeViewData(String viewData, Table table, Cata
                        .map(column -> new ConnectorViewDefinition.ViewColumn(column.getName(), TypeId.of(column.getType().getTypeSignature().toString())))
                        .collect(toImmutableList()),
                Optional.ofNullable(table.getParameters().get(TABLE_COMMENT)),
-                table.getOwner(),
+                hiveViewsRunAsInvoker ? Optional.empty() : table.getOwner(),


I think similar should be done in the non-legacy reader https://github.com/trinodb/trino/pull/12221/files#r970434417

The previous behavior with hiveViewsRunAsInvoker==false is the same as it was before, and with true the owner should not be passed on. So I believe there is nothing to change here.

The PR changes LegacyHiveViewReader and this looks OK.
Doesn't it need to change io.trino.plugin.hive.ViewReaderUtil.HiveViewReader#decodeViewData too?

The non-legacy reader does already handle setting the owner correctly. HiveViewReader#decodeViewData initially returns with an empty owner, but sets an owner, if using DEFINER and if the owner exists, in HiveMetadata#getView.
However, I think we can also just streamline this in HiveViewReader#decodeViewData and not need the additional logic HiveMetadata#getView.

Looks like we smeared the responsibility across 3 places:

legacy HiveViewReader

non-legacy HiveViewReader

HiveMetadata#getView

and the result is quite inconsistent (hence questions like above)

Looking into this more, i no longer understand why the change here is desireable.
We have the table.getOwner() and it's some information, why do we need the LegacyHiveViewReader not to pass it further?

OK, I get it.
But why would we pass hiveViewsRunAsInvoker ? Optional.empty() : table.getOwner() here and
Optional.empty() in non-legacy HiveViewReader?

HiveViewReader and LegacyHiveViewReader should differ in how they translate view SQL (with coral or not), but they should not differ in how they translate authorization/ownership, or whether the produce INVOKER or DEFINER view.

That's part of the smeared responsibility that you mentioned. The non-legacy HiveViewReader passes an Optional.empty(), but afterwards HiveMetadata#getView will create another ConnectorViewDefinition object with the proper owner set, if needed. I feel the same, which is why I added the extra commit.
If there is agreement with this, perhaps we can add that commit to this PR as well.

For now, let's have

Suggested change

hiveViewsRunAsInvoker ? Optional.empty() : table.getOwner(),

Optional.empty(), // will be filled in later by HiveMetadata

and same in the other class.

Would that work?

No objections from me. I can confirm that the suggested change will also resolve the issue. What do you think @skrzypo987?

That would definitely be more consistent. Pushed the new version already

testing/trino-product-tests/src/main/java/io/trino/tests/product/hive/TestHiveViewsLegacy.java

findepi · 2022-09-19T18:56:57Z

Looks like CI failed for some transient reason
Please push an empty commit to have it rerun.

plugin/trino-hive/src/main/java/io/trino/plugin/hive/LegacyHiveViewReader.java

findepi · 2022-09-20T07:41:18Z

Squashed commits so that the fix and the test go together.
Also added some explanation to the commit message trying to capture conclusions from the conversation here and in #12403.
@skrzypo987 @joechu1 let me know if i did it right.

skrzypo987 · 2022-09-20T08:00:49Z

@findepi Squashed, see if You like the commit message

Prevent ViewReaders from passing view owner to ConnectorViewDefinition constructor, as it may fail the sanity check. The owner is added later on in HiveMetadata class Also add product test for `hive.hive-views.run-as-invoker` config option. Unit tests will not work here since we need to create a hive view not via Trino. Co-authored-by: Joe Chu <5282594+joechu1@users.noreply.github.com>

findepi · 2022-09-20T08:05:47Z

@skrzypo987 i used paste tense because was informing you about what i did.
but it looks you did the same, with similar words. Fine

i only improved the commit title and authorship again

skrzypo987 · 2022-09-20T08:28:17Z

@skrzypo987 i used paste tense because was informing you about what i did. but it looks you did the same, with similar words. Fine

i only improved the commit title and authorship again

Yeah, it appears that I forced pushed your changes Sorry about that. I shouldn't be allowed to use the --force option before the morning coffee.

cla-bot bot added the cla-signed label Sep 9, 2022

github-actions bot added the tests:hive label Sep 9, 2022

skrzypo987 requested review from findepi and joechu1 September 9, 2022 12:49

findepi reviewed Sep 9, 2022

View reviewed changes

skrzypo987 force-pushed the skrzypo/116-hive-view-run-as-invoker branch from f87aa99 to de3fd81 Compare September 12, 2022 08:03

skrzypo987 requested a review from findepi September 12, 2022 08:07

skrzypo987 force-pushed the skrzypo/116-hive-view-run-as-invoker branch from de3fd81 to 17ec1a5 Compare September 12, 2022 08:42

findepi reviewed Sep 14, 2022

View reviewed changes

skrzypo987 force-pushed the skrzypo/116-hive-view-run-as-invoker branch from 17ec1a5 to 2fbca89 Compare September 14, 2022 09:30

skrzypo987 requested a review from findepi September 14, 2022 09:31

findepi mentioned this pull request Sep 16, 2022

Ignore owner if translating Hive view in INVOKER mode #12403

Closed

skrzypo987 force-pushed the skrzypo/116-hive-view-run-as-invoker branch from 2fbca89 to 7b710c2 Compare September 20, 2022 07:00

findepi reviewed Sep 20, 2022

View reviewed changes

plugin/trino-hive/src/main/java/io/trino/plugin/hive/LegacyHiveViewReader.java Show resolved Hide resolved

skrzypo987 force-pushed the skrzypo/116-hive-view-run-as-invoker branch from 7b710c2 to 4e74ebc Compare September 20, 2022 07:34

findepi force-pushed the skrzypo/116-hive-view-run-as-invoker branch from 4e74ebc to 16d0e3f Compare September 20, 2022 07:40

findepi approved these changes Sep 20, 2022

View reviewed changes

skrzypo987 force-pushed the skrzypo/116-hive-view-run-as-invoker branch from 16d0e3f to 0715c75 Compare September 20, 2022 07:59

findepi force-pushed the skrzypo/116-hive-view-run-as-invoker branch from 0715c75 to 49faf54 Compare September 20, 2022 08:05

findepi changed the title ~~Ignore owner if translating Hive view in INVOKER mode + product tests~~ Fix hive.hive-views.run-as-invoker handling with legacy view translation Sep 20, 2022

findepi merged commit 2b77883 into trinodb:master Sep 20, 2022

findepi mentioned this pull request Sep 20, 2022

Release notes for 397 #14154

Closed

github-actions bot added this to the 397 milestone Sep 20, 2022

colebow mentioned this pull request Sep 20, 2022

Add Trino 397 release notes #14194

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix hive.hive-views.run-as-invoker handling with legacy view translation #14077

Fix hive.hive-views.run-as-invoker handling with legacy view translation #14077

skrzypo987 commented Sep 9, 2022

findepi Sep 9, 2022

skrzypo987 Sep 12, 2022

findepi Sep 12, 2022

skrzypo987 Sep 12, 2022

findepi Sep 12, 2022

skrzypo987 Sep 12, 2022

skrzypo987 Sep 14, 2022

findepi Sep 14, 2022

skrzypo987 Sep 14, 2022

findepi Sep 14, 2022

joechu1 Sep 16, 2022

findepi Sep 19, 2022

findepi Sep 19, 2022

joechu1 Sep 19, 2022 •

edited

findepi Sep 19, 2022

joechu1 Sep 20, 2022

skrzypo987 Sep 20, 2022

findepi commented Sep 19, 2022

findepi commented Sep 20, 2022

skrzypo987 commented Sep 20, 2022

findepi commented Sep 20, 2022

skrzypo987 commented Sep 20, 2022

	if (isHiveViewsLegacyTranslation(session)) {
	return new LegacyHiveViewReader(runHiveViewRunAsInvoker);
	}

	return new HiveViewReader(
	new CoralSemiTransactionalHiveMSCAdapter(metastore, coralTableRedirectionResolver(session, tableRedirectionResolver, metadataProvider)),
	typeManager,
	runHiveViewRunAsInvoker);

	hiveViewsRunAsInvoker ? Optional.empty() : table.getOwner(),
	Optional.empty(), // will be filled in later by HiveMetadata

Fix hive.hive-views.run-as-invoker handling with legacy view translation #14077

Fix hive.hive-views.run-as-invoker handling with legacy view translation #14077

Conversation

skrzypo987 commented Sep 9, 2022

Description

Non-technical explanation

Release notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joechu1 Sep 19, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findepi commented Sep 19, 2022

findepi commented Sep 20, 2022

skrzypo987 commented Sep 20, 2022

findepi commented Sep 20, 2022

skrzypo987 commented Sep 20, 2022

joechu1 Sep 19, 2022 •

edited