Document table function support #12338

kasiafi · 2022-05-11T11:34:20Z

This is documentation for Table Functions #11336.
The feature is already in.

kasiafi · 2022-05-11T11:37:26Z

@martint @mosabua could we try to merge the docs before the upcoming release so that the new feature goes live already documented?

findepi · 2022-05-11T12:40:38Z

docs/src/main/sphinx/sql/table-functions.rst

+    public Optional<TableFunctionApplicationResult<ConnectorTableHandle>> applyTableFunction(ConnectorSession session, ConnectorTableFunctionHandle handle)
+    {
+        // get the ConnectorTableHandle and a list of ColumnHandles representing the table function result
+
+        return Optional.of(new TableFunctionApplicationResult<>(tableHandle, columnHandles));


Let's not copy code into docs to avoid going out of sync. Pointer to a method (interface method name to implement) should suffice and then that method javadoc should describe the technical details

In this case, I can indeed remove the pasted code, as it doesn't add value to the docs.
However, I'd like to leave the other code samples (the constructor and analyze()), even at the risk of going out of sync, because they provide a convenient way of explaining things. Anyway, I hope I won't miss the docs update if the SPI changes in the future.

mosabua

First pass..

docs/src/main/sphinx/sql/table-functions.rst

docs/src/main/sphinx/sql.rst

docs/src/main/sphinx/sql/table-functions.rst

kasiafi · 2022-05-12T09:03:47Z

@mosabua thanks for your feedback.

I split the document in two, one from the user's perspective, and one from the developer's perspective.

The "user guide" is linked from select.rst, and it is in the functions folder. We will add there a list of built-in functions when we have them.
The "developer guide" went to develop.

The changes are in a separate commit.
Please take a look.

mosabua · 2022-05-13T00:21:28Z

docs/src/main/sphinx/develop/table-functions.rst

+===============
+Table functions
+===============
+


Need an intro sentence about table functions .. what is it? And why would I use it..

this could just link to the user docs actually

mosabua

Needs a bunch of clarifications to allow users to actually use this

mosabua · 2022-05-13T00:22:04Z

docs/src/main/sphinx/develop/table-functions.rst

+===============
+Table functions
+===============
+


this could just link to the user docs actually

mosabua · 2022-05-13T00:23:14Z

docs/src/main/sphinx/develop/table-functions.rst

+Table function declaration
+--------------------------
+
+Trino supports adding custom table functions. They are declared by connectors


So ... thats kinda weird .. so users have to maintain a fork of the connector? It cant be a separate plugin that is somehow associated with a specific connector?

No, it doesn't require a fork or a separate plugin. Any existing connector can expose table functions. Soon we'll have the query pass-through feature, which is a good examples of (existing) connectors registering polymorphic table functions.

Generally, I believe that we will be adding a lot of custom table functions to Trino. Of course, we will have to judge if the function is generally useful. If someone need a function that we find "controversial", they might have to fork.

okay ... maybe we should say something like.. want another table function added to trino .. file a ticket or PR and contact us ... although thats kinda odd for the docs.. not sure at this stage how we should proceed .. if there are not currently usable table functions maybe we should merge..

Yeah, we don't write things like "want another connector...file a ticket..." in the docs.

At the moment, we have a bunch of table functions in review. They should be in soon. When we merge them, we'll document them, and they will serve as a good example of what people have to do if they want a new table function in Trino.

Also, we will likely have some publications around table functions.

mosabua · 2022-05-13T00:24:51Z

docs/src/main/sphinx/develop/table-functions.rst

+Trino supports adding custom table functions. They are declared by connectors
+through implementing dedicated interfaces.
+
+To declare a table function, you need to subclass ``ConnectorTableFunction``.


we probably need to add a whole bunch more info about the mechanics .. like separate module or fork of specific connector, pom.xml, dependencies?, what about package structure and so on..

I added another sentence about passing the custom implementation through a Provider. It is all that the implementor must do. Also, this is a pretty common pattern of exposing things from connectors in Trino.

I thought the user has to do the development .. and that is not the case it seems.. so it just goes into existing connectors .. we should mention that somehow

When thinking of it, I can identify at least four categories of table functions, and they will be added in different ways:

functions that we will add to the existing connectors. The query pass-through capability is a good example, because it is specific to the connector: you pass different syntax to Elasticsearch than to Teradata.

functions that users add to their own connectors (forked). Generally, these are functions too controversial or too specific to be included in Trino

"global" functions, which will be built-in in Trino, but they don't belong to any specific connector. Example: the pivot capability. We will support them only when we have the dedicated Operator

I think that having table function support might result in creating a whole new kind of connectors. Currently, we have connectors over different data sources. Now that we support adding table functions, it makes sense to connect to an outer system for its processing capabilities and not necessarily for the data. Such connector could expose table functions that give access to the outer system's abilities or resources, so that they can be used for processing data in Trino.

@mosabua could you help me with the wording? The problem is, the table functions can go both to the existing and new connectors. I can't predict what kind of use case will be dominant.

mosabua · 2022-05-13T00:25:46Z

docs/src/main/sphinx/develop/table-functions.rst

+- returned row type
+
+In the example, the returned row type is ``GENERIC_TABLE``, which means that
+the row type is not known statically, and it will be determined dynamically


and it is determined

mosabua · 2022-05-13T00:26:10Z

docs/src/main/sphinx/develop/table-functions.rst

+
+In order to provide all the necessary information to the Trino engine, the
+class must implement the ``analyze()`` method. This method is called by the
+engine during the Analysis phase of query processing. The ``analyze()`` method


the analysis phase

mosabua · 2022-05-13T00:30:21Z

docs/src/main/sphinx/develop/table-functions.rst

+
+The access control for table functions can be provided both on system and
+connector level. It is based on the fully qualified table function name,
+which consists of the catalog name, the schema name, and the function name.


in the syntax of ?

catalog.schema.function

I assume.. but spell that out..

mosabua · 2022-05-13T00:32:58Z

docs/src/main/sphinx/functions/table.rst

+the catalog. You can qualify the function name with a schema name, or with
+catalog and schema names::
+
+    SELECT * FROM schema_name.my_function(1, 100)


really.. I assume that only works if you specified the current catalog with USE
does that also work for catalog.schema ?

Yes, we now support path resolution (#11476).

mosabua · 2022-05-13T00:33:34Z

docs/src/main/sphinx/functions/table.rst

+
+Otherwise, the standard Trino name resolution is applied. The connection
+between the function and the catalog must be identified, because the function
+is executed by the corresponding connector. If the function is not registered


is not registered ... that begs the question .. is there anything I have to do as a user?

Hm... that probably means the user misspelled something or they don't have the right catalog on the path. How could I word it?

mosabua · 2022-05-13T00:34:45Z

docs/src/main/sphinx/functions/table.rst

+within the SQL query. They can be used for working with external systems as
+well as for enhancing Trino with capabilities going beyond the SQL standard.
+
+Trino supports adding custom table functions. They are declared by connectors


we need to add info about what table functions are there for each connector .. even if that is none

and then be explicit about linking to the dev guide to figure out how to add any

mosabua · 2022-05-13T00:36:17Z

docs/src/main/sphinx/functions/table.rst

+
+There are two conventions of passing arguments to a table function:
+
+- arguments passed by name::


Maybe we should start with uppercase .. and maybe bold it .. whatever you think looks good in the output

kasiafi · 2022-05-13T10:04:53Z

@mosabua I applied your comments as another fixup commit. Please take a look.

losipiuk

Lgtm

cla-bot bot added the cla-signed label May 11, 2022

kasiafi requested review from mosabua, martint and wendigo May 11, 2022 11:37

github-actions bot added the docs label May 11, 2022

findepi reviewed May 11, 2022

View reviewed changes

kasiafi force-pushed the 344PTFDocs branch from f702131 to fd396a9 Compare May 11, 2022 16:36

mosabua requested changes May 11, 2022

View reviewed changes

kasiafi mentioned this pull request May 12, 2022

Add builders for table function arguments #12350

Merged

mosabua reviewed May 13, 2022

View reviewed changes

mosabua requested changes May 13, 2022

View reviewed changes

mosabua mentioned this pull request May 13, 2022

Add Trino 381 release notes #12277

Merged

losipiuk approved these changes May 13, 2022

View reviewed changes

This was referenced May 15, 2022

Polymorphic table functions #1839

Open

Refactor table function analysis #12407

Merged

kasiafi force-pushed the 344PTFDocs branch from 64cdc5a to d21d99a Compare May 16, 2022 13:49

Document table function support

c6bca05

kasiafi force-pushed the 344PTFDocs branch from d21d99a to c6bca05 Compare May 16, 2022 17:31

kasiafi merged commit 27cb909 into trinodb:master May 16, 2022

github-actions bot added this to the 381 milestone May 16, 2022

kasiafi mentioned this pull request Jun 14, 2022

Support file based access control for table functions #12833

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document table function support #12338

Document table function support #12338

kasiafi commented May 11, 2022

kasiafi commented May 11, 2022

findepi May 11, 2022

kasiafi May 11, 2022

mosabua left a comment

kasiafi commented May 12, 2022

mosabua May 13, 2022

mosabua May 13, 2022

mosabua left a comment

mosabua May 13, 2022

mosabua May 13, 2022

kasiafi May 13, 2022

kasiafi May 13, 2022

mosabua May 16, 2022

kasiafi May 16, 2022

mosabua May 13, 2022

kasiafi May 13, 2022

mosabua May 16, 2022

kasiafi May 16, 2022

mosabua May 13, 2022

mosabua May 13, 2022

mosabua May 13, 2022

mosabua May 13, 2022

kasiafi May 13, 2022

mosabua May 13, 2022

kasiafi May 13, 2022

mosabua May 13, 2022

mosabua May 13, 2022

kasiafi commented May 13, 2022

losipiuk left a comment


		There are two conventions of passing arguments to a table function:

		- arguments passed by name::

Document table function support #12338

Document table function support #12338

Conversation

kasiafi commented May 11, 2022

kasiafi commented May 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mosabua left a comment

Choose a reason for hiding this comment

kasiafi commented May 12, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mosabua left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kasiafi commented May 13, 2022

losipiuk left a comment

Choose a reason for hiding this comment