API: Add SupportsReferencedBy interface#15895
API: Add SupportsReferencedBy interface#15895singhpk234 wants to merge 3 commits intoapache:mainfrom
Conversation
sfc-gh-prsingh
force-pushed
the
feature/supports-referenced-by-interface
branch
from
e597a1f to
b6d8da2
Compare
| import org.apache.iceberg.view.View; | ||
|
|
||
| /** Catalog methods for loading tables and views with a referenced-by view chain. */ | ||
| public interface SupportsReferencedBy { |
There was a problem hiding this comment.
For the universal load endpoints for relational objects, I guess we will also need to support referenced-by.
https://github.com/apache/iceberg/pull/15831/changes#diff-91e43b034980f7feb9d0d959ef4101f26254677c8ba49ad93e1edd31ecd77614
There was a problem hiding this comment.
referenced-by just have view identifier :), so table identifier is suff, when we define something for view specific then we would be good !
There was a problem hiding this comment.
We have been using TableIdentifier for views probably because we don't want to introduce a ViewIdentifier exactly same as TableIdentifier.
I would hope that we can align on one generic CatalogObjectIdentifier for all new REST spec and Java implementations.
https://docs.google.com/document/d/1NTQhgNbP2dkIMuXUMA5JdwliVQKCp1TU_ux5J_AaPiw/edit?tab=t.0#heading=h.lrmay9r6i8ai
We probably don't need a specific view identifier in the REST spec.
My initial comment is if we need to support referenced-by for the relations endpoint.
There was a problem hiding this comment.
Continuning some of the discussion on the other thread, who is this interface for? It looks like it would only be used for non-REST catalog implementations (which wouldn't use the SessionCatalog/ViewSessionCatalog)
There was a problem hiding this comment.
My initial comment is if we need to support referenced-by for the relations endpoint.
TableIdentifier is the universal identifier for views today — happy to adopt a generic type when it lands and yes i agree we would require to send the reference-by to the relations endpoint as well :)
who is this interface for
Its instanceof discovery pattern used by SparkCatalog as the underlying icebergCatalog that sparkCatalog holds would be implementing it like RESTCatalog [1], i plan to make RESTCatalog implement SupportsReferencedBy interface [2]
Please let me know wdyt
please ref E2E impl i have in mind : #13979
| * @return instance of {@link Table} implementation referred by {@code ident} | ||
| * @throws NoSuchTableException if the table does not exist | ||
| */ | ||
| default Table loadTable( |
There was a problem hiding this comment.
what's the reason for adding the default here? the SupportsReferencedBy decorator interface is not extended by this interface. can we just add the proper class that implements this new interface?
Maybe only RESTSessionCatalog should implement the SupportsReferencedBy decorator interface.
There was a problem hiding this comment.
we have SessionContext for this api, it can't implement SupportsReferencedBy which doesn't have sessionContext
There was a problem hiding this comment.
Why not default to calling the parent version without "referencedBy"? I guess I have similar thoughts as Steven here.
There was a problem hiding this comment.
Also noted by the agents, you commented that this would delegate to old implementations for backwards compatibility :)
There was a problem hiding this comment.
I had that earlier version : e597a1f#diff-93c6a4d20e7c7d3a8e9b90cac139da020d6a75138c2a5718941a4c4e7b9d8217R198-R210
forgot to update the description :(, wow agents caught that, thats pretty cool !
Then i went backwards as in this scenario if we let referencedBy skip then its felt a bit incorrect as i am swallowing the intent of the api, throwing unsupported exception would better prompt the user for this is what i think, please let me know you thoughts considering above ?
|
I think we need to override the delegates here as well. Once |
| import org.apache.iceberg.Table; | ||
| import org.apache.iceberg.view.View; | ||
|
|
||
| /** Catalog methods for loading tables and views with a referenced-by view chain. */ |
There was a problem hiding this comment.
This is a description of the contents of the interface, and not a description of what the interface is for
| * | ||
| * @param identifier a table identifier | ||
| * @param referencedBy ordered list of view identifiers from outermost to innermost | ||
| * @return an iceberg table |
| * | ||
| * @param identifier a view identifier | ||
| * @param referencedBy ordered list of view identifiers from outermost to innermost | ||
| * @return an iceberg view |
| * @throws NoSuchTableException if the table does not exist | ||
| */ | ||
| default Table loadTable( | ||
| SessionContext context, TableIdentifier ident, List<TableIdentifier> referencedBy) { |
There was a problem hiding this comment.
Can referencedBy be null?
There was a problem hiding this comment.
it can be null, in that case it would be better for the caller to just call the loadTable(context, ident) ? i can mark this explicitly as @Nullable, wdyt ?
precisley, i was thinking of taking that as part of core changes, here is the complete e2e flow that i had in mind : #13979 please let know if you prefer the delegates too in this pr, happy to put that in the current change. |
- Update class Javadoc to describe purpose (authorization, credential-scoping, auditing) rather than just listing methods - Fix iceberg -> Iceberg capitalization in @return tags
4 participants
About the change
SupportsReferencedBy - a new opt-in catalog capability interface with loadTable and loadView methods
that accept an ordered list of referencing view identifiers
Default overloads on SessionCatalog.loadTable and ViewSessionCatalog.loadView that accept the
referenced-by view chain, delegating to the existing methods for backward compatibility
When a table or view is loaded as part of resolving a view, the chain of referencing views can be passed
to the catalog. This enables catalog servers to make authorization, credential scoping, and auditing
decisions based on the full view reference chain.
This is the first in a series of PRs to implement the referenced-by feature end-to-end, broken out from
#13979:
Related