Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-20420][SQL] Add events to the external catalog #17710

Closed
wants to merge 1 commit into from

Conversation

Projects
None yet
3 participants
@hvanhovell
Copy link
Contributor

commented Apr 20, 2017

What changes were proposed in this pull request?

It is often useful to be able to track changes to the ExternalCatalog. This PR makes the ExternalCatalog emit events when a catalog object is changed. Events are fired before and after the change.

The following events are fired per object:

  • Database
    • CreateDatabasePreEvent: event fired before the database is created.
    • CreateDatabaseEvent: event fired after the database has been created.
    • DropDatabasePreEvent: event fired before the database is dropped.
    • DropDatabaseEvent: event fired after the database has been dropped.
  • Table
    • CreateTablePreEvent: event fired before the table is created.
    • CreateTableEvent: event fired after the table has been created.
    • RenameTablePreEvent: event fired before the table is renamed.
    • RenameTableEvent: event fired after the table has been renamed.
    • DropTablePreEvent: event fired before the table is dropped.
    • DropTableEvent: event fired after the table has been dropped.
  • Function
    • CreateFunctionPreEvent: event fired before the function is created.
    • CreateFunctionEvent: event fired after the function has been created.
    • RenameFunctionPreEvent: event fired before the function is renamed.
    • RenameFunctionEvent: event fired after the function has been renamed.
    • DropFunctionPreEvent: event fired before the function is dropped.
    • DropFunctionPreEvent: event fired after the function has been dropped.

The current events currently only contain the names of the object modified. We add more events, and more details at a later point.

A user can monitor changes to the external catalog by adding a listener to the Spark listener bus checking for ExternalCatalogEvents using the SparkListener.onOtherEvent hook. A more direct approach is add listener directly to the ExternalCatalog.

How was this patch tested?

Added the ExternalCatalogEventSuite.

@SparkQA

This comment has been minimized.

Copy link

commented Apr 21, 2017

Test build #76007 has finished for PR 17710 at commit 8086e1a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • trait ExternalCatalogEventListener
  • trait DatabaseEvent extends ExternalCatalogEvent
  • case class CreateDatabasePreEvent(database: String) extends DatabaseEvent
  • case class CreateDatabaseEvent(database: String) extends DatabaseEvent
  • case class DropDatabasePreEvent(database: String) extends DatabaseEvent
  • case class DropDatabaseEvent(database: String) extends DatabaseEvent
  • trait TableEvent extends DatabaseEvent
  • case class CreateTablePreEvent(database: String, name: String) extends TableEvent
  • case class CreateTableEvent(database: String, name: String) extends TableEvent
  • case class DropTablePreEvent(database: String, name: String) extends TableEvent
  • case class DropTableEvent(database: String, name: String) extends TableEvent
  • case class RenameTablePreEvent(
  • case class RenameTableEvent(
  • trait FunctionEvent extends DatabaseEvent
  • case class CreateFunctionPreEvent(database: String, name: String) extends FunctionEvent
  • case class CreateFunctionEvent(database: String, name: String) extends FunctionEvent
  • case class DropFunctionPreEvent(database: String, name: String) extends FunctionEvent
  • case class DropFunctionEvent(database: String, name: String) extends FunctionEvent
  • case class RenameFunctionPreEvent(
  • case class RenameFunctionEvent(
@rxin

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2017

Merging in master/branch-2.2.

@asfgit asfgit closed this in e2b3d23 Apr 21, 2017

asfgit pushed a commit that referenced this pull request Apr 21, 2017

[SPARK-20420][SQL] Add events to the external catalog
## What changes were proposed in this pull request?
It is often useful to be able to track changes to the `ExternalCatalog`. This PR makes the `ExternalCatalog` emit events when a catalog object is changed. Events are fired before and after the change.

The following events are fired per object:

- Database
  - CreateDatabasePreEvent: event fired before the database is created.
  - CreateDatabaseEvent: event fired after the database has been created.
  - DropDatabasePreEvent: event fired before the database is dropped.
  - DropDatabaseEvent: event fired after the database has been dropped.
- Table
  - CreateTablePreEvent: event fired before the table is created.
  - CreateTableEvent: event fired after the table has been created.
  - RenameTablePreEvent: event fired before the table is renamed.
  - RenameTableEvent: event fired after the table has been renamed.
  - DropTablePreEvent: event fired before the table is dropped.
  - DropTableEvent: event fired after the table has been dropped.
- Function
  - CreateFunctionPreEvent: event fired before the function is created.
  - CreateFunctionEvent: event fired after the function has been created.
  - RenameFunctionPreEvent: event fired before the function is renamed.
  - RenameFunctionEvent: event fired after the function has been renamed.
  - DropFunctionPreEvent: event fired before the function is dropped.
  - DropFunctionPreEvent: event fired after the function has been dropped.

The current events currently only contain the names of the object modified. We add more events, and more details at a later point.

A user can monitor changes to the external catalog by adding a listener to the Spark listener bus checking for `ExternalCatalogEvent`s using the `SparkListener.onOtherEvent` hook. A more direct approach is add listener directly to the `ExternalCatalog`.

## How was this patch tested?
Added the `ExternalCatalogEventSuite`.

Author: Herman van Hovell <hvanhovell@databricks.com>

Closes #17710 from hvanhovell/SPARK-20420.

(cherry picked from commit e2b3d23)
Signed-off-by: Reynold Xin <rxin@databricks.com>

cmonkey added a commit to cmonkey/spark that referenced this pull request Apr 28, 2017

[SPARK-20420][SQL] Add events to the external catalog
## What changes were proposed in this pull request?
It is often useful to be able to track changes to the `ExternalCatalog`. This PR makes the `ExternalCatalog` emit events when a catalog object is changed. Events are fired before and after the change.

The following events are fired per object:

- Database
  - CreateDatabasePreEvent: event fired before the database is created.
  - CreateDatabaseEvent: event fired after the database has been created.
  - DropDatabasePreEvent: event fired before the database is dropped.
  - DropDatabaseEvent: event fired after the database has been dropped.
- Table
  - CreateTablePreEvent: event fired before the table is created.
  - CreateTableEvent: event fired after the table has been created.
  - RenameTablePreEvent: event fired before the table is renamed.
  - RenameTableEvent: event fired after the table has been renamed.
  - DropTablePreEvent: event fired before the table is dropped.
  - DropTableEvent: event fired after the table has been dropped.
- Function
  - CreateFunctionPreEvent: event fired before the function is created.
  - CreateFunctionEvent: event fired after the function has been created.
  - RenameFunctionPreEvent: event fired before the function is renamed.
  - RenameFunctionEvent: event fired after the function has been renamed.
  - DropFunctionPreEvent: event fired before the function is dropped.
  - DropFunctionPreEvent: event fired after the function has been dropped.

The current events currently only contain the names of the object modified. We add more events, and more details at a later point.

A user can monitor changes to the external catalog by adding a listener to the Spark listener bus checking for `ExternalCatalogEvent`s using the `SparkListener.onOtherEvent` hook. A more direct approach is add listener directly to the `ExternalCatalog`.

## How was this patch tested?
Added the `ExternalCatalogEventSuite`.

Author: Herman van Hovell <hvanhovell@databricks.com>

Closes apache#17710 from hvanhovell/SPARK-20420.

peter-toth pushed a commit to peter-toth/spark that referenced this pull request Oct 6, 2018

[SPARK-20420][SQL] Add events to the external catalog
## What changes were proposed in this pull request?
It is often useful to be able to track changes to the `ExternalCatalog`. This PR makes the `ExternalCatalog` emit events when a catalog object is changed. Events are fired before and after the change.

The following events are fired per object:

- Database
  - CreateDatabasePreEvent: event fired before the database is created.
  - CreateDatabaseEvent: event fired after the database has been created.
  - DropDatabasePreEvent: event fired before the database is dropped.
  - DropDatabaseEvent: event fired after the database has been dropped.
- Table
  - CreateTablePreEvent: event fired before the table is created.
  - CreateTableEvent: event fired after the table has been created.
  - RenameTablePreEvent: event fired before the table is renamed.
  - RenameTableEvent: event fired after the table has been renamed.
  - DropTablePreEvent: event fired before the table is dropped.
  - DropTableEvent: event fired after the table has been dropped.
- Function
  - CreateFunctionPreEvent: event fired before the function is created.
  - CreateFunctionEvent: event fired after the function has been created.
  - RenameFunctionPreEvent: event fired before the function is renamed.
  - RenameFunctionEvent: event fired after the function has been renamed.
  - DropFunctionPreEvent: event fired before the function is dropped.
  - DropFunctionPreEvent: event fired after the function has been dropped.

The current events currently only contain the names of the object modified. We add more events, and more details at a later point.

A user can monitor changes to the external catalog by adding a listener to the Spark listener bus checking for `ExternalCatalogEvent`s using the `SparkListener.onOtherEvent` hook. A more direct approach is add listener directly to the `ExternalCatalog`.

## How was this patch tested?
Added the `ExternalCatalogEventSuite`.

Author: Herman van Hovell <hvanhovell@databricks.com>

Closes apache#17710 from hvanhovell/SPARK-20420.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.