Introduce SystemSchema tables (#5989) #6094
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -468,6 +468,11 @@ plan SQL queries. This metadata is cached on broker startup and also updated per | |
| [SegmentMetadata queries](segmentmetadataquery.html). Background metadata refreshing is triggered by | ||
| segments entering and exiting the cluster, and can also be throttled through configuration. | ||
|
|
||
| Druid exposes system information through special system tables. There are two such schemas available: Information Schema and System Schema | ||
|
There was a problem hiding this comment. Please use better grammar here: some punctuation is missing. There was a problem hiding this comment. added the missing period, let me know if you want to restructure it further. |
||
| Information schema provides details about table and column types. Sys schema provides information about Druid internals like segments/tasks/servers. | ||
|
|
||
|
|
||
| ## INFORMATION SCHEMA | ||
|
|
||
| You can access table and column metadata through JDBC using `connection.getMetaData()`, or through the | ||
| INFORMATION_SCHEMA tables described below. For example, to retrieve metadata for the Druid | ||
| datasource "foo", use the query: | ||
|
|
@@ -519,6 +524,89 @@ SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = 'druid' AND TABLE_ | |
| |COLLATION_NAME|| | ||
| |JDBC_TYPE|Type code from java.sql.Types (Druid extension)| | ||
|
|
||
| ## SYSTEM SCHEMA | ||
|
|
||
| The SYS schema provides visibility into Druid segments, servers and tasks. | ||
|
|
||
| For example to retrieve all segments for datasource "wikipedia", use the query: | ||
| ```sql | ||
| SELECT * FROM sys.segments WHERE datasource = 'wikipedia' | ||
| ``` | ||
|
|
||
| ### SEGMENTS table | ||
| Segments table provides details on all Druid segments, whether they are published yet or not. | ||
|
|
||
|
|
||
| |Column|Notes| | ||
| |------|-----| | ||
| |segment_id|Unique segment identifier| | ||
| |datasource|Name of datasource| | ||
| |start|Interval start time (in ISO 8601 format)| | ||
| |end|Interval end time (in ISO 8601 format)| | ||
| |size|Size of segment in bytes| | ||
| |version|Version number (generally an ISO8601 timestamp corresponding to when the segment set was first started)| | ||
|
There was a problem hiding this comment. The version format can be changed in the future. Probably better to tell There was a problem hiding this comment. I think i copied it from some other documentation for There was a problem hiding this comment. Got it. Let's do this later. |
||
| |partition_num|Partition number (an integer, unique within a datasource+interval+version; may not necessarily be contiguous)| | ||
| |num_replicas|Number replicas of this segment currently being served| | ||
|
|
||
| |num_rows|Number of rows in current segment, this value could be null if unkown to broker at query time| | ||
| |is_published|True if this segment has been published to the metadata store| | ||
| |is_available|True if this segment is currently being served by any server| | ||
|
There was a problem hiding this comment. I think There was a problem hiding this comment. This means |
||
| |is_realtime|True if this segment is being served on a realtime server| | ||
|
There was a problem hiding this comment.
There was a problem hiding this comment. I didn't understand your suggestion. Do you mean to provide some example of There was a problem hiding this comment. We have realtime node and realtime tasks (kafka index tasks, realtime index task). I believe you meant the all types of realtime tasks. So, |
||
| |payload|Jsonified datasegment payload| | ||
|
There was a problem hiding this comment. suggest "JSON-serialized DataSegment payload" instead |
||
|
|
||
| ### SERVERS table | ||
|
There was a problem hiding this comment. There should be a blurb here explaining what this table is all about. Currently, it's listing all data servers (anything that might host a segment) and that includes both historicals and ingestion tasks. |
||
| Servers table lists all data servers(any server that hosts a segment). It includes both historicals and peons. | ||
|
|
||
| |Column|Notes| | ||
| |------|-----| | ||
| |server|Server name in the form host:port| | ||
|
There was a problem hiding this comment. It'd be useful to have another field for just the host. Imagine doing stuff like There was a problem hiding this comment. sure, added another column for There was a problem hiding this comment. Does SystemSchema support concatenating strings? Then, There was a problem hiding this comment. It does support concatenating, but I think we should keep this here. It's basically the primary key of the server table, and is used for joins with the segment_servers table. The other fields (host, plaintext_port, etc) are provided too as conveniences. With system tables, since they're all generated dynamically anyway, it's ok to have some redundancy when it makes the user experience more convenient. There was a problem hiding this comment. Just tried concat, it does not support that call |
||
| |scheme|Server scheme http or https| | ||
| |server_type|Type of druid service for example historical, realtime, bridge, indexer_executor| | ||
|
There was a problem hiding this comment. Capitalize Druid, and this sentence needs a bit more punctuation. How about:
|
||
| |tier|Distribution tier see [druid.server.tier](#../configuration/index.html#Historical-General-Configuration)| | ||
| |current_size|Current size of segments in bytes on this server| | ||
| |max_size|Max size in bytes this server recommends to assign to segments see [druid.server.maxSize](#../configuration/index.html#Historical-General-Configuration)| | ||
|
|
||
| To retrieve information about all servers, use the query: | ||
| ```sql | ||
| SELECT * FROM sys.servers; | ||
| ``` | ||
|
|
||
| ### SEGMENT_SERVERS table | ||
|
|
||
| SEGMENT_SERVERS is used to join SEGMENTS with SERVERS table | ||
|
|
||
| |Column|Notes| | ||
| |------|-----| | ||
| |server|Server name in format host:port (Primary key of [servers table](#SERVERS-table))| | ||
| |segment_id|Segment identifier (Primary key of [segments table](#SEGMENTS-table))| | ||
|
|
||
| To retrieve information from segment_servers table, use the query: | ||
| ```sql | ||
| SELECT * FROM sys.segment_servers; | ||
|
There was a problem hiding this comment. Would be awesome to include an example here of a JOIN between "segments" and "servers". Maybe a query that shows the breakdown of number of segments for a specific datasource, by server. Something like grouping by server, filter by datasource, count segments. There was a problem hiding this comment. Added an example of query using JOIN |
||
| ``` | ||
|
|
||
| ### TASKS table | ||
|
|
||
| The tasks table provides information about active and recently-completed indexing tasks. For more information | ||
|
There was a problem hiding this comment. "check out" not "checkout out" |
||
| checkout out [ingestion tasks](#../ingestion/tasks.html) | ||
|
|
||
| |Column|Notes| | ||
| |------|-----| | ||
| |task_id|Unique task identifier| | ||
| |type|Task type, this should be "index" for indexing tasks| | ||
|
There was a problem hiding this comment.
|
||
| |datasource|Datasource name being indexed| | ||
| |created_time|Timestamp in ISO8601 format corresponding to when the ingestion task was created. Note that this value is populated for completed and waiting tasks. For running and pending tasks this value is set to DateTimes.EPOCH| | ||
|
There was a problem hiding this comment.
|
||
| |queue_insertion_time|Timestamp in ISO8601 format corresponding to when this task was added to the queue on the overlord| | ||
| |status|Status of a task can be RUNNING, FAILED, SUCCESS| | ||
| |runner_status|Runner status of a completed task would be NONE, for in-progress tasks this can be RUNNING, WAITING, PENDING| | ||
| |duration|Time it took to finish the task in milliseconds, this value is present only for completed tasks| | ||
| |location|Server name where this task is running in the format host:port, this information is present only for RUNNING tasks| | ||
| |error_msg|Detailed error message in case of FAILED tasks| | ||
|
|
||
| For example, to retrieve tasks information filtered by status, use the query | ||
| ```sql | ||
| SELECT * FROM sys.tasks where status='FAILED'; | ||
| ``` | ||
|
|
||
|
|
||
| ## Server configuration | ||
|
|
||
| The Druid SQL server is configured through the following properties on the broker. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I would suggest to add
NoopDruidLeaderClientrather than mocking withEasyMock. Even though this is not being used in this benchmark, I feel a bit weird because EasyMock is mostly for testing purpose not for benchmark.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
created anLeaderClientinterface and addedNoopDruidLeaderClientclassusing empty anonymous subclass instead