Skip to content

[Merged by Bors] - Implement TrinoCatalog according to ADR #263

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 29 commits into from
Closed

[Merged by Bors] - Implement TrinoCatalog according to ADR #263

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
d647156
Something working
sbernauer Aug 23, 2022
e3e62bf
Update tests. S3 verification is failing
sbernauer Aug 23, 2022
16feddf
Support S3 tls verification
sbernauer Aug 23, 2022
2f17082
Get tests green by fixing chmod
sbernauer Aug 25, 2022
a7c41a2
changelog
sbernauer Aug 25, 2022
fae8609
Increase test timeout
sbernauer Aug 25, 2022
b0ec2ab
changelog
sbernauer Aug 25, 2022
0dd9b8d
fix link syntax
sbernauer Aug 25, 2022
f603968
charts
sbernauer Aug 25, 2022
148ca93
Enable all tests again
sbernauer Aug 25, 2022
b8bebd9
Update example in docs
sbernauer Aug 25, 2022
8faf2b3
Update Terminology in docs
sbernauer Aug 25, 2022
38336a5
changelog
sbernauer Aug 25, 2022
30cef35
Minor things in tests
sbernauer Aug 25, 2022
88e10b3
Update rust/crd/src/catalog.rs
sbernauer Aug 25, 2022
49915e8
Update rust/crd/src/catalog.rs
sbernauer Aug 25, 2022
c215aac
Move error type up
sbernauer Aug 25, 2022
1657ed9
Fixed duplicate Impl
sbernauer Aug 25, 2022
ac53226
Add CatalogConfig::new
sbernauer Aug 25, 2022
8dc7297
Remove all metadata.name from all TestAsserts
sbernauer Aug 25, 2022
01aad32
Add permission to read s3connections and trinocatalogs
sbernauer Aug 25, 2022
4601154
Remove S3ConnectionDef from TrinoClusterSpec
sbernauer Aug 25, 2022
678a2d8
Refactor catalogs into submodule
sbernauer Aug 29, 2022
a9c3cc5
review
sbernauer Aug 29, 2022
e76050c
Extract constant
sbernauer Aug 29, 2022
2b660a2
Merge remote-tracking branch 'origin/main' into feature/trino-catalog-2
sbernauer Aug 29, 2022
cffc6a2
Remove pip install from tests
sbernauer Aug 29, 2022
5b36905
Dont let the cp's fail the tests
sbernauer Aug 29, 2022
47ce864
Increase timeouts
sbernauer Aug 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,31 @@ All notable changes to this project will be documented in this file.

## [Unreleased]

### Added

- Add support for connecting to HDFS ([#263]).
- Add support for Hive 3.1.3 ([#243]).

### Changed

- BREAKING: TrinoCatalogs now have their own CRD object and get referenced by the TrinoCluster according to [ADR19](https://docs.stackable.tech/home/contributor/adr/ADR019-trino_catalog_definitions.html) and [ADR20](https://docs.stackable.tech/home/contributor/adr/ADR020-trino_catalog_usage.html) ([#263]).
- Include chart name when installing with a custom release name ([#233], [#234]).
- `operator-rs` `0.21.1` -> `0.22.0` ([#235]).
- Add support for Hive 3.1.3 ([#243])
- Internal and client TLS now configurable instead of defaulting to "tls" secret class ([#244]).
- S3 TLS properly supported ([#244]).
- Introduced global `config` for `TLS` settings ([#244]).

### Fixed

- Add missing role to read S3Connection objects ([#263]).
- Disable Hive connector setting that disallow dropping tables. This check is now done by normal Trino authorization (e.g. OPA) ([#263]).

[#233]: https://github.com/stackabletech/trino-operator/pull/233
[#234]: https://github.com/stackabletech/trino-operator/pull/234
[#235]: https://github.com/stackabletech/trino-operator/pull/235
[#243]: https://github.com/stackabletech/trino-operator/pull/243
[#244]: https://github.com/stackabletech/trino-operator/pull/244
[#263]: https://github.com/stackabletech/trino-operator/pull/263

## [0.4.0] - 2022-06-30

Expand Down
1 change: 1 addition & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

167 changes: 167 additions & 0 deletions deploy/crd/trinocatalog.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: trinocatalogs.trino.stackable.tech
spec:
group: trino.stackable.tech
names:
categories: []
kind: TrinoCatalog
plural: trinocatalogs
shortNames: []
singular: trinocatalog
scope: Namespaced
versions:
- additionalPrinterColumns: []
name: v1alpha1
schema:
openAPIV3Schema:
description: "Auto-generated derived type for TrinoCatalogSpec via `CustomResource`"
properties:
spec:
properties:
configOverrides:
additionalProperties:
type: string
default: {}
type: object
connector:
oneOf:
- required:
- hive
properties:
hive:
properties:
hdfs:
description: Connection to an HDFS cluster
nullable: true
properties:
configMap:
description: Name of the discovery-configmap providing information about the HDFS cluster
type: string
required:
- configMap
type: object
metastore:
description: "Mandatory connection to a Hive Metastore, which will be used as a storage for metadata"
properties:
configMap:
description: Name of the discovery-configmap providing information about the Hive metastore
type: string
required:
- configMap
type: object
s3:
description: Connection to an S3 store
nullable: true
oneOf:
- required:
- inline
- required:
- reference
properties:
inline:
description: S3 connection definition as CRD.
properties:
accessStyle:
description: "Which access style to use. Defaults to virtual hosted-style as most of the data products out there. Have a look at the official documentation on <https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html>"
enum:
- Path
- VirtualHosted
nullable: true
type: string
credentials:
description: "If the S3 uses authentication you have to specify you S3 credentials. In the most cases a SecretClass providing `accessKey` and `secretKey` is sufficient."
nullable: true
properties:
scope:
description: "[Scope](https://docs.stackable.tech/secret-operator/scope.html) of the [SecretClass](https://docs.stackable.tech/secret-operator/secretclass.html)"
nullable: true
properties:
node:
default: false
type: boolean
pod:
default: false
type: boolean
services:
default: []
items:
type: string
type: array
type: object
secretClass:
description: "[SecretClass](https://docs.stackable.tech/secret-operator/secretclass.html) containing the LDAP bind credentials"
type: string
required:
- secretClass
type: object
host:
description: Hostname of the S3 server without any protocol or port
nullable: true
type: string
port:
description: Port the S3 server listens on. If not specified the products will determine the port to use.
format: uint16
minimum: 0.0
nullable: true
type: integer
tls:
description: If you want to use TLS when talking to S3 you can enable TLS encrypted communication with this setting.
nullable: true
properties:
verification:
description: The verification method used to verify the certificates of the server and/or the client
oneOf:
- required:
- none
- required:
- server
properties:
none:
description: "Use TLS but don't verify certificates"
type: object
server:
description: Use TLS and ca certificate to verify the server
properties:
caCert:
description: Ca cert to verify the server
oneOf:
- required:
- webPki
- required:
- secretClass
properties:
secretClass:
description: "Name of the SecretClass which will provide the ca cert. Note that a SecretClass does not need to have a key but can also work with just a ca cert. So if you got provided with a ca cert but don't have access to the key you can still use this method."
type: string
webPki:
description: Use TLS and the ca certificates trusted by the common web browsers to verify the server. This can be useful when you e.g. use public AWS S3 or other public available services.
type: object
type: object
required:
- caCert
type: object
type: object
required:
- verification
type: object
type: object
reference:
type: string
type: object
required:
- metastore
type: object
type: object
required:
- connector
type: object
required:
- spec
title: TrinoCatalog
type: object
served: true
storage: true
subresources: {}
134 changes: 31 additions & 103 deletions deploy/crd/trinocluster.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,37 @@ spec:
required:
- method
type: object
catalogLabelSelector:
description: "[LabelSelector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors) selecting the Catalogs to include in the Trino instance"
nullable: true
properties:
matchExpressions:
description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
items:
description: "A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values."
properties:
key:
description: key is the label key that the selector applies to.
type: string
operator:
description: "operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist."
type: string
values:
description: "values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch."
items:
type: string
type: array
required:
- key
- operator
type: object
type: array
matchLabels:
additionalProperties:
type: string
description: "matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is \"key\", the operator is \"In\", and the values array contains only \"value\". The requirements are ANDed."
type: object
type: object
config:
default:
tls:
Expand Down Expand Up @@ -188,10 +219,6 @@ spec:
required:
- roleGroups
type: object
hiveConfigMapName:
description: The discovery ConfigMap name of the Hive cluster (usually the same as the Hive cluster name).
nullable: true
type: string
opa:
description: The discovery ConfigMap name of the OPA cluster (usually the same as the OPA cluster name).
nullable: true
Expand All @@ -204,105 +231,6 @@ spec:
required:
- configMapName
type: object
s3:
description: A reference to a S3 bucket.
nullable: true
oneOf:
- required:
- inline
- required:
- reference
properties:
inline:
description: S3 connection definition as CRD.
properties:
accessStyle:
description: "Which access style to use. Defaults to virtual hosted-style as most of the data products out there. Have a look at the official documentation on <https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html>"
enum:
- Path
- VirtualHosted
nullable: true
type: string
credentials:
description: "If the S3 uses authentication you have to specify you S3 credentials. In the most cases a SecretClass providing `accessKey` and `secretKey` is sufficient."
nullable: true
properties:
scope:
description: "[Scope](https://docs.stackable.tech/secret-operator/scope.html) of the [SecretClass](https://docs.stackable.tech/secret-operator/secretclass.html)"
nullable: true
properties:
node:
default: false
type: boolean
pod:
default: false
type: boolean
services:
default: []
items:
type: string
type: array
type: object
secretClass:
description: "[SecretClass](https://docs.stackable.tech/secret-operator/secretclass.html) containing the LDAP bind credentials"
type: string
required:
- secretClass
type: object
host:
description: Hostname of the S3 server without any protocol or port
nullable: true
type: string
port:
description: Port the S3 server listens on. If not specified the products will determine the port to use.
format: uint16
minimum: 0.0
nullable: true
type: integer
tls:
description: If you want to use TLS when talking to S3 you can enable TLS encrypted communication with this setting.
nullable: true
properties:
verification:
description: The verification method used to verify the certificates of the server and/or the client
oneOf:
- required:
- none
- required:
- server
properties:
none:
description: "Use TLS but don't verify certificates"
type: object
server:
description: Use TLS and ca certificate to verify the server
properties:
caCert:
description: Ca cert to verify the server
oneOf:
- required:
- webPki
- required:
- secretClass
properties:
secretClass:
description: "Name of the SecretClass which will provide the ca cert. Note that a SecretClass does not need to have a key but can also work with just a ca cert. So if you got provided with a ca cert but don't have access to the key you can still use this method."
type: string
webPki:
description: Use TLS and the ca certificates trusted by the common web browsers to verify the server. This can be useful when you e.g. use public AWS S3 or other public available services.
type: object
type: object
required:
- caCert
type: object
type: object
required:
- verification
type: object
type: object
reference:
type: string
type: object
stopped:
description: "Emergency stop button, if `true` then all pods are stopped without affecting configuration (as setting `replicas` to `0` would)."
nullable: true
Expand Down
Loading