If you are not installing the operator using Helm then after installation the CRD for this operator must be created:
kubectl apply -f deploy/nificluster.yaml
To create a three-node Apache NiFi cluster with SingleUser authentication enabled apply the following to your Kubernetes cluster.
Please note that the version you need to specify is not only the version of NiFi which you want to roll out, but has to be amended with a Stackable version as shown. This Stackable version is the version of the underlying container image which is used to execute the processes. For a list of available versions please check our image registry. It should generally be safe to simply use the latest image version that is available.
The admin credentials that you can then log in with are: admin:supersecretpassword
If you do not provide the admin user credentials the operator can auto generate a random password for you, if you enable this functionality. You can retrieve this password by running the following command:
kubectl get secret nifi-admin-credentials-simple -o jsonpath="{.data.password}" | base64 --decode | cat - <(echo)
You may need to adjust this command if you change the configuration for adminCredentialsSecret in the example below.
apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
name: simple-nifi
spec:
version: 1.16.3-stackable0.1.0
zookeeperConfigMapName: simple-nifi-znode
config:
authentication:
method:
singleUser:
adminCredentialsSecret: nifi-admin-credentials-simple
sensitiveProperties:
keySecret: nifi-sensitive-property-key
nodes:
roleGroups:
default:
selector:
matchLabels:
kubernetes.io/os: linux
config:
log:
rootLogLevel: INFO
replicas: 3If you want to set a password for the initial admin user you can do this by applying the following object:
apiVersion: v1
kind: Secret
metadata:
name: nifi-admin-credentials-simple
stringData:
username: admin
password: supersecretpasswordYou can create the ZNode config map referenced in zookeeperConfigMapName via (assuming there is a ZooKeeper cluster called simple-zk:
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
name: simple-nifi-znode
spec:
clusterRef:
name: simple-zkEvery user has to authenticate themselves before using NiFI. There are multiple options to set up the authentication of users.
The default setting is to only provision a single user with administrative privileges. You need to specify the username and password of the user. Further users can not be added.
NiFI supports authentication of users against an LDAP server.
Have a look at the LDAP test and the general Stackable Authentication documentation on how to set it up.
In general, it requires you to specify a AuthenticationClass which is used to authenticate the users.
Here you have an example usage
---
apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
name: test-nifi
spec:
config:
authentication:
method:
authenticationClass: ldap-without-tls
---
apiVersion: authentication.stackable.tech/v1alpha1
kind: AuthenticationClass
metadata:
name: ldap-without-tls
spec:
provider:
ldap:
hostname: openldap.default.svc.cluster.local
searchBase: ou=users,dc=example,dc=org
bindCredentials:
secretClass: nifi-with-ldap-bind
port: 1389
---
apiVersion: secrets.stackable.tech/v1alpha1
kind: SecretClass
metadata:
name: nifi-with-ldap-bind
spec:
backend:
k8sSearch:
searchNamespace:
pod: {}
---
apiVersion: v1
kind: Secret
metadata:
name: nifi-with-ldap-bind
labels:
secrets.stackable.tech/class: nifi-with-ldap-bind
stringData:
user: cn=integrationtest,ou=users,dc=example,dc=org
password: integrationtestNiFi supports multiple authorization methods documented here. The available authorization methods depend on the chosen authentication method.
Authorization is not fully implemented by the stackable nifi-operator. Because of this the operator currently infers the authorization method to use from the configured authentication method.
For this authentication method the Single user authorization will be used. Basically the single user is allowed to do everything.
In the future retrieving groups from LDAP and providing policies via OPA is planned.
Currently we will use the org.apache.nifi.authorization.FileUserGroupProvider and org.apache.nifi.authorization.FileAccessPolicyProvider.
The user specified in the LDAP bind will be the admin user which can log into the NiFi cluster and create and modify other users.
He is also able to create and modify groups and polices in the Webinterface.
The changes made will be written to local files which are not persistent.
The managed NiFi cluster is automatically configured to export Prometheus metrics. See home:operators:monitoring.adoc for more details.
The cluster definition also supports overriding configuration properties and environment variables, either per role or per role group, where the more specific override (role group) has precedence over the less specific one (role).
|
Important
|
Do not override port numbers. This will lead to cluster malfunction. |
Apache NiFi runtime configuration is stored in the files bootstrap.conf and nifi.properties. The configOverrides block allows you to add custom parameters to these files. A Full list of the available configuration options can be found in the official Apache NiFi documentation at https://nifi.apache.org/docs/nifi-docs/components/nifi-docs/html/administration-guide.html#system_properties.
Overrides consist of a key, which must match the property name in the configuration file and a value. This allows arbitrary configuration properties to be added to Spark. In the example above a property aws.region is being explicitly set to 'eu-west-1', overriding the default value.
|
Warning
|
Please be aware that by overriding config settings in this section you have a very high risk of breaking things, because the product does not behave the way the Stackable Operator for Apache NiFi expects it to behave anymore. |
The following snippet shows an example of how this would look in the NifiCluster definition:
config:
sensitivePropertyKeySecret: nifi-sensitive-property-
configOverrides:
nifi.properties:
nifi.config.property: eu-west-1
bootstrap.conf:
boostrap.config.property: some_string
replicas: 2Environment variables can be (over)written by adding the envOverrides property.
For example per role group:
nodes:
roleGroups:
default:
config: {}
replicas: 1
envOverrides:
MY_ENV_VAR: "MY_VALUE"or per role:
nodes:
envOverrides:
MY_ENV_VAR: "MY_VALUE"
roleGroups:
default:
config: {}
replicas: 1By default, a Nifi cluster will create five different persistent volume claims for flow files, provenance, database, content and state folders. These PVCs will request 2Gi. It is recommended that you configure these volume requests according to your needs.
Storage requests can be configured at role or group level, for one or more of the persistent volumes as follows:
nodes:
roleGroups:
default:
config:
resources:
storage:
flowfile_repo:
capacity: 12Gi
provenance_repo:
capacity: 12Gi
database_repo:
capacity: 12Gi
content_repo:
capacity: 12Gi
state_repo:
capacity: 12GiIn the above example, all nodes in the default group will request 12Gi of storage the various folders.
You can request a certain amount of memory for each individual role group as shown below:
nodes:
roleGroups:
default:
config:
resources:
memory:
limit: '2Gi'In this example, each node container in the "default" group will have a maximum of 2Gi of memory.
Setting this property will automatically also set the maximum Java heap size for the corresponding process to 80% of the available memory. Be aware that if the memory constraint is too low, the cluster might fail to start. If pods terminate with an 'OOMKilled' status and the cluster doesn’t start, try increasing the memory limit.
For more details regarding Kubernetes memory requests and limits see: Assign Memory Resources to Containers and Pods.
Similarly to memory resources, you can also configure CPU limits, as shown below:
nodes:
roleGroups:
default:
config:
resources:
cpu:
max: '500m'
min: '250m'For more details regarding Kubernetes CPU limits see: Assign CPU Resources to Containers and Pods.