ClouderaPublicCloudCDFWorkshop
Cloudera Public Cloud CDF Workshop - AWS or Azure
NiFi - Kafka Data Hubs Tutorial
See
https://docs.cloudera.com/cdf-datahub/7.2.8/index.html
See also
https://github.com/tspannhw/ClouderaFlowManagementWorkshop
See also
https://github.com/cloudera-labs/edge2ai-workshop
See also
https://github.com/tspannhw/ClouderaStreamingAnalyticsSQLWorkshop
Workshop
General Cloud Activities
- Create Hbase table - See SSH
- Create Hive table - Apache Hue (From DWH)
- Create Kudu table - Apache Hue (From DWH)
- Create schema in Schema Registry(From Kafka DH)
NiFi Focused
- NiFi Ingest Kafka https://docs.cloudera.com/cdf-datahub/7.2.8/nifi-kafka-ingest/topics/cdf-datahub-fm-kafka-ingest-overview.html
- NiFi Ingest Hive https://docs.cloudera.com/cdf-datahub/7.2.8/nifi-hive-ingest/topics/cdf-datahub-nifi-hive-ingest.html
- NiFi Ingest HBase https://docs.cloudera.com/cdf-datahub/7.2.8/nifi-hbase-ingest/topics/cdf-datahub-nifi-hbase-ingest.html
- NiFi Ingest Kudu https://docs.cloudera.com/cdf-datahub/7.2.8/nifi-kudu-ingest/topics/cdf-datahub-nifi-kudu-ingest.html
- NiFi Ingest S3 https://docs.cloudera.com/cdf-datahub/7.2.8/nifi-aws-ingest/topics/cdf-datahub-fm-s3-ingest-overview.html
- NiFi Ingest ADLS https://docs.cloudera.com/cdf-datahub/7.2.8/nifi-azure-ingest/topics/cdf-datahub-fm-adls-ingest-overview.html
Kafka
- Kafka Latency & Monitoring https://github.com/asdaraujo/cdp-examples
Flink SQL
- https://docs.cloudera.com/csa/1.2.0/sql-client/topics/csa-sql-client-config.html
- https://github.com/cloudera/flink-tutorials/tree/master/flink-sql-tutorial
Initial Setup
HBase Connectivity
- Download Configuration Files
- SCP Configuration Files
- Use Configuration Files in NiFi /etc/hadoop/conf.cloudera.core_settings/core-site.xml,/tmp/hbase-site.xml
Kafka Connectivity
- Get brokers (multiple) from hardware tab of Kafka Data Hub, they use port 9093 and are SSL.
Hive Connectivity
- thrift://HIVEServerNameVeryLong.cloudera.site:9083
- Download Configuration Files
- SCP Configuration Files
- Use Configuration Files in NiFi /tmp/hive-site.xml,/tmp/core-site.xml,/tmp/hdfs-site.xml
Security Credentials Location
-
/var/lib/nifi/cm-auto-host_keystore.jks
-
/var/lib/nifi/cm-auto-in_cluster_truststore.jks
-
the keystore password can be found in this file, on a DH node: /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pw
-
the truststore password can be found by calling this API: https://[your_cm_host]/api/v40/certs/truststorePassword
Example SSH Connection
ssh -i MySSHPemIsAwesome.pem cloudbreak@3.82.183.16
Example NiFi CDP CDF Azure Application
https://github.com/tspannhw/cdp-datahub-azure-nifikafka