Skip to content

Latest commit

 

History

History
148 lines (86 loc) · 5.72 KB

configuration_private.md

File metadata and controls

148 lines (86 loc) · 5.72 KB
Raw

Environment Variables

Before running the application, you need to set the following environment variables:

Cloudera mode

export CLOUDERA_MODE=private

Livy Server URL Configuration

export LIVY_URL=<Your LIVY_URL>

The LIVY_URL parameter must be set to point to the endpoint where the Livy server is accessible. This URL varies based on your runtime environment and Livy server setup. Both HTTP and HTTPS protocols can be used, depending on the security configuration of your server. Ensure to replace <Your LIVY_URL> with the actual hostname or IP address of your Livy server.

Configuration examples:

  • for HTTP: http://yourLivyUrl:8998
  • for HTTPS: https://yourLivyUrl:8998

Note: The port number 8998 is default for Livy servers but may need to be adjusted based on your server's specific configuration.

Kerberos configuration

  1. Kerberos authentication flag

  • To enable Kerberos authentication, you should set the USE_KERBEROS_AUTH to true.

  • If you do not wish to use Kerberos authentication, set this flag to any other value.

    export USE_KERBEROS_AUTH=<true|anyValue>

  1. Login Context Name: The LOGIN_CONTEXT variable should be set to the name of your login context, as defined in your JAAS configuration file.

    export LOGIN_CONTEXT=<Your login context name>

Example JAAS Configuration for Kerberos Authentication

A Kerberos JAAS (Java Authentication and Authorization Service) configuration file is used to define the authentication and authorization modules for Java applications. It specifies how these applications should authenticate using Kerberos and other details necessary for secure communication. The structure of this file is critical for setting up Kerberos authentication properly.

Here is a basic example of what entries in a Kerberos JAAS configuration file might look like:

LoginContext {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="/path/to/keytab/file.keytab"
    storeKey=true
    useTicketCache=false
    principal="principalName@REALM.COM";
};

HDFS

  1. Name Node Variables: The HDFS cluster utilizes primary and secondary name nodes to ensure redundancy and failover capabilities. Assign the connection details for these name nodes in your environment with the export commands:

    export HDFS_NAMENODE0=<Your first HDFS name node address>
export HDFS_NAMENODE1=<Your second HDFS name node address>

    Replace <Your first HDFS name node address> and <Your second HDFS name node address> with the actual addresses of your primary and secondary name nodes respectively.

    Example Configuration:

    • Primary name node: HDFS_NAMENODE0=cluster0-vm0.cluster0.example.com
    • Secondary name node: HDFS_NAMENODE1=cluster0-vm1.cluster0.example.com

  2. WebHDFS Port Configuration

    export WEBHDFS_PORT=<Your WebHDFS port number>

    Replace <Your WebHDFS port number> with the designated port number for WebHDFS in your configuration to allow HTTP access to HDFS data.

  3. WebHDFS Protocol Specification: To define whether WebHDFS communicates over HTTP or HTTPS, set the WEBHDFS_PROTOCOL environment variable accordingly:

    export WEBHDFS_PROTOCOL=<http or https>

    The value of WEBHDFS_PROTOCOL should be either http for standard HTTP communication or https for secure HTTP communication. This setting ensures that your HDFS interactions meet the security requirements of your environment.

Quartz scheduler configuration file

export SCHEDULER_PROP=<Path to your quartz.properties file>

Replace <Path to your quartz.properties file> with the actual path to your configuration file.

Configuration File Example and Documentation

Database Tables Setup

  • Quartz requires specific database tables for its operation. The scripts to create these tables are available here. Ensure that these tables are created in your database to facilitate the Quartz job scheduling processes.

JVM properties

Before running the application, you need to set the following properties:

1. Kerberos Configuration File Path

-Djava.security.krb5.conf=<Path to your krb5.conf file>

This property specifies the path to the krb5.conf file, which contains configuration settings for the Kerberos protocol.

2. JAAS Configuration File Path

-Djava.security.auth.login.config=<Path to your kerberos.jaas.conf file>

This property indicates the path to the kerberos.jaas.conf file, which contains login information for Java Authentication and Authorization Service (JAAS) used for Kerberos authentication. This file defines configurations for Kerberos authentication, including service principals and their respective keys.

3. Subject Creds Only

-Djavax.security.auth.useSubjectCredsOnly=true

This property sets the flag to indicate that only subject credentials should be used for authentication. In Kerberos authentication contexts, this ensures that credentials acquired during the authentication process are appropriately utilized and not overridden or ignored by other credential sources.