Skip to content

Commit

Permalink
NIFI-7668 Implemented support for additional AEAD property encryption…
Browse files Browse the repository at this point in the history
… methods

- Added support for PBKDF2 and Scrypt property encryption methods in addition to Argon2
- Refactored StringEncryptor class to PropertyEncryptor interface with implementations
- Added PasswordBasedCipherPropertyEncryptor and KeyedCipherPropertyEncryptor
- Replaced direct instantiation of encryptor with PropertyEncryptorFactory
- Refactored applicable unit tests to use mocked PropertyEncryptor

NIFI-7668 Consolidated similar methods to CipherPropertyEncryptor

NIFI-7668 Updated AbstractTimeBasedSchedulingAgent with PropertyEncryptor

NIFI-7668 Added support for bcrypt secure hashing algorithm

NIFI-7668 Updated comments to clarify implementation of bcrypt key derivation

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4809.
  • Loading branch information
exceptionfactory authored and thenatog committed Feb 25, 2021
1 parent 99fe548 commit 5608f43
Show file tree
Hide file tree
Showing 68 changed files with 1,570 additions and 1,917 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -96,9 +96,8 @@ public static boolean isEmpty(String src) {
*
* @param arrays the component byte[] in order
* @return a concatenated byte[]
* @throws IOException this should never be thrown
*/
public static byte[] concatByteArrays(byte[]... arrays) throws IOException {
public static byte[] concatByteArrays(byte[]... arrays) {
int totalByteLength = 0;
for (byte[] bytes : arrays) {
totalByteLength += bytes.length;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,7 @@

import at.favre.lib.crypto.bcrypt.Radix64Encoder;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.security.NoSuchProviderException;
import java.security.SecureRandom;
import java.util.Arrays;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import javax.crypto.Cipher;
Expand Down Expand Up @@ -172,7 +168,7 @@ protected Cipher getInitializedCipher(EncryptionMethod encryptionMethod, String
}

try {
SecretKey tempKey = deriveKey(password, keyLength, algorithm, provider, rawSalt, workFactor, useLegacyKeyDerivation);
SecretKey tempKey = deriveKey(password, keyLength, algorithm, rawSalt, workFactor, useLegacyKeyDerivation);
KeyedCipherProvider keyedCipherProvider = new AESKeyedCipherProvider();
return keyedCipherProvider.getCipher(encryptionMethod, tempKey, iv, encryptMode);
} catch (IllegalArgumentException e) {
Expand All @@ -187,29 +183,13 @@ protected Cipher getInitializedCipher(EncryptionMethod encryptionMethod, String
}
}

private SecretKey deriveKey(String password, int keyLength, String algorithm, String provider, byte[] rawSalt,
int workFactor, boolean useLegacyKeyDerivation) throws NoSuchAlgorithmException, NoSuchProviderException {
/* The SHA-512 hash is required in order to derive a key longer than 184 bits (the resulting size of the Bcrypt hash) and ensuring the avalanche effect causes higher key entropy (if all
derived keys follow a consistent pattern, it weakens the strength of the encryption) */
MessageDigest digest = MessageDigest.getInstance("SHA-512", provider);
BcryptSecureHasher bcryptSecureHasher = new BcryptSecureHasher(workFactor);
byte[] fullHashOutputBytes = bcryptSecureHasher.hashRaw(password.getBytes(StandardCharsets.UTF_8), rawSalt);
byte[] derivedKeyBytes;
private SecretKey deriveKey(String password, int keyLength, String algorithm, byte[] rawSalt,
int workFactor, boolean useLegacyKeyDerivation) {

// Depending on the legacy key derivation process indicator, run the digest over the full or partial hash output
if (useLegacyKeyDerivation) {
// The "legacy" process included the algorithm, work factor, and salt in the digest input
derivedKeyBytes = digest.digest(fullHashOutputBytes);
logger.warn("Using legacy key derivation process for backward compatibility (digest run on full {} bytes of Bcrypt hash output)", fullHashOutputBytes.length);
} else {
// Only digest "hash" (last 31 bytes) not the algorithm, version, work factor, and salt (first 29 bytes)
final int HASH_OUTPUT_START = 29;
byte[] hashBytes = Arrays.copyOfRange(fullHashOutputBytes, HASH_OUTPUT_START, fullHashOutputBytes.length);
derivedKeyBytes = digest.digest(hashBytes);
}

derivedKeyBytes = Arrays.copyOf(derivedKeyBytes, keyLength / 8);
return new SecretKeySpec(derivedKeyBytes, algorithm);
final int derivedKeyLength = keyLength / 8;
final SecureHasher secureHasher = new KeyDerivationBcryptSecureHasher(derivedKeyLength, workFactor, useLegacyKeyDerivation);
byte[] derivedKey = secureHasher.hashRaw(password.getBytes(StandardCharsets.UTF_8), rawSalt);
return new SecretKeySpec(derivedKey, algorithm);
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,7 @@
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.charset.StandardCharsets;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.security.NoSuchProviderException;
import java.security.spec.InvalidKeySpecException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Arrays;
Expand All @@ -38,11 +34,6 @@
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import javax.crypto.Cipher;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.SecretKey;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.PBEParameterSpec;
import org.apache.commons.codec.binary.Base64;
import org.apache.commons.lang3.StringUtils;
import org.apache.nifi.processor.exception.ProcessException;
Expand Down Expand Up @@ -297,8 +288,7 @@ public static byte[] readBytesFromInputStream(InputStream in, String label, int
byte[] stoppedBy = StreamUtils.copyExclusive(in, bytesOut, limit + delimiter.length, delimiter);

if (stoppedBy != null) {
byte[] bytes = bytesOut.toByteArray();
return bytes;
return bytesOut.toByteArray();
}

// If no delimiter was found, reset the cursor
Expand Down Expand Up @@ -344,65 +334,10 @@ public static boolean isUnlimitedStrengthCryptoSupported() {
}
}

public static boolean isPBECipher(String algorithm) {
EncryptionMethod em = EncryptionMethod.forAlgorithm(algorithm);
return em != null && em.isPBECipher();
}

public static boolean isKeyedCipher(String algorithm) {
EncryptionMethod em = EncryptionMethod.forAlgorithm(algorithm);
return em != null && em.isKeyedCipher();
}

/**
* Initializes a {@link Cipher} object with the given PBE parameters.
*
* @param algorithm the algorithm
* @param provider the JCA provider
* @param password the password
* @param salt the salt
* @param iterationCount the KDF iteration count
* @param encryptMode true to encrypt; false to decrypt
* @return the initialized Cipher
* @throws IllegalArgumentException if any parameter is invalid
*/
public static Cipher initPBECipher(String algorithm, String provider, String password, byte[] salt, int iterationCount, boolean encryptMode) throws IllegalArgumentException {
try {
// Initialize secret key from password
final PBEKeySpec pbeKeySpec = new PBEKeySpec(password.toCharArray());
final SecretKeyFactory factory = SecretKeyFactory.getInstance(algorithm, provider);
SecretKey tempKey = factory.generateSecret(pbeKeySpec);

final PBEParameterSpec parameterSpec = new PBEParameterSpec(salt, iterationCount);
Cipher cipher = Cipher.getInstance(algorithm, provider);
cipher.init(encryptMode ? Cipher.ENCRYPT_MODE : Cipher.DECRYPT_MODE, tempKey, parameterSpec);
return cipher;
} catch (NoSuchAlgorithmException | NoSuchProviderException | InvalidKeySpecException | NoSuchPaddingException | InvalidKeyException | InvalidAlgorithmParameterException e) {
throw new IllegalArgumentException("One or more parameters to initialize the PBE cipher were invalid", e);
}
}

/**
* Returns the KDF iteration count for various PBE algorithms. These values were determined empirically from configured/chosen legacy values from the earlier version of the project.
* Code demonstrating this is available at {@link StringEncryptorTest#testPBEncryptionShouldBeExternallyConsistent}.
*
* @param algorithm the {@link EncryptionMethod#algorithm}
* @return the iteration count. Default is 0.
*/
public static int getIterationCountForAlgorithm(String algorithm) {
int iterationCount = 0;
// DES/RC*/SHA-1/-256 algorithms use custom iteration counts
if (algorithm.matches("DES|RC|SHAA|SHA256")) {
iterationCount = 1000;
}
return iterationCount;
}

/**
* Returns the salt length for various PBE algorithms. These values were determined empirically from configured/chosen legacy values from the earlier version of the project.
* Code demonstrating this is available at {@link StringEncryptorTest#testPBEncryptionShouldBeExternallyConsistent}.
*
* @param algorithm the {@link EncryptionMethod#algorithm}
* @param algorithm the {@link EncryptionMethod#getAlgorithm()}
* @return the salt length in bytes. Default is 16.
*/
public static int getSaltLengthForAlgorithm(String algorithm) {
Expand Down Expand Up @@ -431,7 +366,7 @@ public static String getLoggableRepresentationOfSensitiveValue(String sensitiveP
// There is little initialization cost, so it doesn't make sense to cache this as a field
SecureHasher secureHasher = new Argon2SecureHasher();

// TODO: Extend {@link StringEncryptor} with secure hashing capability and inject?
// TODO: Extend with secure hashing capability and inject?
return getLoggableRepresentationOfSensitiveValue(sensitivePropertyValue, secureHasher);
}

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.security.util.crypto;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.Arrays;

/**
* Extension of Bcrypt Secure Hasher used for Key Derivation support. Allows specifying a Derived Key Length in bytes.
*/
public class KeyDerivationBcryptSecureHasher extends BcryptSecureHasher {
private static final Logger LOGGER = LoggerFactory.getLogger(KeyDerivationBcryptSecureHasher.class);

private static final String DIGEST_ALGORITHM = "SHA-512";

private static final int HASH_START_INDEX = 29;

private final int derivedKeyLength;

private final boolean digestBcryptHash;

/**
* Key Deriviation Bcrypt Secure Hasher with specified Derived Key Length
*
* @param derivedKeyLength Derived Key Length in bytes
*/
public KeyDerivationBcryptSecureHasher(final int derivedKeyLength) {
this.derivedKeyLength = derivedKeyLength;
this.digestBcryptHash = false;
}

/**
* Key Deriviation Bcrypt Secure Hasher with specified Derived Key Length and Cost Parameters
*
* @param derivedKeyLength Derived Key Length in bytes
* @param cost Cost Parameter for calculation
* @param digestBcryptHash Enable to disable digesting of bcrypt hash to support legacy derivation functions
*/
public KeyDerivationBcryptSecureHasher(final int derivedKeyLength, final int cost, final boolean digestBcryptHash) {
super(cost);
this.derivedKeyLength = derivedKeyLength;
this.digestBcryptHash = digestBcryptHash;
}

/**
* Hash raw bytes using provided salt and then leverage SHA-512 to digest the results and truncate to length requested
*
* @param input Raw bytes to be hashed
* @param rawSalt Raw salt bytes to be hashed
* @return Hash bytes digested using SHA-512 and truncated to derived key length configured
*/
@Override
byte[] hash(final byte[] input, final byte[] rawSalt) {
final byte[] costSaltBcryptHash = super.hash(input, rawSalt);

final MessageDigest messageDigest = getMessageDigest();
byte[] digest;
if (digestBcryptHash) {
LOGGER.warn("Using Legacy Key Derivation on bcrypt hash including cost and salt");
digest = messageDigest.digest(costSaltBcryptHash);
} else {
// Remove cost and salt from bcrypt function results and retain bcrypt hash
byte[] hash = Arrays.copyOfRange(costSaltBcryptHash, HASH_START_INDEX, costSaltBcryptHash.length);
digest = messageDigest.digest(hash);
}

return Arrays.copyOf(digest, derivedKeyLength);
}

private MessageDigest getMessageDigest() {
try {
return MessageDigest.getInstance(DIGEST_ALGORITHM);
} catch (final NoSuchAlgorithmException e) {
throw new UnsupportedOperationException(DIGEST_ALGORITHM, e);
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,15 @@ public PBKDF2SecureHasher() {
this(DEFAULT_PRF, DEFAULT_ITERATION_COUNT, 0, DEFAULT_DK_LENGTH);
}

/**
* Instantiates a PBKDF2 secure hasher with the default number of iterations and the default PRF using the specified derived key length.
*
* @param dkLength Derived Key Length in bytes
*/
public PBKDF2SecureHasher(final int dkLength) {
this(DEFAULT_PRF, DEFAULT_ITERATION_COUNT, 0, dkLength);
}

/**
* Instantiates a PBKDF2 secure hasher with the provided number of iterations and derived key (output) length in bytes, using the default PRF ({@code SHA512}).
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,14 @@ public ScryptSecureHasher() {
this(DEFAULT_N, DEFAULT_R, DEFAULT_P, DEFAULT_DK_LENGTH, 0);
}

/**
* Instantiates an Scrypt secure hasher using the default cost parameters and specified derived key length
* @param dkLength Derived Key Length
*/
public ScryptSecureHasher(final int dkLength) {
this(DEFAULT_N, DEFAULT_R, DEFAULT_P, dkLength, 0);
}

/**
* Instantiates an Scrypt secure hasher using the provided cost parameters. A static
* {@link #DEFAULT_SALT_LENGTH} byte salt will be generated on every hash request.
Expand Down
51 changes: 43 additions & 8 deletions nifi-docs/src/main/asciidoc/administration-guide.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -1614,14 +1614,49 @@ It is preferable to request upstream/downstream systems to switch to link:https:
[[nifi_sensitive_props_key]]
== Encrypted Passwords in Flows

NiFi always stores all sensitive values (passwords, tokens, and other credentials) populated into a flow in an encrypted format on disk. The encryption algorithm used is specified by `nifi.sensitive.props.algorithm` and the password from which the encryption key is derived is specified by `nifi.sensitive.props.key` in _nifi.properties_ (see <<security_configuration,Security Configuration>> for additional information). Prior to version 1.12.0, the list of available algorithms was all password-based encryption (PBE) algorithms supported by the `EncryptionMethod` enum in that version. Unfortunately many of these algorithms are provided for legacy compatibility, and use weak key derivation functions and block cipher algorithms & modes of operation. In 1.12.0, a pair of custom algorithms was introduced for security-conscious users looking for more robust protection of the flow sensitive values. These options combine the <<argon2-kdf, Argon2id>> KDF with reasonable cost parameters (2^16^ or `65,536 KB` of memory, `5` iterations, and parallelism `8`) with an authenticated encryption with associated data (AEAD) mode of operation, `AES-G/CM` (Galois Counter Mode). The algorithms are specified as:

* `NIFI_ARGON2_AES_GCM_256` -- 256-bit key length
* `NIFI_ARGON2_AES_GCM_128` -- 128-bit key length

Both options require a password (`nifi.sensitive.props.key` value) of *at least 12 characters*. This means the "default" value (if left empty, a hard-coded default is used) will not be sufficient.

These options provide a bridge solution to higher security without requiring a change to the structure of _nifi.properties_. A more full-featured configuration process, allowing for arbitrary combinations of KDFs and encryption algorithms, will be added in a future release. See link:https://issues.apache.org/jira/browse/NIFI-7668[NIFI-7668^] and link:https://issues.apache.org/jira/browse/NIFI-7670[NIFI-7670^] for more details.
NiFi always stores all sensitive values (passwords, tokens, and other credentials) populated into a flow in an encrypted format on disk.
The encryption algorithm used is specified by `nifi.sensitive.props.algorithm` and the password from which the encryption key is derived is specified by `nifi.sensitive.props.key` in _nifi.properties_ (see <<security_configuration,Security Configuration>> for additional information).
Prior to version 1.12.0, the list of available algorithms was all password-based encryption (PBE) algorithms supported by the `EncryptionMethod` enum in that version.
Unfortunately many of these algorithms are provided for legacy compatibility, and use weak key derivation functions and block cipher algorithms & modes of operation.
In 1.12.0, a pair of custom algorithms was introduced for security-conscious users looking for more robust protection of the flow sensitive values.

NiFi supports several configuration options to provide authenticated encryption with associated data (AEAD) using AES Galois/Counter Mode (AES-GCM).
These algorithms use a strong Key Derivation Function to derive a secret key of specified length based on the sensitive properties key configured.
Each Key Derivation Function uses a static salt in order to support flow configuration comparison across cluster nodes.
Each Key Derivation Function also uses default iteration and cost parameters as defined in the associated secure hashing implementation class.

The following strong encryption methods can be configured in the `nifi.sensitive.props.algorithm` property:

* `NIFI_ARGON2_AES_GCM_128`
* `NIFI_ARGON2_AES_GCM_256`
* `NIFI_BCRYPT_AES_GCM_128`
* `NIFI_BCRYPT_AES_GCM_256`
* `NIFI_PBKDF2_AES_GCM_128`
* `NIFI_PBKDF2_AES_GCM_256`
* `NIFI_SCRYPT_AES_GCM_128`
* `NIFI_SCRYPT_AES_GCM_256`

Each Key Derivation Function uses the following default parameters:

* Argon2
** Iterations: 5
** Memory: 65536 KB
** Parallelism: 8

* Bcrypt
** Cost: 12
** Derived Key Digest Algorithm: SHA-512

* PBKDF2
** Iterations: 160,000
** Pseudorandom Function Family: SHA-512

* Scrypt
** Cost Factor (N): 16384
** Block Size Factor (r): 8
** Parallelization Factor (p): 1

All options require a password (`nifi.sensitive.props.key` value) of *at least 12 characters*. This means the "default" value (if left empty, a hard-coded default is used) will not be sufficient.

[[encrypt-config_tool]]
== Encrypted Passwords in Configuration Files
Expand Down
Loading

0 comments on commit 5608f43

Please sign in to comment.