Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NIFI-4942 Add capability for encrypt-config tool to use securely hashed key/password for demonstration of previous knowledge #2628

Closed
wants to merge 10 commits into from

Conversation

alopresto
Copy link
Contributor

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

  • Is there a JIRA ticket associated with this PR? Is it referenced
    in the commit message?

  • Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

  • Has your PR been rebased against the latest commit within the target branch (typically master)?

  • Is your initial contribution a single, squashed commit?

For code changes:

  • Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
  • Have you written or updated unit tests to verify your changes?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
  • If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
  • If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

  • Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

…nfig toolkit.

Added test resource for Python scrypt implementation/verifier.
Added unit tests.
Added logic to check hashed password/key.
Added logic to retrieve secure hash from file to compare.
Added unit tests (125/125).
Added logic to return current hash params in JSON for Ambari to consume.
Fixed typos in error messages.
Added unit tests (129/129).
@alopresto
Copy link
Contributor Author

Here are some instructions and expected outputs to demonstrate that the tool works as intended:

# CD to $NIFI_HOME/conf because secure_hash.key must be written to immediate directory

# Populate sensitive properties in nifi.properties in order for something to be encrypted
sed 's/asswd=/&thisIsABadPassword/' nifi.properties >nifi-sensitive.properties

# Initial encryption of nifi.properties
../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v \
-b bootstrap.conf \
-n nifi-sensitive.properties \
-o nifi-encrypted.properties \
-p passwordpassword

# Example hashes for "passwordpassword"
# secureHashKey=$s0$100801$H8N5sEErC9hOVpQLxUt+oA$RrwImM1uWD59KuA1AxFamK7oPHlnI1uBXEN2lt4CpbM
# secureHashPassword=$s0$100801$dZ04VTEBHxTR8tb6j29q/w$mXsXKxvd3nYXXOSoxobO7gkLaLAdz2dZRqAvPNfOzWE

# Verify secure_hash.key file generated and populated w/ both key and password hash
more secure_hash.key

# Derived key for "passwordpassword"
# nifi.bootstrap.sensitive.key=A2EA52795B33AB2F21C93E7E820D08369F1448478C877F4C710D6E85FD904AE6

# Verify bootstrap.conf file updated with master key value
more bootstrap.conf

# Verify encryption of sensitive properties occurred
more nifi-sensitive.properties | grep 'assw'
more nifi-encrypted.properties | grep 'assw'

# Migration using raw password
../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m \
-b bootstrap.conf \
-n nifi-encrypted.properties \
-o nifi-migrated.properties \
-p thisIsABadPassword \
-w passwordpassword

# Example hashes for "thisIsABadPassword"
# secureHashKey=$s0$100801$Y5rcY+pECpOBw5JBT1esMw$OEfnR/cze9u6ZjHMbd6NzvQltz2cC0qskSH8XeiXcp4
# secureHashPassword=$s0$100801$rxjtgO5m859l6aI1xHIjpA$jAqTpGrJNiTkcIei6HtbCuZmhkPnqDlC3G4RjxRtf18

# Migration using hashed password (single quote escape hash to avoid dollar-sign variable evaluation)
../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m \
-b bootstrap.conf \
-n nifi-migrated.properties \
-o nifi-migrated-from-hash.properties \
-p thisIsABadPassword2 \
-z '$s0$100801$rxjtgO5m859l6aI1xHIjpA$jAqTpGrJNiTkcIei6HtbCuZmhkPnqDlC3G4RjxRtf18'

# Example output
hw12203:.../nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf (NIFI-4942) alopresto
🔓 174714s @ 14:37:45 $ ../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v \
> -m -b bootstrap.conf \
> -n nifi-migrated.properties \
> -o nifi-migrated-from-hash.properties \
> -p thisIsABadPassword2 \
> -z '$s0$100801$rxjtgO5m859l6aI1xHIjpA$jAqTpGrJNiTkcIei6HtbCuZmhkPnqDlC3G4RjxRtf18'
Listening for transport dt_socket at address: 8000
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool:        bootstrap.conf:               bootstrap.conf
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  nifi.properties:              nifi-migrated.properties
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties:              nifi-migrated-from-hash.properties
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  login-identity-providers.xml: null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  authorizers.xml:              null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml:              null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  flow.xml.gz:                  null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz:                  null
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Key migration mode activated
2018/04/11 14:38:28 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Secure hash argument present
2018/04/11 14:38:39 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 151 properties from /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf/nifi-migrated.properties
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 151 properties from /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf/nifi-migrated.properties
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ProtectedNiFiProperties: There are 3 protected properties of 5 sensitive properties (100%)
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 147 properties
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.keyPasswd with aes/gcm/256 -> RDHo9b1qNOhGsvke||lOPiNxqlII0l6V07KsKQKSXgaZPmphdKLnL9QQB9LsGLLQ
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.keyPasswd.protected
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.keystorePasswd with aes/gcm/256 -> 	/IefK+XgVTpqGSFt||5O8oPFVXMYc6T1WK99++EWJ92V1TIlF9M7+H7xDnno6+mw
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.keystorePasswd.protected
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.truststorePasswd with aes/gcm/256 -> 	NyF6m6sg7ZH+eZF1||/ex+7LMFmD9iZ964k/BwlzA+Gd9z6Uuv5S5xv2oHQc9ynQ
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.truststorePasswd.protected
2018/04/11 14:39:03 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Final result: 150 keys including 3 protected keys

# Example hashes for "thisIsABadPassword2"
# secureHashKey=$s0$100801$HdGObf10x+ou+Vru5kns4A$U7kDSAwpoyvrWg7P9iRZxersUPln51TQ2IyKJYkiuTI
# secureHashPassword=$s0$100801$h+9JoIzRU8ENNrKLECVPSQ$rZoWTxMe0er7EwW+ZtaWc1TF5N1oV19y1LXK4AjgZ/A

# Migration using hashed key
../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m \
-b bootstrap.conf \
-n nifi-migrated-from-hash.properties \
-o nifi-migrated-from-hash-using-key.properties \
-p thisIsABadPassword3 \
-y '$s0$100801$HdGObf10x+ou+Vru5kns4A$U7kDSAwpoyvrWg7P9iRZxersUPln51TQ2IyKJYkiuTI'

# Example hashes for "thisIsABadPassword3"
# secureHashKey=$s0$100801$t7L8nq9Bc2gtbCnodjKG+A$i4z4dFHnV7CL3LlC5yt8N56me4pQUMx1Z0buZw0MsHA
# secureHashPassword=$s0$100801$SDL+y7LI1HHxeBo4SEe1ig$OtfQWc4cHtTd0msxbgYytVjS0mB2VDm/z8S78sBjcGI

# Migration using "correct" hashed password but with incorrect master key
../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m \
-b bootstrap.conf \
-n nifi-migrated.properties \
-o nifi-migrated-from-hash.properties \
-p thisIsABadPassword2 \
-z '$s0$100801$t7L8nq9Bc2gtbCnodjKG+A$i4z4dFHnV7CL3LlC5yt8N56me4pQUMx1Z0buZw0MsHA'

# Example output
hw12203:.../nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf (NIFI-4942) alopresto
🔓 175013s @ 14:42:44 $ ../../../../../nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m \
> -b bootstrap.conf \
> -n nifi-migrated.properties \
> -o nifi-migrated-from-hash.properties \
> -p thisIsABadPassword2 \
> -z '$s0$100801$t7L8nq9Bc2gtbCnodjKG+A$i4z4dFHnV7CL3LlC5yt8N56me4pQUMx1Z0buZw0MsHA'
Listening for transport dt_socket at address: 8000
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool:        bootstrap.conf:               bootstrap.conf
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  nifi.properties:              nifi-migrated.properties
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties:              nifi-migrated-from-hash.properties
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  login-identity-providers.xml: null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  authorizers.xml:              null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml:              null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  flow.xml.gz:                  null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz:                  null
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Key migration mode activated
2018/04/11 14:45:58 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Secure hash argument present
2018/04/11 14:46:15 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 151 properties from /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf/nifi-migrated.properties
2018/04/11 14:46:21 ERROR [main] org.apache.nifi.properties.ConfigEncryptionTool: Encountered an error
java.security.InvalidKeyException: The provided hashed key/password is not correct
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
	at org.apache.nifi.properties.ConfigEncryptionTool.getMigrationKey(ConfigEncryptionTool.groovy:553)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
	at org.apache.nifi.properties.ConfigEncryptionTool.main(ConfigEncryptionTool.groovy:1677)
	at org.apache.nifi.properties.ConfigEncryptionTool$main.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.apache.nifi.toolkit.encryptconfig.LegacyMode.run(LegacyMode.groovy:30)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain.main(EncryptConfigMain.groovy:109)
The provided hashed key/password is not correct

usage: org.apache.nifi.properties.ConfigEncryptionTool [-h] [-v] [-n <file>] [-o <file>] [-l <file>] [-i <file>] [-a <file>] [-u <file>] [-f <file>] [-g <file>]
       [-b <file>] [-k <keyhex>] [-e <keyhex>] [-p <password>] [-w <password>] [-y <hashed_keyhex>] [-z <hashed_password>] [-r] [-m] [-x] [-s <password|keyhex>]
       [-A <algorithm>] [-P <algorithm>] [--currentHashParams]

This tool reads from a nifi.properties and/or login-identity-providers.xml file with plain sensitive configuration values, prompts the user for a master key,
and encrypts each value. It will replace the plain value with the protected value in the same file (or write to a new file if specified). It can also be used to
migrate already-encrypted values in those files or in flow.xml.gz to be encrypted with a new key.

 -h,--help                                   Show usage information (this message)
 -v,--verbose                                Sets verbose mode (default false)
 -n,--niFiProperties <file>                  The nifi.properties file containing unprotected config values (will be overwritten unless -o is specified)
 -o,--outputNiFiProperties <file>            The destination nifi.properties file containing protected config values (will not modify input nifi.properties)
 -l,--loginIdentityProviders <file>          The login-identity-providers.xml file containing unprotected config values (will be overwritten unless -i is
                                             specified)
 -i,--outputLoginIdentityProviders <file>    The destination login-identity-providers.xml file containing protected config values (will not modify input
                                             login-identity-providers.xml)
 -a,--authorizers <file>                     The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)
 -u,--outputAuthorizers <file>               The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)
 -f,--flowXml <file>                         The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)
 -g,--outputFlowXml <file>                   The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)
 -b,--bootstrapConf <file>                   The bootstrap.conf file to persist master key
 -k,--key <keyhex>                           The raw hexadecimal key to use to encrypt the sensitive properties
 -e,--oldKey <keyhex>                        The old raw hexadecimal key to use during key migration
 -p,--password <password>                    The password from which to derive the key to use to encrypt the sensitive properties
 -w,--oldPassword <password>                 The old password from which to derive the key during migration
 -y,--secureHashKey <hashed_keyhex>          The old securely-hashed hexadecimal key to authenticate during key migration (see NiFi Admin Guide)
 -z,--secureHashPassword <hashed_password>   The old securely-hashed password to authenticate during key migration (see NiFi Admin Guide)
 -r,--useRawKey                              If provided, the secure console will prompt for the raw key value in hexadecimal form
 -m,--migrate                                If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted with
                                             a new key
 -x,--encryptFlowXmlOnly                     If provided, the properties in flow.xml.gz will be re-encrypted with a new key but the nifi.properties and/or
                                             login-identity-providers.xml files will not be modified
 -s,--propsKey <password|keyhex>             The password or key to use to encrypt the sensitive processor properties in flow.xml.gz
 -A,--newFlowAlgorithm <algorithm>           The algorithm to use to encrypt the sensitive processor properties in flow.xml.gz
 -P,--newFlowProvider <algorithm>            The security provider to use to encrypt the sensitive processor properties in flow.xml.gz
    --currentHashParams                      Returns the current salt and cost params used to store the hashed key/password

Java home: /Users/alopresto/.jenv/versions/1.8
NiFi Toolkit home: /Users/alopresto/Workspace/nifi/nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT
hw12203:.../nifi/nifi-assembly/target/nifi-1.7.0-SNAPSHOT-bin/nifi-1.7.0-SNAPSHOT/conf (NIFI-4942) alopresto
🔓 175232s @ 14:46:23 $

@YolandaMDavis
Copy link
Contributor

@alopresto thanks for addressing this, happy to review. It does look like Travis is failing on a ratcheck related error in nifi-toolkit-encrypt-config

@YolandaMDavis
Copy link
Contributor

YolandaMDavis commented Apr 12, 2018

@alopresto ran through test cases and this works as expected. One question for you is there a way to designate an output location for the secure-hash.key file? Such as if I want to pipe it to stdin or just to another location?

Also as a side note I tested with -p (password) input where it may contain certain characters (@, -, and &). The '&' caused the script to fail but also to stall, needed to do a Control-C to break out of it. I'm guessing the command line is seeing it as a concatenated command. The below is resolved by simply including quotes around the password but may be good to document for users:

ydavis$ /Users/ydavis/dev/projects/nifi/nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m -b bootstrap.conf -n nifi-migrated-from-hash-key-break-2.properties -o nifi-migrated-from-hash-key-break-3.properties -p thisIs&ABadPassword4 -y '$s0$100801$j8z9NeI9DZEBTbCzOaQJbA$MI0iN/ZPQ5bk4YxcgJ2H95gCToQy3ZbIr7B6OMxB3oA'
[1] 5576
-bash: ABadPassword4: command not found
HW13535:conf ydavis$ 2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool:        bootstrap.conf:               bootstrap.conf
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  nifi.properties:              nifi-migrated-from-hash-key-break-2.properties
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties:              nifi-migrated-from-hash-key-break-3.properties
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  login-identity-providers.xml: null
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  authorizers.xml:              null
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml:              null
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  flow.xml.gz:                  null
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz:                  null
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Key migration mode activated
2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 151 properties from /Users/ydavis/dev/tools/nifi-1.7.0-SNAPSHOT/conf/nifi-migrated-from-hash-key-break-2.properties
2018/04/12 17:30:13 ERROR [main] org.apache.nifi.properties.ConfigEncryptionTool: Encountered an error
java.security.KeyException: Cannot derive key from empty/short password -- password must be at least 12 characters
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
	at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77)
	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
	at org.apache.nifi.properties.ConfigEncryptionTool.deriveKeyFromPassword(ConfigEncryptionTool.groovy:1493)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce.invoke(StaticMetaMethodSite.java:151)
	at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.callStatic(StaticMetaMethodSite.java:102)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:206)
	at org.apache.nifi.properties.ConfigEncryptionTool.getKeyInternal(ConfigEncryptionTool.groovy:527)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:190)
	at org.apache.nifi.properties.ConfigEncryptionTool.getKey(ConfigEncryptionTool.groovy:542)
	at org.apache.nifi.properties.ConfigEncryptionTool.getKey(ConfigEncryptionTool.groovy:541)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
	at org.apache.nifi.properties.ConfigEncryptionTool.main(ConfigEncryptionTool.groovy:1659)
	at org.apache.nifi.properties.ConfigEncryptionTool$main.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.apache.nifi.toolkit.encryptconfig.LegacyMode.run(LegacyMode.groovy:30)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain.main(EncryptConfigMain.groovy:109)
Cannot derive key from empty/short password -- password must be at least 12 characters

usage: org.apache.nifi.properties.ConfigEncryptionTool [-h] [-v] [-n <file>] [-o <file>] [-l <file>] [-i <file>] [-a <file>] [-u <file>] [-f <file>] [-g <file>]
       [-b <file>] [-k <keyhex>] [-e <keyhex>] [-p <password>] [-w <password>] [-y <hashed_keyhex>] [-z <hashed_password>] [-r] [-m] [-x] [-s <password|keyhex>]
       [-A <algorithm>] [-P <algorithm>] [--currentHashParams]

This tool reads from a nifi.properties and/or login-identity-providers.xml file with plain sensitive configuration values, prompts the user for a master key,
and encrypts each value. It will replace the plain value with the protected value in the same file (or write to a new file if specified). It can also be used to
migrate already-encrypted values in those files or in flow.xml.gz to be encrypted with a new key.

 -h,--help                                   Show usage information (this message)
 -v,--verbose                                Sets verbose mode (default false)
 -n,--niFiProperties <file>                  The nifi.properties file containing unprotected config values (will be overwritten unless -o is specified)
 -o,--outputNiFiProperties <file>            The destination nifi.properties file containing protected config values (will not modify input nifi.properties)
 -l,--loginIdentityProviders <file>          The login-identity-providers.xml file containing unprotected config values (will be overwritten unless -i is
                                             specified)
 -i,--outputLoginIdentityProviders <file>    The destination login-identity-providers.xml file containing protected config values (will not modify input
                                             login-identity-providers.xml)
 -a,--authorizers <file>                     The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)
 -u,--outputAuthorizers <file>               The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)
 -f,--flowXml <file>                         The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)
 -g,--outputFlowXml <file>                   The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)
 -b,--bootstrapConf <file>                   The bootstrap.conf file to persist master key
 -k,--key <keyhex>                           The raw hexadecimal key to use to encrypt the sensitive properties
 -e,--oldKey <keyhex>                        The old raw hexadecimal key to use during key migration
 -p,--password <password>                    The password from which to derive the key to use to encrypt the sensitive properties
 -w,--oldPassword <password>                 The old password from which to derive the key during migration
 -y,--secureHashKey <hashed_keyhex>          The old securely-hashed hexadecimal key to authenticate during key migration (see NiFi Admin Guide)
 -z,--secureHashPassword <hashed_password>   The old securely-hashed password to authenticate during key migration (see NiFi Admin Guide)
 -r,--useRawKey                              If provided, the secure console will prompt for the raw key value in hexadecimal form
 -m,--migrate                                If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted with
                                             a new key
 -x,--encryptFlowXmlOnly                     If provided, the properties in flow.xml.gz will be re-encrypted with a new key but the nifi.properties and/or
                                             login-identity-providers.xml files will not be modified
 -s,--propsKey <password|keyhex>             The password or key to use to encrypt the sensitive processor properties in flow.xml.gz
 -A,--newFlowAlgorithm <algorithm>           The algorithm to use to encrypt the sensitive processor properties in flow.xml.gz
 -P,--newFlowProvider <algorithm>            The security provider to use to encrypt the sensitive processor properties in flow.xml.gz
    --currentHashParams                      Returns the current salt and cost params used to store the hashed key/password

Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home
NiFi Toolkit home: /Users/ydavis/dev/projects/nifi/nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT

@YolandaMDavis
Copy link
Contributor

YolandaMDavis commented Apr 13, 2018

Spoke with @alopresto offline. He highlighted that is it important to maintain control of location of secure-hash.key file in order to prevent calling application from piping into a file that is controlled externally. We want to keep it as secured as possible.

Concerning the documentation update recommendation that can be addressed in a separate item.

+1

Will merge after reconfirming tests/rat-check.

@asfgit asfgit closed this in 6d06def Apr 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants