Could not decrypt key with given passphrase

[wbt]

I am getting a reliably reproducible error "Could not decrypt key with given passphrase" when attempting to use the --password command-line argument to unlock an account on a mining node. Omitting that command line flag, and on prompt interactively entering the same password that the file contains (below, that's "issuedemo1"), works as expected.

Below is a shell session showing a close to minimal reproducible example, developed on Windows 10.

Shell> mkdir issuedemo
Shell> cd issuedemo
Shell> geth version
INFO [07-30|10:05:54.542] Bumping default cache on mainnet         provided=1024 updated=4096
Geth
Version: 1.9.1-stable
Git Commit: b7b2f60f866d92e75db5bfe5be51b48eadeb3150
Git Commit Date: 20190724
Architecture: amd64
Protocol Versions: [63]
Network Id: 1
Go Version: go1.12.7
Operating System: windows
GOPATH=
GOROOT=C:\go
Shell> geth --datadir node1/ account new
INFO [07-30|10:06:07.556] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [07-30|10:06:07.561] Maximum peer count                       ETH=50 LES=0 total=50
Your new account is locked with a password. Please give a password. Do not forget this password.
Passphrase: issuedemo1
Repeat passphrase: issuedemo1
Your new key was generated
Public address of the key:   0xD6Fe976DCC6FdB1a383B0ed2e2b1147DDEc20a9a
Path of the secret key file: node1\keystore\UTC--2019-07-30T14-06-28.729791800Z--d6fe976dcc6fdb1a383b0ed2e2b1147ddec20a9a
- You can share your public address with anyone. Others need it to interact with you.
- You must NEVER share the secret key with anyone! The key controls access to your funds!
- You must BACKUP your key file! Without the key, it's impossible to access account funds!
- You must REMEMBER your password! Without the password, it's impossible to decrypt the key!

Shell> geth --datadir node2/ account new
INFO [07-30|10:06:38.080] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [07-30|10:06:38.085] Maximum peer count                       ETH=50 LES=0 total=50
Your new account is locked with a password. Please give a password. Do not forget this password.
Passphrase: issuedemo2
Repeat passphrase: issuedemo2
Your new key was generated
Public address of the key:   0x7c7A06D87016726813Bf26e2e1a256793d22b1Ef
Path of the secret key file: node2\keystore\UTC--2019-07-30T14-06-45.152784600Z--7c7a06d87016726813bf26e2e1a256793d22b1ef
- You can share your public address with anyone. Others need it to interact with you.
- You must NEVER share the secret key with anyone! The key controls access to your funds!
- You must BACKUP your key file! Without the key, it's impossible to access account funds!
- You must REMEMBER your password! Without the password, it's impossible to decrypt the key!

Shell> echo 'issuedemo1' > node1/password.txt
Shell> echo 'issuedemo2' > node2/password.txt
Shell> puppeth
+-----------------------------------------------------------+
| Welcome to puppeth, your Ethereum private network manager |
|                                                           |
| This tool lets you create a new Ethereum network down to  |
| the genesis block, bootnodes, miners and ethstats servers |
| without the hassle that it would normally entail.         |
|                                                           |
| Puppeth uses SSH to dial in to remote servers, and builds |
| its network components out of Docker containers using the |
| docker-compose toolset.                                   |
+-----------------------------------------------------------+

Please specify a network name to administer (no spaces, hyphens or capital letters please)
> issuedemo

Sweet, you can set this via --network=issuedemo next time!
�[32mINFO �[0m[07-30|10:07:16.132] Administering Ethereum network           �[32mname�[0m=issuedemo
�[33mWARN �[0m[07-30|10:07:16.134] No previous configurations found         �[33mpath�[0m=.puppeth\\issuedemo
What would you like to do? (default = stats)
 1. Show network stats
 2. Configure new genesis
 3. Track new remote server
 4. Deploy network components
> 2

What would you like to do? (default = create)
 1. Create new genesis from scratch
 2. Import already existing genesis
> 1

Which consensus engine to use? (default = clique)
 1. Ethash - proof-of-work
 2. Clique - proof-of-authority
> 2

How many seconds should blocks take? (default = 15)
> 15

Which accounts are allowed to seal? (mandatory at least one)
> 0xD6Fe976DCC6FdB1a383B0ed2e2b1147DDEc20a9a
> 0x7c7A06D87016726813Bf26e2e1a256793d22b1Ef
> 0x

Which accounts should be pre-funded? (advisable at least one)
> 0xD6Fe976DCC6FdB1a383B0ed2e2b1147DDEc20a9a
> 0x7c7A06D87016726813Bf26e2e1a256793d22b1Ef
> 0x

Should the precompile-addresses (0x1 .. 0xff) be pre-funded with 1 wei? (advisable yes)
> yes

Specify your chain/network ID if you want an explicit one (default = random)
> 1234
�[32mINFO �[0m[07-30|10:08:06.982] Configured new genesis block

What would you like to do? (default = stats)
 1. Show network stats
 2. Manage existing genesis
 3. Track new remote server
 4. Deploy network components
> 2

 1. Modify existing configurations
 2. Export genesis configurations
 3. Remove genesis configuration
> 2

Which folder to save the genesis specs into? (default = current)
  Will create issuedemo.json, issuedemo-aleth.json, issuedemo-harmony.json, issuedemo-parity.json
>
�[32mINFO �[0m[07-30|10:08:17.511] Saved native genesis chain spec          �[32mpath�[0m=issuedemo.json
�[31mERROR�[0m[07-30|10:08:17.511] Failed to create Aleth chain spec        �[31merr�[0m="unsupported consensus engine"
�[31mERROR�[0m[07-30|10:08:17.511] Failed to create Parity chain spec       �[31merr�[0m="unsupported consensus engine"
�[32mINFO �[0m[07-30|10:08:17.513] Saved genesis chain spec                 �[32mclient�[0m=harmony �[32mpath�[0m=issuedemo-harmony.json

What would you like to do? (default = stats)
 1. Show network stats
 2. Manage existing genesis
 3. Track new remote server
 4. Deploy network components
> ^C
�[35mCRIT �[0m[07-30|10:08:25.783] Failed to read user input                �[35merr�[0m=EOF
Shell> geth --datadir node1/ init issuedemo.json
INFO [07-30|10:08:43.590] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [07-30|10:08:43.597] Maximum peer count                       ETH=50 LES=0 total=50
INFO [07-30|10:08:43.641] Allocated cache and file handles         database=C:\\mypath\\issuedemo\\node1\\geth\\chaindata cache=16.00MiB handles=16
INFO [07-30|10:08:43.659] Writing custom genesis block
INFO [07-30|10:08:43.669] Persisted trie from memory database      nodes=358 size=51.03KiB time=1.9951ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [07-30|10:08:43.674] Successfully wrote genesis state         database=chaindata hash=e5a5b8…a6e5f0
INFO [07-30|10:08:43.677] Allocated cache and file handles         database=C:\\mypath\\issuedemo\\node1\\geth\\lightchaindata cache=16.00MiB handles=16
INFO [07-30|10:08:43.694] Writing custom genesis block
INFO [07-30|10:08:43.704] Persisted trie from memory database      nodes=358 size=51.03KiB time=1.9951ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [07-30|10:08:43.709] Successfully wrote genesis state         database=lightchaindata hash=e5a5b8…a6e5f0
Shell> geth --datadir node2/ init issuedemo.json
INFO [07-30|10:08:49.350] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [07-30|10:08:49.356] Maximum peer count                       ETH=50 LES=0 total=50
INFO [07-30|10:08:49.403] Allocated cache and file handles         database=C:\\mypath\\issuedemo\\node2\\geth\\chaindata cache=16.00MiB handles=16
INFO [07-30|10:08:49.433] Writing custom genesis block
INFO [07-30|10:08:49.448] Persisted trie from memory database      nodes=358 size=51.03KiB time=1.9948ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [07-30|10:08:49.454] Successfully wrote genesis state         database=chaindata hash=e5a5b8…a6e5f0
INFO [07-30|10:08:49.457] Allocated cache and file handles         database=C:\\mypath\\issuedemo\\node2\\geth\\lightchaindata cache=16.00MiB handles=16
INFO [07-30|10:08:49.475] Writing custom genesis block
INFO [07-30|10:08:49.484] Persisted trie from memory database      nodes=358 size=51.03KiB time=996.8µs  gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [07-30|10:08:49.489] Successfully wrote genesis state         database=lightchaindata hash=e5a5b8…a6e5f0
Shell> bootnode -genkey boot.key
Shell> bootnode -nodekey boot.key -verbosity 9 -addr :54321
enode://b14e7d0152be40b87530a7e6b07ce57197a2edf7afb7a18ef94020732355573c0e771c61372c6b2b92454fe3af9b02a28e9a22b61613570c3baef465a0581a3c@127.0.0.1:0?discport=54321
Note: you're using cmd/bootnode, a developer tool.
We recommend using a regular node as bootstrap node for production deployments.

In a new shell window:

Shell> cd issuedemo
Shell> # The "enode://..." in the next line should be copied from the output of the previous command.  The address is copied from the address created for node1 above.  
Shell> geth --datadir node1/ --syncmode 'full' --port 2345 --ipcdisable -bootnodes 'enode://b14e7d0152be40b87530a7e6b07ce57197a2edf7afb7a18ef94020732355573c0e771c61372c6b2b92454fe3af9b02a28e9a22b61613570c3baef465a0581a3c@127.0.0.1:54321' --networkid 1234 --mine -unlock '0xD6Fe976DCC6FdB1a383B0ed2e2b1147DDEc20a9a' --password 'node1/password.txt'
INFO [07-30|10:11:06.291] Maximum peer count                       ETH=50 LES=0 total=50
INFO [07-30|10:11:06.334] Starting peer-to-peer node               instance=Geth/v1.9.1-stable-b7b2f60f/windows-amd64/go1.12.7
INFO [07-30|10:11:06.337] Allocated trie memory caches             clean=256.00MiB dirty=256.00MiB
INFO [07-30|10:11:06.339] Allocated cache and file handles         database=C:\\mypath\\issuedemo\\node1\\geth\\chaindata cache=512.00MiB handles=8192
INFO [07-30|10:11:06.420] Opened ancient database                  database=C:\\mypath\\issuedemo\\node1\\geth\\chaindata\\ancient
INFO [07-30|10:11:06.425] Initialised chain configuration          config="{ChainID: 1234 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0  Petersburg: 0 Engine: clique}"
INFO [07-30|10:11:06.431] Initialising Ethereum protocol           versions=[63] network=1234 dbversion=<nil>
WARN [07-30|10:11:06.434] Upgrade blockchain database version      from=<nil> to=7
INFO [07-30|10:11:06.485] Loaded most recent local header          number=0 hash=e5a5b8…a6e5f0 td=1 age=3m42s
INFO [07-30|10:11:06.488] Loaded most recent local full block      number=0 hash=e5a5b8…a6e5f0 td=1 age=3m42s
INFO [07-30|10:11:06.490] Loaded most recent local fast block      number=0 hash=e5a5b8…a6e5f0 td=1 age=3m42s
INFO [07-30|10:11:06.494] Regenerated local transaction journal    transactions=0 accounts=0
INFO [07-30|10:11:06.497] Stored checkpoint snapshot to disk       number=0 hash=e5a5b8…a6e5f0
INFO [07-30|10:11:06.512] New local node record                    seq=1 id=89f70454847a2d62 ip=127.0.0.1 udp=2345 tcp=2345
INFO [07-30|10:11:06.515] Started P2P networking                   self=enode://e1968878ceb6ebbfb5a3c14a1e707b48cf88afec8cbe30d69805ca71f35a8292249c86fb84c4df0b71d07513fbaec3a3901acbb85ffcf6e55b96824f715b7188@127.0.0.1:2345
Fatal: Failed to unlock account 0xD6Fe976DCC6FdB1a383B0ed2e2b1147DDEc20a9a (could not decrypt key with given passphrase)

Back in the Bootnode window, additional output:

INFO [07-30|10:09:15.114] New local node record                    seq=1 id=324e37bb7dca8812 ip=<nil> udp=0 tcp=0
TRACE[07-30|10:11:06.516] << PING/v4                               id=89f70454847a2d62 addr=127.0.0.1:2345 err=nil
TRACE[07-30|10:11:06.517] >> PONG/v4                               id=89f70454847a2d62 addr=127.0.0.1:2345 err=nil
TRACE[07-30|10:11:06.523] >> PING/v4                               id=89f70454847a2d62 addr=127.0.0.1:2345 err=nil
TRACE[07-30|10:11:06.524] << PONG/v4                               id=89f70454847a2d62 addr=127.0.0.1:2345 err=nil
TRACE[07-30|10:11:06.526] << FINDNODE/v4                           id=89f70454847a2d62 addr=127.0.0.1:2345 err=nil
TRACE[07-30|10:11:06.527] >> NEIGHBORS/v4                          id=89f70454847a2d62 addr=127.0.0.1:2345 err=nil
TRACE[07-30|10:11:07.018] << FINDNODE/v4                           id=89f70454847a2d62 addr=127.0.0.1:2345 err=nil
TRACE[07-30|10:11:07.019] >> NEIGHBORS/v4                          id=89f70454847a2d62 addr=127.0.0.1:2345 err=nil
TRACE[07-30|10:11:07.520] << FINDNODE/v4                           id=89f70454847a2d62 addr=127.0.0.1:2345 err=nil
TRACE[07-30|10:11:07.520] >> NEIGHBORS/v4                          id=89f70454847a2d62 addr=127.0.0.1:2345 err=nil
TRACE[07-30|10:11:08.020] << FINDNODE/v4                           id=89f70454847a2d62 addr=127.0.0.1:2345 err=nil
TRACE[07-30|10:11:08.020] >> NEIGHBORS/v4                          id=89f70454847a2d62 addr=127.0.0.1:2345 err=nil
TRACE[07-30|10:11:08.521] << FINDNODE/v4                           id=89f70454847a2d62 addr=127.0.0.1:2345 err=nil
TRACE[07-30|10:11:08.521] >> NEIGHBORS/v4                          id=89f70454847a2d62 addr=127.0.0.1:2345 err=nil
TRACE[07-30|10:11:11.533] >> PING/v4                               id=89f70454847a2d62 addr=127.0.0.1:2345 err=nil
DEBUG[07-30|10:11:12.034] Removed dead node                        b=16 id=89f70454847a2d62 ip=127.0.0.1 checks=0

Omitting the quotes around the password filename produces the same results.

Opening node1/password.txt in Notepad, I can see it contains "issuedemo1" followed by a line break and then the end of the file. Removing that line break, saving, closing, and trying again produces the same results, though on a subsequent reopen of the file the closing line break is no longer there (matching expectations). Clearing the file out completely, as concerningly suggested by comments on this answer, saving, and retrying, also has the same effect, but that is as expected.

The expected behavior is that the account is unlocked instead of geth terminating with error Fatal: Failed to unlock account. Alternative expected behavior is a better error message indicating what went wrong and how I might fix it, as this might be a simple error.

[karalabe]

My hunch is that the first time you had an extra line break from echo, afterwards Notepad might have changed some of the binary data (e.g. set some encoding on the file). Please upload the file in question to this issue.

[wbt]

The file created by echo, recreated from the command above and renamed:
password-echo.txt
That file, opened in Notepad, cursor placed at the end, one backspace to remove the line break, saved closed, and renamed:
password-linebreak-removed.txt
With the password contents removed (not expected to work):
password-empty.txt

I would support a rule that a trim() operation be applied to all passwords, making leading and trailing whitespace characters no longer an impactful part of a password, at the next breaking-change release, if it isn't there already.

[karalabe]

So, here's the reason it fails (your three files, dumped out with hexdump):

$ hexdump password-echo.txt 
0000000 feff 0069 0073 0073 0075 0065 0064 0065
0000010 006d 006f 0031 000d 000a

$ hexdump password-linebreak-removed.txt 
0000000 feff 0069 0073 0073 0075 0065 0064 0065
0000010 006d 006f 0031

$ hexdump password-empty.txt 
0000000 feff

All of the files actually include 2 invisible prefix bytes, which Windows (Notpad) probably prepends in there to act as a byte order mark. From Wikipedia:

Although UTF-8 does not suffer from endianness problems, many Microsoft Windows programs (i.e. Notepad) prepend the contents of UTF-8-encoded files with BOM,[2] to differentiate UTF-8 encoding from other 8-bit encodings.[3]

Additionally, all the characters in your password files are actually represented by 2 bytes, whereas 1 is enough. E.g. here's the same password echoed out under Linux and hexdumped:

$ echo issuedemo1 > password-mine.txt
$ hexdump password-mine.txt 
0000000 7369 7573 6465 6d65 316f 000a          

My hunch is that Geth simply loads the content of the file as a plain blob and expects it to work as is. We don't expect that some Windows program will inject extra characters that it interprets as metadata and not real data.

CC @holiman ^ interesting issue on Windows

[wbt]

The first one was created by echo (in Windows PowerShell), not Notepad. It appears Notepad then just ignores these.

[SjonHortensius]

while in theory the contents could be converted, instead of going down the unicode rabbithole I'd instead suggest failing in MakePasswordList when a BOM is detected

[FrankSzendzielarz]

Windows powershell echo defaults to specific utf encodings depending on the platform. Echo can be modified as follows in powershell:
echo "test:" | out-file test.txt -encoding ASCII
More info here: https://stackoverflow.com/questions/40098771/changing-powershells-default-output-encoding-to-utf-8

[adamschmideg]

We should add this issue to our FAQ