Azcopy login fails with failed to get keyring during saving token

[Sharathmk99]

Which version of the AzCopy was used?

AzCopy 10.4.3

Which platform are you using? (ex: Windows, Mac, Linux)

Linux
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04

What command did you run?

import os
os.system("keyctl session mysession & /<path>/azcopy login --service-principal --application-id <app id> --tenant-id=<tenant id>")
os.system("/<path>/azcopy copy <source path> <target path> --recursive")

What problem was encountered?

We are trying to invoke azcopy to copy files from Azure VM to Azure Data Lake Storage Gen2 using Python in a batch mode. On every azcopy copy command we do azcopy login. Azcopy login works most of the time, but it will fail by below error,

Failed to perform login command: 
failed to get keyring during saving token, key has been revoked 

As per issue #452 people suggested to add "keyctl session mysession" before login, but it still fails sometime within 2hrs or 24hrs

How can we reproduce the problem in the simplest way?

Execute below code in loop by adding delay of 2hrs for 48hrs,

import os
os.system("keyctl session mysession & /<path>/azcopy login --service-principal --application-id <app id> --tenant-id=<tenant id>")
os.system("/<path>/azcopy copy <source path> <target path> --recursive")

Have you found a mitigation/solution?

No

[mohsha-msft]

Hey @Sharathmk99 ,

Thanks for reaching out to us!

This is a known issue in CentOS, RHEL and some flavors of Linux.

• I see you're using Ubuntu, so I advice you to try login again.
• Alternatively, you can creating a new keyctl session. Read this to know more.
• Another workaround to this is using autologin, a feature which we released in AzCopy version 10.7.0. Please read this comment to understand how autologin works.

If this doesn't answer your question, please don't hesitate to reach out to us for more clarity.

[franklsf95]

This still doesn't work as of azcopy 10.10.0.

[franklsf95]

I was able to get a workaround according to #193 (comment), but this is still an issue.

[mcronce]

This still needs a fix. I'm using a container to run the Azure CLI so that my host system isn't polluted with a bunch of dependencies. azcopy fails in the container.

Sharing credentials with az would be a lot more sane than managing its own credentials. Barring that, simply storing them in a file in e.g. ~/.azcopy would work fine; if it's good enough for az, it should be good enough for azcopy.

[jimdigriz]

Still a problem on 10.16.1 after you wait 2+ hours for the key to be revoked, or you manually revoke it.

At a guess, it is because the saving credential code does not handle revoked keys:

azure-storage-azcopy/common/credCache_linux.go

Lines 152 to 184 in 058bd5b

	func (c *CredCache) saveTokenInternal(token OAuthTokenInfo) error {
	c.isPermSet = false
	c.key = nil
	b, err := token.toJSON()
	if err != nil {
	return fmt.Errorf("failed to marshal during saving token, %v", err)
	}
	keyring, err := keyctl.SessionKeyring()
	if err != nil {
	return fmt.Errorf("failed to get keyring during saving token, %v", err)
	}
	k, err := keyring.Add(c.keyName, b)
	if err != nil {
	return fmt.Errorf("failed to save key, %v", err)
	}
	c.key = k
	// Set permissions to only current user.
	err = keyctl.SetPerm(k, keyctl.PermUserAll)
	if err != nil {
	// which indicates Permission is by default ProcessAll
	unlinkErr := k.Unlink()
	if unlinkErr != nil {
	panic(errors.New("failed to set permission, and cannot unlink key, please logout current login session for safety consideration"))
	}
	return fmt.Errorf("failed to set permission for cached token, %v", err)
	}
	c.isPermSet = true
	return nil
	}

Once a key is revoked, all operations on it (other than prune/reap) will fail:

https://www.man7.org/linux/man-pages/man3/keyctl_revoke.3.html

A possible fix would be to purge the revoked key before doing the login so later on it can be updated.

@nakulkar-msft why mark as "won't fix"? Running 'login' should always work if the credentials are still valid, right? A revoked key should be handled far more cleanly, by purging the old key via keyctl and then letting the normal login process add a new fresh key; rather than latching in a way no one understands.

[nakulkar-msft]

@jimdigriz
'azcopy login' is old workflow and we dont recommend it. Use auto-login instead. Let us know if auto-login does not fit your workflow, we'll modify it for you.

[jimdigriz]

@nakulkar-msft the problem is not 'login' or 'auto-login', the problem is azcopy attempts to access a revoked key and that just fails and aborts

Using your described method I still get after two hours (which amusingly recommends I instead use the "old workflow and we dont recommend it"):

INFO: Authenticating to destination using Azure AD

Cannot perform sync due to error: no cached token found, please log in with azcopy's login command, key has been revoked

This is the method I am using here:

export AZCOPY_AUTO_LOGIN_TYPE="SPN"
export AZCOPY_TENANT_ID=....
export AZCOPY_SPA_APPLICATION_ID=....
export AZCOPY_SPA_CERT_PATH="$PWD/service-principle.pem"
./azcopy_linux_amd64_10.16.1/azcopy sync /stuff/ https://myaccount.blob.core.windows.net/my-container/stuff/
sleep 7200 && ./azcopy_linux_amd64_10.16.1/azcopy sync /stuff/ https://myaccount.blob.core.windows.net/my-container/stuff/

The second 'sync' fails after 2 hours with the above message.

[jimdigriz]

For those running into this bug, and vexed by MS's disinterest to assign one of its 100k programmers to knock out a two liner Python bug fix, the rest of us can workaround this by calling azcopy from a different user session which is done by prepending setsid to azcopy.

For example, I use something like this:

( while sleep 7200; do setsid nice -n19 ionice -c3 env AZCOPY_AUTO_LOGIN_TYPE="SPN" AZCOPY_TENANT_ID=11111111-2222-3333-4444-555555555555 AZCOPY_SPA_APPLICATION_ID=11111111-2222-3333-4444-555555555555 AZCOPY_SPA_CERT_PATH="$PWD/upload.pem" ./azcopy_linux_amd64_10.16.1/azcopy sync /STUFF/ https://STORAGEACCOUNT.blob.core.windows.net/CONTAINER/STUFF/; done )

Such there are plenty of 'technical' arguments that this should not be in a screen/tmux session, but when prototyping and figuring out the moving pieces early in a project you need to be doing things like this so you can focus on the harder problems.

[hholst80]

@jimdigriz 'azcopy login' is old workflow and we dont recommend it. Use auto-login instead. Let us know if auto-login does not fit your workflow, we'll modify it for you.

az login is stated on the tin. az login and az logout should just work. I think as azure customers we should expect to have tooling that works for basic services like blob storage?

az auto login does not work for me because I need to login every single time.

[thammegowda]

I got this error today on v10.24.0. The message in the logs said to use auto login and referenced [1]. But that page was not helpful as none of the autologin types specified in that wiki page worked for me.

Found additional info on [2]

So, export AZCOPY_AUTO_LOGIN_TYPE=AZCLI fixed the problem for me.

[1] https://github.com/Azure/azure-storage-azcopy/wiki/Improved-login-support-for-AzCopy-commands-(with-in-memory-secret-store)
[2] https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-authorize-azure-active-directory#authorize-with-azure-cli