`az` behaves similarly on Ubuntu as it did on windows

[jessehouwing]

Description

See:

• Improve performance of az on ubuntu hosted runners #8441
• [windows] warmup Azure CLI #8427
• [windows] warmup "az devops" for the first run #8410

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

Image: ubuntu-22.04
Version: 20230924.1.0
Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20230924.1/images/linux/Ubuntu2204-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20230924.1

Is it regression?

No

Expected behavior

No rebuild of the command cache when running az command for the first time

Logs should read:

DEBUG: cli.knack.cli: Command arguments: ['devops', 'login', '--debug']
DEBUG: cli.knack.cli: __init__ debug log:
Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f4e8dd393f0>, <function OutputProducer.on_global_arguments at 0x7f4e8dca0310>, <function CLIQuery.on_global_arguments at 0x7f4e8dac5510>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Modules found from index for 'devops': ['azext_devops']
DEBUG: cli.azure.cli.core: Loading command modules:
DEBUG: cli.azure.cli.core: Name                  Load Time    Groups  Commands
DEBUG: cli.azure.cli.core: Total (0)                 0.000         0         0
DEBUG: cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
DEBUG: cli.azure.cli.core: Loading extensions:
DEBUG: cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
DEBUG: cli.azure.cli.core: azure-devops              0.077        60       192  /opt/az/azcliextensions/azure-devops
DEBUG: cli.azure.cli.core: Total (1)                 0.077        60       192  
DEBUG: cli.azure.cli.core: Loaded 60 groups, 192 commands.
DEBUG: cli.azure.cli.core: Found a match in the command table.
DEBUG: cli.azure.cli.core: Raw command  : devops login
DEBUG: cli.azure.cli.core: Command table: devops login

No installation of keychain when running az devops login for the first time

Logs should read:

INFO: az_command_data_logger: extension name: azure-devops
INFO: az_command_data_logger: extension version: 0.26.0
DEBUG: cli.knack.prompting: No tty available.
INFO: cli.azext_devops.dev.team.credentials: Getting PAT token in non-interactive mode.
INFO: keyring.backend: Loading KWallet
INFO: keyring.backend: Loading SecretService
INFO: keyring.backend: Loading Windows
INFO: keyring.backend: Loading chainer
INFO: keyring.backend: Loading macOS
DEBUG: cli.azext_devops.dev.common.credential_store: Keyring backend : keyring.backends.fail.Keyring (priority: 0)

Actual behavior

Calling az devops login results in a command index rebuild:

DEBUG: cli.knack.cli: Command arguments: ['devops', 'login', '--debug']
DEBUG: cli.knack.cli: __init__ debug log:
Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7fb941d113f0>, <function OutputProducer.on_global_arguments at 0x7fb941c78310>, <function CLIQuery.on_global_arguments at 0x7fb941a9d510>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Command index version or cloud profile is invalid or doesn't match the current command.
DEBUG: cli.azure.cli.core: Command index has been invalidated.
DEBUG: cli.azure.cli.core: No module found from index for '['devops', 'login', '--debug']'
DEBUG: cli.azure.cli.core: Loading all modules and extensions
DEBUG: cli.azure.cli.core: Discovered command modules: ['acr', 'acs', 'advisor', 'ams', 'apim', 'appconfig', 'appservice', 'aro', 'backup', 'batch', 'batchai', 'billing', 'botservice', 'cdn', 'cloud', 'cognitiveservices', 'config', 'configure', 'consumption', 'container', 'cosmosdb', 'databoxedge', 'dla', 'dls', 'dms', 'eventgrid', 'eventhubs', 'extension', 'feedback', 'find', 'hdinsight', 'identity', 'interactive', 'iot', 'keyvault', 'kusto', 'lab', 'managedservices', 'maps', 'marketplaceordering', 'monitor', 'mysql', 'netappfiles', 'network', 'policyinsights', 'privatedns', 'profile', 'rdbms', 'redis', 'relay', 'resource', 'role', 'search', 'security', 'servicebus', 'serviceconnector', 'servicefabric', 'signalr', 'sql', 'sqlvm', 'storage', 'synapse', 'util', 'vm']
DEBUG: cli.azure.cli.core: Loading command modules:

And installation of keychain:

INFO: az_command_data_logger: extension name: azure-devops
INFO: az_command_data_logger: extension version: 0.26.0
DEBUG: cli.knack.prompting: No tty available.
INFO: cli.azext_devops.dev.team.credentials: Getting PAT token in non-interactive mode.
DEBUG: cli.azext_devops.dev.common.pip_helper: installing keyring~=17.1.1
DEBUG: cli.azure.cli.core.extension.operations: Running: ['/usr/bin/../../opt/az/bin/python3', '-m', 'pip', 'install', 'keyring~=17.1.1', '--target', '/opt/az/azcliextensions/azure-devops', '-vv', '--disable-pip-version-check', '--no-cache-dir']
DEBUG: cli.azure.cli.core.extension.operations: Using pip 23.2.1 from /opt/az/lib/python3.10/site-packages/pip (python 3.10)

Repro steps

In Azure Pipelines run the following workflow:

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
- pwsh: |
      echo "$env:ACCESSTOKEN" | az devops login --debug
      az pipelines runs show --id $env:BUILD_BUILDID --query "definition.id" --verbose --debug --organization $env:SYSTEM_COLLECTIONURI --project $env:SYSTEM_TEAMPROJECT
  env:
    ACCESSTOKEN: $(System.AccessToken)

[ilia-shipitsin]

it looks like moving complexity from Azure CLI to runner images, can this be addressed on Azure CLI (which I believe makes more sense)

[jessehouwing]

Not sure how actually, they optimize for a different goal than you.

• Theirs isolates the az stuff to the user's profile by default.
• They have a few "let's be sure about this" things that cost extra time in the runner.
• They delay/cache a bunch of stuff to the first run / first run after extension installation.

So the bits that redirect the caching folder should be in the runner image.

The installation of keyring, I completely agree should be in az devops by default, but as long as they make it optional, the runner should trigger that option.

The Python warm-up thing, is something I feel they should just stick in the installer.

I've files issues with the az team, but it's been hard getting through to them.

For now, this is a 30 line fix. You could probably skip the az --help calls, since python is already much faster on ubuntu and make it a 15 line fix. Not sure it'll ever be much smaller.

[ilia-shipitsin]

initially I thought that we would keep warm up for Windows no longer than it will be implemented on Azure CLI side.

[jessehouwing]

Yeah, the calls to az --help and az devops --help should probably be the thing that can go if it is implemented in az. But for now, extension installation doesn't trigger any code in the extension. And that might be a security feature, I haven't checked, but not immediately executing code downloaded from the webs is probably a good thing.

With regards to the perf on Windows, it's a shame that Python is a lot slower on Windows than it is on linux. The Windows image clearly has a lot more to gain. Then again... 13s... It's another 13s.

[mikhailkoliada]

Closing as commented at #8441 (comment)

[jessehouwing]

@mikhailkoliada if I remove the folder redirection? I think this should be possible on Linux. The behavior on windows would differ from linux thiugh, since windows does set all these environment variables.

Would it make sense to keep the cleanup routine? Cause now running the tests leaves telemetry and command history data from provisioning.