Cloud profile settings don't override provider settings

[gitbrianb]

Description of Issue/Question

When using the "location" attribute within AWS, if I specify the "location" attribute in the profile settings and nothing within the provider settings, it defaults to something I don't want. For instance, if I default my profile settings to "location" us-east-2, and nothing in the provider file, I've noticed in the debug output it uses "us-east-1". But only for the password check, for the most part, the creation of a new instance works fine, but when it comes to validating the "auto" generated password from AWS, I notice in the debug output that the URL is being changed. This is reproducible very easily. Basically I have to update the provider file each time I work in different AWS locations because the provider file is taken precedence and/or being ignored from my profiler settings where I declare the "location" value.

Setup

Launching a new instance with no "location" provider value set:
provider file:
aws-cloud:
driver: ec2
id: XXXXXXXXXXXXXXXXXXXXXXXXXXX
key: XXXXXXXXXXXXXXXXXXXXXXXXXX
keyname: brian_keypair
private_key: /opt/salt/brian_keypair.pem
ssh_interface: public_ips

minion:
master: servername.domain.com
win_installer: /srv/salt/win/files/Salt-Minion-2018.3.3-Py3-AMD64-Setup.exe
win_username: Administrator
win_password: auto
use_winrm: True
winrm_use_ssl: True
winrm_verify_ssl: False
userdata_file: /srv/salt/win/scripts/windows-firewall.ps1
userdata_template: jinja
win_deploy_auth_retries: 10
win_deploy_auth_retry_delay: 1

Launch server with this command:
salt-cloud -p ec2-east-test salt-test01

profile file:
ec2-east-test:
provider: aws-cloud
location: us-east-2
image: ami-0170ffc1abff2ebdc
size: t2.medium
network_interfaces:
- DeviceIndex: 0
AssociatePublicIpAddress: True
SubnetId: subnet-08d67c1cadc7f948e
SecurityGroupId:
- sg-0641ea4055e5de2f2
del_root_vol_on_destroy: True
del_all_vol_on_destroy: True
rename_on_destroy: True

debug log output:
[DEBUG ] Waiting for auto-generated Windows EC2 password
[DEBUG ] Using AWS endpoint: ec2.us-east-1.amazonaws.com
[DEBUG ] AWS Request: https://ec2.us-east-1.amazonaws.com/?Action=DescribeInstances&Filter.1.Name=tag%3AName&Filter.1.Value.1=salt-test01&Version=2014-10-01
[DEBUG ] Starting new HTTPS connection (1): ec2.us-east-1.amazonaws.com:443
[DEBUG ] https://ec2.us-east-1.amazonaws.com:443 "GET /?Action=DescribeInstances&Filter.1.Name=tag%3AName&Filter.1.Value.1=salt-test01&Version=2014-10-01 HTTP/1.1" 200 230
[DEBUG ] AWS Response Status Code: 200
[ERROR ] There was a profile error:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/salt/cloud/cli.py", line 281, in run
self.config.get('names')
File "/usr/lib/python2.7/site-packages/salt/cloud/init.py", line 1454, in run_profile
ret[name] = self.create(vm_)
File "/usr/lib/python2.7/site-packages/salt/cloud/init.py", line 1284, in create
output = self.cloudsfunc
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 2705, in create
vm_, data, ip_address, display_ssh_output
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 2333, in wait_for_instance
call='action',
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 4727, in get_password_data
instance_id = _get_node(name)['instanceId']
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3393, in _get_node
return next(iter(instance_info))
StopIteration

Everything goes through fine until it gets to the password validation, and you can see that the URL changes to something that is not "us-east-2" as I've defined it in my profile settings.

Steps to Reproduce Issue

See above.

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
Salt: 2018.3.3

Dependency Versions:
cffi: 1.11.5
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: 0.6.4
gitpython: 2.1.11
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: 2.19
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: centos 7.6.1810 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-862.3.2.el7.x86_64
system: Linux
version: CentOS Linux 7.6.1810 Core

Salt Minion is a Windows box, attempting to install latest Python 3 client: Salt-Minion-2018.3.3-Py3-AMD64-Setup.exe

[gitbrianb]

I should mention, as a work around to this, if I use the same "location" setting in the provider file as in the profile file, this error that I posted in the debug output as using "us-east-1", doesn't exist, and things work fine, but since I work between multiple zones, it's annoying. I would expect that if I don't specify a "default" location via the provider file that the profile file would take precedence. It does for the most part, but seems to get lost during the launch of new instances, particularly with the password validation piece with Windows minions, where I logged the error and I have to provide some "location" value in the provider file in addition to the profile file in order to successfully launch a server.

[dwoz]

@gitbrianb Thanks for reporting this, I think it has to do with key's being tied to a location. Can you make multiple provider's each with a different location to suite your needs?

[gitbrianb]

@dwoz Not sure if this answers your question or not, but I definitely have to use a different set of keys for each zone, but those settings are defined in the profile file for the instances individually and not configured in the provider file where I consider those values to be "global" parameter values that are defined across any profile I use when I indicate the value for the "provider" attribute in the profile file.

A similar test is below, where I have configured the provider file with the "us-east-2" value, I launch a "us-west-2" server and it the debug logs show it's using "us-east-2", which is coming from the provider file, but not the profile settings where I define the "location" value (see below).

provider file (with keys defined and location defined):
aws-cloud:
driver: ec2
id: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
key: XXXXXXXXXXXXXXXXXXXXXXXXXXX
keyname: brian_keypair
private_key: /opt/salt/brian_keypair.pem
location: us-east-2
ssh_interface: public_ips

minion:
master: salt.pi2solutions.com
win_installer: /srv/salt/win/files/Salt-Minion-2018.3.3-Py3-AMD64-Setup.exe
win_username: Administrator
win_password: auto
use_winrm: True
winrm_use_ssl: True
winrm_verify_ssl: False
userdata_file: /srv/salt/win/scripts/windows-firewall.ps1
userdata_template: jinja
win_deploy_auth_retries: 10
win_deploy_auth_retry_delay: 1

profile file (fields are defined for keys to be used in west region and location, not east location):
ec2-west-test:
provider: aws-cloud
location: us-west-2
image: ami-019e99815e07ceb49
size: t2.medium
network_interfaces:
- DeviceIndex: 0
AssociatePublicIpAddress: True
SubnetId: subnet-fe0ab0b7
SecurityGroupId:
- sg-0d0b42774c9b6b33b
keyname: PI2-BBEARD
private_key: /opt/salt/pi2-bbeard.pem
del_root_vol_on_destroy: True
del_all_vol_on_destroy: True
tag: {'Environment': 'salt-testing'}

Salt command with debug output:
salt-cloud -p ec2-west-test salt-west02 -l debug
[DEBUG ] Waiting for auto-generated Windows EC2 password
[DEBUG ] Using AWS endpoint: ec2.us-east-2.amazonaws.com
[DEBUG ] AWS Request: https://ec2.us-east-2.amazonaws.com/?Action=DescribeInstances&Filter.1.Name=tag%3AName&Filter.1.Value.1=salt-west02&Version=2014-10-01
[DEBUG ] Starting new HTTPS connection (1): ec2.us-east-2.amazonaws.com:443
[DEBUG ] https://ec2.us-east-2.amazonaws.com:443 "GET /?Action=DescribeInstances&Filter.1.Name=tag%3AName&Filter.1.Value.1=salt-west02&Version=2014-10-01 HTTP/1.1" 200 None
[DEBUG ] AWS Response Status Code: 200
[ERROR ] There was a profile error:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/salt/cloud/cli.py", line 281, in run
self.config.get('names')
File "/usr/lib/python2.7/site-packages/salt/cloud/init.py", line 1454, in run_profile
ret[name] = self.create(vm_)
File "/usr/lib/python2.7/site-packages/salt/cloud/init.py", line 1284, in create
output = self.cloudsfunc
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 2705, in create
vm_, data, ip_address, display_ssh_output
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 2333, in wait_for_instance
call='action',
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 4727, in get_password_data
instance_id = _get_node(name)['instanceId']
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3393, in _get_node
return next(iter(instance_info))
StopIteration

If I don't define the location or key parameters in the provider file, but do have them in the profile file, this happens:
[DEBUG ] Waiting for auto-generated Windows EC2 password
[DEBUG ] Using AWS endpoint: ec2.us-east-1.amazonaws.com
[DEBUG ] AWS Request: https://ec2.us-east-1.amazonaws.com/?Action=DescribeInstances&Filter.1.Name=tag%3AName&Filter.1.Value.1=salt-west03&Version=2014-10-01
[DEBUG ] Starting new HTTPS connection (1): ec2.us-east-1.amazonaws.com:443
[DEBUG ] https://ec2.us-east-1.amazonaws.com:443 "GET /?Action=DescribeInstances&Filter.1.Name=tag%3AName&Filter.1.Value.1=salt-west03&Version=2014-10-01 HTTP/1.1" 200 230
[DEBUG ] AWS Response Status Code: 200
[ERROR ] There was a profile error:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/salt/cloud/cli.py", line 281, in run
self.config.get('names')
File "/usr/lib/python2.7/site-packages/salt/cloud/init.py", line 1454, in run_profile
ret[name] = self.create(vm_)
File "/usr/lib/python2.7/site-packages/salt/cloud/init.py", line 1284, in create
output = self.cloudsfunc
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 2705, in create
vm_, data, ip_address, display_ssh_output
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 2333, in wait_for_instance
call='action',
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 4727, in get_password_data
instance_id = _get_node(name)['instanceId']
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3393, in _get_node
return next(iter(instance_info))
StopIteration

The URL points to "us-east-1", something I don't have defined anywhere, so it would seem some logic is present to pick up "defaults" on my account, but ignoring my settings defined in the profile and not taking order of precedence for provider being a "global" setting and profile taking more specific settings. Hopefully I'm saying that right?

[gitbrianb]

I should mention that if I update the "location" field in the provider file, all things being the way I've posted earlier, provider file for instance has my "east" keys configured, but my profile settings have my "west" keys, all is fine. So if I match the "location" field in the provider file each time for my profiles I'm launching I can work around this, but I would expect these fields to be defined on the profile settings, which for the most part they are, just not the validation of the password piece.

Ideally I would like to leave the provider file as generic as possible just having the API keys for AWS and the individual private key specs and location to be configured respectively for each of my instance settings within the profile file.

[dwoz]

@gitbrianb Yes, that makes sense.

@techhat Do you have any thoughts on this?