Failed to configure SSL certificates

[azinsharaf]

Hi @pbobov

I am getting the following error when running ha-primary.json file. Please advise.

arcgis_enterprise_server[Configure HTTPS] action configure_https[2019-05-08T13:28:12-07:00] ERROR: Failed to configure SSL certificates in ArcGIS Server. Admin URL 'https://egist.santaclaraca.gov:6443/arcgis/admin' is not reachable from 'VSRVGIST1.CITY.SANTACLARA.LOCAL'. For more information, see the ArcGIS Server help topic "About adding a GIS server to a site". You can access this topic in the table of contents by navigating to Administering ArcGIS Server > Common administrative tasks > Adding a GIS Server to a site.
[2019-05-08T13:28:43-07:00] ERROR: Failed to configure SSL certificates in ArcGIS Server. Importing CA certificate failed.

Compiled Resource:
------------------
# Declared in c:/chef/local-mode-cache/cache/cookbooks/arcgis-enterprise/recipes/server.rb:104:infrom_file'`

arcgis_enterprise_server("Configure HTTPS") do
  action [:configure_https]
  default_guard_interpreter :default
  declared_type :arcgis_enterprise_server
  cookbook_name "arcgis-enterprise"
  recipe_name "server"
  retries 5
  retry_delay 30
  **server_url "https://VSRVGIST1.City.SantaClara.local:6443/arcgis"**
  username "REDACTED"
  password "REDACTED"
  **server_admin_url "https://egist.santaclaraca.gov:6443/arcgis/admin"**
  keystore_file "\\\\ssl\\egist.pfx"
  keystore_password "REDACTED"
  cert_alias "egist.santaclaraca.gov"
  not_if { #code block }
end

This is json file:

{
  "arcgis":{
    "run_as_user":"CITY\\~svcREDACTED",
    "run_as_password":"REDACTED",
    "iis":{
      "keystore_file":"\\\\ssl\\egist.pfx",
      "keystore_password":"REDACTED"
    },
    
    "server":{
      **"domain_name":"egist.santaclaraca.gov",**
      "admin_username":"gisadmint",
      "admin_password":"REDACTED",
      "directories_root":"\\\\arcgistfs\\arcgisserver",
      "setup_archive":"\\\\ArcGIS_Enterprise\\ArcGIS_Server_Windows_1061_163968.exe",
      "authorization_file":"\\\\server\\ArcGISGISServerAdvanced_ArcGISServer_755122.prvc",
      **"keystore_file":"\\\\licenses_ssl\\ssl\\egist.pfx",**
      "keystore_password":"REDACTED"
    },
    "portal":{
      "domain_name":"egist.santaclaraca.gov",
      "admin_username":"portaladmint",
      "admin_password":"REDACTED",
      "admin_email":"REDACTED",
      "security_question":"Your favorite ice cream flavor?",
      "security_question_answer":"vanilla",
      "content_dir":"\\\\arcgistfs\\arcgisportal\\content",
      "setup_archive":"\\\\ArcGIS_Enterprise\\Portal_for_ArcGIS_Windows_1061_163979.exe",
      "authorization_file":"\\\\portal\\PortalforArcGIS_752980.prvc",
      "keystore_file":"\\\\ssl\\egist.pfx",
      "keystore_password":"REDACTED"
    }
  },
  "run_list":[
    "recipe[arcgis-enterprise::system]",
    "recipe[esri-iis]",
    "recipe[arcgis-enterprise::server]",
    "recipe[arcgis-enterprise::server_wa]",
    "recipe[arcgis-enterprise::datastore]",
    "recipe[arcgis-enterprise::portal]",
    "recipe[arcgis-enterprise::portal_wa]"
  ]
}

[cameronkroeker]

@azinsharaf
The 'Configure HTTPS' uses the following attributes:

arcgis_enterprise_server 'Configure HTTPS' do
server_url node['arcgis']['server']['url']
server_admin_url node['arcgis']['server']['private_url'] + '/admin'
username node['arcgis']['server']['admin_username']
password node['arcgis']['server']['admin_password']
keystore_file node['arcgis']['server']['keystore_file']
keystore_password node['arcgis']['server']['keystore_password']
cert_alias node['arcgis']['server']['cert_alias']
retries 5
retry_delay 30
not_if { node['arcgis']['server']['keystore_file'].empty? }
action :configure_https
end

In your ha-primary.json file may need to specify the following two attributes:

node['arcgis']['server']['url'] = ArcGIS Server URL. The default URL is https://:6443/arcgis using the fully qualified domain name of the machine on which Chef is running. If ArcGIS Server and the web adaptor are running on different machines, then this value should be explicitly set.

Since this wasn't specified chef chose the default to be server_url "https://VSRVGIST1.City.SantaClara.local:6443/arcgis"

node['arcgis']['server']['private_url'] = ArcGIS Server URL without Web Adaptor. Default URL is https://:6443/arcgis.

Since this wasn't specified chef chose: server_admin_url "https://egist.santaclaraca.gov:6443/arcgis/admin"

[azinsharaf]

I added the following parameters but still get the Failed to configure SSL certificates in ArcGIS Server. error.

"server_admin_url":"https://VSRVGIST1.City.SantaClara.local:6443/arcgis/admin",
"private_url":"https://VSRVGIST1.City.SantaClara.local:6443/arcgis"

Since we configure the third party NLB at the end, we have not set the CNAME record on the primary server. I am wondering we need to add dns name to the system32/drivers/etc/hosts file temporarily before running chef.

[cameronkroeker]

@azinsharaf

The certificate that is being imported into AGS needs to match the hostname that is going to be used for the server_admin_url. Sounds like the certificate is issued to egist.santaclaraca.gov, so then yes in this case the DNS for the AGS machine will need to be able to reach egist.santaclaraca.gov. An entry can be placed into the /etc/hosts file until the CNAME is created. I believe there is a recipe within the arcgis-enterprise cookbook that can be used to add the entry into the etc/host file:

https://github.com/Esri/arcgis-cookbook/tree/master/cookbooks/arcgis-enterprise#arcgis-enterprisehosts

{
  "arcgis": {
    "hosts": {
      "egist.santaclaraca.gov": "12.34.56.78"
    }
  },
  "run_list": [
    "recipe[arcgis-enterprise::hosts]"
  ]
}

https://github.com/Esri/arcgis-cookbook/wiki/arcgis%E2%80%90enterprise-Cookbook-Recipes#hosts

[azinsharaf]

@cameronkroeker

I included the host recipe and it maps the IP address to domain name but still returns SSL certificate error. This is the log in debug mode.

DEBUG: Request: POST https://VSRVGIST1.City.SantaClara.local:6443/arcgis/admin/machines/VSRVGIST1.CITY.SANTACLARA.LOCAL/sslcertificates/egist.santaclaraca.gov
DEBUG: token=C9RP6BNc3k0HlR44I26a8BBrjuWmakvOgnydYmZDtDtAmoHjEbuPfk6975u0qHWq&f=json
DEBUG: Response: 200 {"aliasName":"egist.santaclaraca.gov","issuer":"CN=DigiCert SHA2 Secure Server CA, O=DigiCert Inc, C=US","subject":"CN=*.santaclaraca.gov, O=City of Santa Clara, L=Santa Clara, ST=California, C=US","subjectAlternativeNames":["DNSName: *.santaclaraca.gov","DNSName: egist.santaclaraca.gov","DNSName: santaclaraca.gov"],"validFrom":"Sun Apr 14 17:00:00 PDT 2019","validUntil":"Tue Aug 25 05:00:00 PDT 2020","keyAlgorithm":"RSA","keySize":4096,"serialNumber":"c05d7b03408538034b19baea310877d","version":3,"signatureAlgorithm":"SHA256withRSA","keyUsage":["Digital_Signature","Key_Encipherment"],"md5Fingerprint":"b3535383e550fd39294f8e345e762f7e","sha1Fingerprint":"dacd2f19f1874351e3ef1ac492aafdfbb991d1d1","sha256Fingerprint":"83e4528b58bb7d2557449827707135182f412dae5fdc24596a84e0d9bc5b89e0"}
DEBUG: Request: POST https://VSRVGIST1.City.SantaClara.local:6443/arcgis/admin/generateToken
DEBUG: username=gisadmint&password=REDACTED&client=referer&referer=referer&expiration=600&f=json
DEBUG: Response: 200 {"token":"C9RP6BNc3k0HlR44I26a8BvnTqZAIuAmcFYyWCfwAkBe9VZ2SDSfpZgnxuQdiaCv","expires":"1557461954904"}
DEBUG: Request: POST https://VSRVGIST1.City.SantaClara.local:6443/arcgis/admin/machines/VSRVGIST1.CITY.SANTACLARA.LOCAL/sslcertificates/importExistingServerCertificate
DEBUG: Response: 200 {"status":"error","messages":["Importing CA certificate failed. "],"code":500}
ERROR: Failed to configure SSL certificates in ArcGIS Server. Importing CA certificate failed.

[cameronkroeker]

@azinsharaf

I believe this is because the request being made POST https://VSRVGIST1.City.SantaClara.local:6443/arcgis/admin/generateToken does not match the hostname the certificate is issued to. Looks like when AGS installed it used VSRVGIST1.City.SantaClara.local as its Admin URL instead of egist.santaclaraca.gov.

If you sign into https://VSRVGIST1.City.SantaClara.local:6443/arcgis/admin/machines/ what is listed as its Admin URL? I suspect its still https://VSRVGIST1.City.SantaClara.local:6443/arcgis/admin.

This gets set during AGS Site creation. Now that you have the hosts file set, AGS site may need to be recreated to pick up the change.

[azinsharaf]

@cameronkroeker

Thank you for your help. We could fix it by following these changes:
1- Enabling all TLS versions (We would disable 1.0 and 1.1 after finishing deployment). We had disabled 1.0 and 1.1.
2- We added NLB IP address to the list of IP addresses of server in Advanced TCP/IP settings
3- We included hosts recipe in the run list and used IP address of NLB
4- Deleting site & CA certificate from arcgis server admin page manually so chef re-creates it again
4- Run the script

No SSL certificate error.

Now following URLs work fine with valid certificate but the web-adaptor URLs still don't work for some reason. Web adapter URLs are showing server name instead of domain name. I create a new post for web adaptor since SSL issue has been resolved here.

https://egist.santaclaraca.gov:6443/arcgis/rest/services
https://egist.santaclaraca.gov:7443/arcgis/portaladmin
https://egist.santaclaraca.gov:6443/arcgis/manager/

[azinsharaf]

@cameronkroeker and @pbobov

Now on the second machine (GIST2) I am getting this error. Any idea?

Failed to join the site 'http://VSRVGIST1.City.SantaClara.local:6080/arcgis/admin'. Server machine 'http://VSRVGIST1.City.SantaClara.local:6080/arcgis/admin/machines/register' returned an error. 'Failed to register the server machine 'VSRVGIST2.CITY.SANTACLARA.LOCAL'. Machine 'VSRVGIST2.CITY.SANTACLARA.LOCAL' cannot access 'AGSDataStore_ds_bkyk1t9z' data store(s) registered with the site. Please ensure that the ArcGIS Server account has read and write access to the data store(s).'"],"code":500}

I am using same Win service account that has read/write access to config-store folder as a HA shared folder.

These are output of chef:

Resource Declaration:
    ---------------------
    # In c:/chef/local-mode-cache/cache/cookbooks/arcgis-enterprise/recipes/server_node.rb

     64: arcgis_enterprise_server 'Join ArcGIS Server Site' do
     65:   server_url node['arcgis']['server']['url']
     66:   install_dir node['arcgis']['server']['install_dir']
     67:   use_join_site_tool node['arcgis']['server']['use_join_site_tool']
     68:   if node['arcgis']['server']['use_join_site_tool']
     69:     config_store_connection_string node['arcgis']['server']['config_store_connection_string']
     70:     config_store_connection_secret node['arcgis']['server']['config_store_connection_secret']
     71:     config_store_type node['arcgis']['server']['config_store_type']
     72:   else
     73:     username node['arcgis']['server']['admin_username']
     74:     password node['arcgis']['server']['admin_password']
     75:     primary_server_url node['arcgis']['server']['primary_server_url']
     76:   end
     77:   retries 10

    Compiled Resource:
    ------------------
    # Declared in c:/chef/local-mode-cache/cache/cookbooks/arcgis-enterprise/recipes/server_node.rb:64:in `from_file'

    arcgis_enterprise_server("Join ArcGIS Server Site") do
      action [:join_site]
      default_guard_interpreter :default
      declared_type :arcgis_enterprise_server
      cookbook_name "arcgis-enterprise"
      recipe_name "server_node"
      install_dir "C:\\Program Files\\ArcGIS\\Server"
      retries 10
      retry_delay 30
      server_url "https://VSRVGIST2.City.SantaClara.local:6443/arcgis"
      use_join_site_tool false
      username "gisadmint"
      password "REDACTED"
      primary_server_url "http://VSRVGIST1.City.SantaClara.local:6080/arcgis"
    end

[azinsharaf]

Update:

I worked with Trevor from Esri support and he assisted me to finalize the scripts. We are implementing a HA ArcGIS Enterprise 10.6.1 in Win environment with a third party network load balancer.

Note 1)
The important part about ssl certificate is that arcgis machines don't need to have CA certificate (pfx file). The primary and secondary json files both indicate that we need pxf file but that is not correct, at least in a HA deployment with third party NLB. We just used pfx file in iis recipe and will import pfx file in third party NLB. We ran the following recipes:

 ]
"run_list":[
    "recipe[arcgis-enterprise::system]",
    "recipe[esri-iis]",
    "recipe[arcgis-enterprise::server]",
    "recipe[arcgis-enterprise::server_wa]",
    "recipe[arcgis-enterprise::datastore]",
    "recipe[arcgis-enterprise::portal]",
    "recipe[arcgis-enterprise::portal_wa]",
    "recipe[arcgis-enterprise::hosts]"
  ]

and secondary:

"run_list":[
   "recipe[arcgis-enterprise::system]",
   "recipe[esri-iis]",
   "recipe[arcgis-enterprise::server_node]",
   "recipe[arcgis-enterprise::server_wa]",
   "recipe[arcgis-enterprise::datastore_standby]",
   "recipe[arcgis-enterprise::portal_standby]",
   "recipe[arcgis-enterprise::portal_wa]",
   "recipe[arcgis-enterprise::hosts]",
   "recipe[arcgis-enterprise::federation]"

Note 2)
We added [arcgis].[hosts] value in json.
"FINAL_NLB_URL":"IP of primary machine"

Note 3)
Portal and Server on the both machines work fine now with a valid CA certificate. The next step which we are doing now is to configure NLB. We will import pfx file on nlb. I update the post after final configuration. We still need to validate the final result.

Note 4)
Another thing that created lots of confusion was the values of [arcgis].[server].[domain_name] and [arcgis].[portal].[domain_name] parameters. They should be FQDN of machines not the final site address which we use in NLB.

Note 5)
Another important thing was the [arcgis].[web_adaptor].[admin_access] value. The default value in cookbook is false which doesn't allow the federation recipe work correctly. We added "admin_access":true in both json files.

Hope these are helpful.

I want to take this opportunity to thank @pbobov , @cameronkroeker and Trevor for the valuable effort to provide and support ArcGIS Cookbook. I was able to deploy a HA ArcGIS Enterprise in 6 hours which is amazing.

Thanks,
Azin