From e7b4ff001af86d1fb91015f98f6c700484b3025d Mon Sep 17 00:00:00 2001 From: Jacob Bednarz Date: Fri, 6 Jul 2018 11:16:22 +1000 Subject: [PATCH 1/2] Define `open_timeout` for `get_ec2_metadata_region` While developing some AMI provisioning tools locally, we attempt to install the CodeDeploy agent. Unfortunately, the process was hanging indefinitely. Digging into the process using `strace`, it is stuck on a `connect` to the EC2 metadata service. ``` connect(7, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("169.254.169.254")}, 16 ``` This is due to the metadata service not being present in our local development environment and the installer isn't able to resolve the service. After locating this in the installation script, I can see there is a (very high) read timeout however there isn't an open timeout which is what the process will use to restrict the amount of time a connection can be pending a connection. This is particularly useful if the service you are attempting to connect to is either unresponsive to accept the connection or not reachable due to things like DNS resolution. To address the issue, I've added an open timeout to the `open` call and now the process fails as expected after a sensible 3s. --- bin/install | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/install b/bin/install index ec10fbde..c990bb3d 100755 --- a/bin/install +++ b/bin/install @@ -199,7 +199,7 @@ EOF def get_ec2_metadata_region begin uri = URI.parse('http://169.254.169.254/latest/meta-data/placement/availability-zone') - az = uri.read(:read_timeout => 120) + az = uri.read(:open_timeout => 3, :read_timeout => 120) az.strip rescue @log.warn("Could not get region from EC2 metadata service at '#{uri.to_s}'") From d068ba176ea25c80d62092da814acab6c92b3feb Mon Sep 17 00:00:00 2001 From: Jacob Bednarz Date: Fri, 6 Jul 2018 12:48:54 +1000 Subject: [PATCH 2/2] Extract out HTTP reused options Consolidate the HTTP options for OpenURI `read` calls to use the same settings. This extends the work done in e7b4ff0 to include a `open_timeout` so that these calls are not subject to potentially the same issue with being unable to establish a connection remotely. --- bin/install | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/bin/install b/bin/install index c990bb3d..f8fb6ac5 100755 --- a/bin/install +++ b/bin/install @@ -89,6 +89,16 @@ EOF ['2.4', '2.3', '2.2', '2.1', '2.0'] end + def http_connection_options + { + :ssl_verify_mode => OpenSSL::SSL::VERIFY_PEER, + :redirect => true, + :open_timeout => 3, + :read_timeout => 120, + :proxy => @http_proxy + } + end + # check ruby version, only version 2.x works def check_ruby_version_and_symlink @log.info("Starting Ruby version check.") @@ -246,7 +256,7 @@ EOF retries ||= 0 exceptions = [OpenURI::HTTPError, OpenSSL::SSL::SSLError] begin - uri.open(:ssl_verify_mode => OpenSSL::SSL::VERIFY_PEER, :redirect => true, :read_timeout => 120, :proxy => @http_proxy) do |s3| + uri.open(http_connection_options) do |s3| package_file.write(s3.read) end rescue *exceptions => e @@ -270,7 +280,7 @@ end begin require 'json' - version_string = uri.read(:ssl_verify_mode => OpenSSL::SSL::VERIFY_PEER, :redirect => true, :read_timeout => 120, :proxy => @http_proxy) + version_string = uri.read(http_connection_options) JSON.parse(version_string) rescue OpenURI::HTTPError => e @log.error("Could not find version file to download at '#{uri.to_s}'")