Generate config.json, creds.json & D7-dev-{sitename}-settings.inc

[ericduran]

So once we start creating memcache instances & database instances we'll need a way to provide these credentials to the specific site without people modifying their settings.php.

Acquia already does this by just generating these files, and we all have this snippet to load them in:

if (file_exists('/var/www/site-php')) {
  require '/var/www/site-php/' . $_ENV["AH_SITE_GROUP"] . '/' . $_ENV["AH_SITE_GROUP"] . '-settings.inc';
}

Luckily for us we're already passing in "$_ENV["AH_SITE_GROUP"]" & $_ENV["AH_SITE_GROUP"]

So we can actually also populate peoples database & memcache settings without them having to do anything.

I think it's pretty cool. Thoughts?

[scottrigby]

@ericduran I <3 it, and as we chatted about today I'm gonna make a PR to auto-generate the settings.php file I showed u if one doesn't already exist on for each defined site (since as you noted we're now creating the DBs!). This should be in there too

[ericduran]

@scottrigby well we should generate the PHP file in the same place as acquia, so I think maybe we use "site-php" as our pubstack home directory?

This way no1 has to change their current settings.php?

[ericduran]

// /var/www/site-php/nbcuqa2dev/D7-dev-nbcuqa2-settings.inc
<?php
// Store the prefix defined by customer or an empty string
$customer_prefix = isset($databases['default']['default']['prefix']) ? $databases['default']['default']['prefix'] : '';

// All databases for this site.
$databases = array (
  "nbcuqa2" => 
  array (
    "default" => 
    array (
      "driver" => "mysql",
      "database" => "''",
      "username" => "''",
      "password" => "''",
      "host" => "",
      "port" => 3306,
      "prefix" => $customer_prefix,
    ),
  ),
  "default" => 
  array (
    "default" => 
    array (
      "driver" => "mysql",
      "database" => "",
      "username" => "",
      "password" => "",
      "host" => "",
      "port" => 3306,
      "prefix" => $customer_prefix,
    ),
  ),
);

// Set the drupal hash salt.  This must come before the first use of
// drupal_valid_test_ua().  Note that this is included for D6 because there is
// a core patch available which utilizes it.
$drupal_hash_salt = hash('sha256', file_get_contents('/var/www/site-php/nbcuqa2dev/.secret') . 'drupal_hash_salt');

// Set the cluster shared secret file for simpletests.  While this is not
// currently used in the D6 simpletest module, but may be necessary in the
// future to support simpletests on D6.
$drupal_cluster_shared_secret_file = '/var/www/site-php/nbcuqa2dev/.secret';

// HA database info for 'default'.
$conf['acquia_hosting_site_info'] = array (
  "name" => "nbcuqa2dev",
  "db" => 
  array (
    "id" => "9147",
    "role" => "nbcuqa2",
    "name" => "nbcuqa2dev",
    "user" => "nbcuqa2dev",
    "pass" => "",
    "db_url_ha" => 
    array (
      "staging-4068" => "mysqli://@staging-4068:3306/nbcuqa2dev",
    ),
    "db_cluster_id" => "785",
    "port" => 3306,
  ),
);

// Check if this is a Simpletest run in a way that is compatible with Pressflow
// 6 and Drupal 7.
$test_ua = FALSE;
if (function_exists('drupal_valid_test_ua')) {
  if (isset($_SERVER['HTTP_USER_AGENT']) && preg_match("/^(simpletest\d+);/", $_SERVER['HTTP_USER_AGENT'], $matches)) {
    $test_ua = drupal_valid_test_ua($_SERVER['HTTP_USER_AGENT']);
  }
}
// Move Drupal temp dir off root partition unless we are running simpletests to
// allow upload tests to pass.
if (!$test_ua) {
  $conf['file_directory_temp'] = "/mnt/tmp/nbcuqa2dev";
  $conf['file_temporary_path'] = "/mnt/tmp/nbcuqa2dev";
}

// Configure Devel XHProf variables
if (empty($_SERVER['SERVER_NAME'])) {
$conf['devel_xhprof_url'] = 'http://localhost/xhprof_html';
} else {
$conf['devel_xhprof_url'] = 'http://' . $_SERVER['SERVER_NAME'] . '/xhprof_html';
}
$conf['devel_xhprof_directory'] = "/usr/local/share/php5-xhprof";

// Set the program name for syslog.module.
$conf['syslog_identity'] = 'nbcuqa2dev';

// Default settings for the Memcache module
$conf['memcache_key_prefix'] = $conf['acquia_hosting_site_info']['db']['name'] . '_';
$conf['memcache_servers'] = array(
   'staging.prod.hosting.acquia.com:11211' => 'default',
);

// Default settings for the Cache Router module
$conf['cacherouter'] = array(
      'default' => array(
        'engine' => 'memcache',  // options are: apc, db, file, memcache and xcache
        'shared' => TRUE,
        'prefix' => $conf['acquia_hosting_site_info']['db']['name'] . '_',
        'servers' => array(
           'staging.prod.hosting.acquia.com:11211',
        ),
   ),
);

// Reverse proxy ip addresses
$conf['reverse_proxies'] = array(
  '10.80.x.x',
  '10.86.x.x',
);


// Set up the Acquia Network connection.
$conf['ah_network_key'] = 'AH_NETWORK_KEY';
$conf['ah_network_identifier'] = 'AH_NETWORK_ID';

// Prevent installation of modules via UI.
// Allow authorized operations in the case of simpletests to allow
// SystemAuthorizeCase to pass.
if (!drupal_valid_test_ua()) {
  $conf['allow_authorize_operations'] = FALSE;
}

// Use AWS internal IP addresses for Acquia Search to reduce latency.
$conf['acquia_search_host'] = 'search.acquia.com';

// Do not put a <?php tag in this file, this file is appended to another php file.
/**
 * General settings for Acquia hosting.
 */

@include_once "/usr/ah/xhprof/acquia_xhprof.php";
if (function_exists('acquia_hosting_xhprof_init')) {
  acquia_hosting_xhprof_init();
}

// Tell Drupal whether the client arrived via HTTPS. Try to ensure the
// request is coming from our load balancers by checking the IP address.
if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https' && isset($_SERVER['REMOTE_ADDR']) && in_array($_SERVER['REMOTE_ADDR'], $conf['reverse_proxies'])) {
  $_SERVER['HTTPS'] = 'on';
}

$x_ips = isset($_SERVER['HTTP_X_FORWARDED_FOR']) ? explode(',', $_SERVER['HTTP_X_FORWARDED_FOR']) : array();
$x_ips = array_map('trim', $x_ips);
// Add REMOTE_ADDR to the X-Forwarded-For list (the ip_address function will
//  also do this) in case it's a 10. internal AWS address; if it is we should
//  add it to the list of reverse proxy addresses so that ip_address will ignore
//  it.
$x_ips[] = $_SERVER['REMOTE_ADDR'];
// Check firstly for the bal and then check for an internal IP immediate
//  following. This is likely the ELB. Testing for extra IPs is the
//  responsibility of the customer as they will need to have a list of trusted
//  IPs.
$conf['reverse_proxy_addresses'] = array();
if ($ip = array_pop($x_ips)) {
  if (in_array($ip, $conf['reverse_proxies'])) {
    if (!in_array($ip, $conf['reverse_proxy_addresses'])) {
      $conf['reverse_proxy_addresses'][] = $ip;
    }

    // We have a reverse proxy so turn the setting on.
    $conf['reverse_proxy'] = TRUE;

    // Get the next IP to test if it is internal. We will assume this is the ELB
    //  if so.
    $ip = array_pop($x_ips);
    if (strpos($ip, '10.') === 0) {
      if (!in_array($ip, $conf['reverse_proxy_addresses'])) {
        $conf['reverse_proxy_addresses'][] = $ip;
      }
    }
  }
}

// Simpletest depends on changing the database prefix in BOOTSTRAP_DATABASE,
// so do not autoconnect if this is a simpletest request.
if (drupal_valid_test_ua()) {
  $conf['acquia_hosting_settings_autoconnect'] = FALSE;
}

// Set the default $databases if available.
if ((!isset($conf['acquia_hosting_settings_autoconnect']) || $conf['acquia_hosting_settings_autoconnect'] == TRUE) && !empty($conf['acquia_hosting_site_info']['db'])) {
  acquia_hosting_db_choose_active($conf['acquia_hosting_site_info']['db']);
}

// Log the Drupal page request at the end of execution.
register_shutdown_function('acquia_hosting_drupal_log', time());

/**
 * Log a Drupal page request.
 */
function acquia_hosting_drupal_log($start_time = 0) {
  $now = time();
  $path = isset($_ENV['ACQUIA_HOSTING_DRUPAL_LOG']) ? $_ENV['ACQUIA_HOSTING_DRUPAL_LOG'] : '/var/log/drupal.log';
  $query = isset($_SERVER['QUERY_STRING']) ? $_SERVER['QUERY_STRING'] : '';
  $date = gmdate('d/M/Y:H:i:s O');

  global $user;
  $uid = isset($user->uid) ? $user->uid : 'na';

  $php_pid = posix_getpid();
  $php_time = $now - $start_time;

  $queue_wait = -1;
  $request_arrived = getenv('AH_REQUEST_ARRIVED_YYYYMMDDHHMMSS');
  if ($request_arrived && preg_match('@^(\d\d\d\d)(\d\d)(\d\d)(\d\d)(\d\d)(\d\d)$@', $request_arrived, $m)) {
    $queue_time = gmmktime($m[4], $m[5], $m[6], $m[2], $m[3], $m[1]);
    // $queue_time is in seconds, so $queue_wait has to be.
    $queue_wait = floor($start_time) - $queue_time;
  }

  $log = sprintf("[%s] %s %s %s query=%s uid=%s php_pid=%d php_time=%.3f queue_wait=%d\n", $date, $_SERVER['HTTP_HOST'], $_SERVER['REQUEST_METHOD'], $_SERVER['REQUEST_URI'], $query, $uid, $php_pid, $php_time, $queue_wait);
  @file_put_contents($path, $log, FILE_APPEND);
}

/**
 * Check if we are running PHP from the command line.
 *
 * Note that this mimics the drupal_is_cli() function in Drupal 7.
 */
function acquia_hosting_is_cli() {
  return !isset($_SERVER['SERVER_SOFTWARE']) && (php_sapi_name() == 'cli' || (is_numeric($_SERVER['argc']) && $_SERVER['argc'] > 0));
}

/**
 * Choose a database server to connect to during the current page request.
 *
 * This function will loop through a list of provided databases and attempt to
 * connect to them in order, stopping when a successful connection is made.
 * However, it also checks a file cache to see if a different database
 * server is preferred and moves that server to the front of the list so that
 * it can be attempted before any of the others. If the database server that
 * is eventually connected to is different from the one stored in the cache,
 * it writes the new, preferred sever to the cache before exiting.
 *
 * @param $db_info
 *   Info including an ordered array of possible database servers to connect to.
 * @param $name
 *   The name of the database that is being connected to. Usually this will be
 *   'default', but alternate values can be specified if a separate, external
 *   set of database servers is being used for part of the page request (in
 *   addition to the main Drupal database).
 * @return
 *   The new database connection.
 */
function acquia_hosting_db_choose_active($db_info = NULL, $name = 'default') {
  // For Drupal 7 we assume we will always connect to the DB if
  // the request reaches Drupal instead of Varnish.
  require_once DRUPAL_ROOT . '/includes/database/database.inc';

  global $databases, $conf;

  if (!isset($db_info)) {
    $db_info = $conf['acquia_hosting_site_info']['db'];
  }

  $db_servers = $db_info['db_url_ha']; // just need server names as keys
  // $databases[$name]['default'] can have properties we don't care about
  // that customers might need to control, e.g. prefix. Preserve them.
  $databases[$name]['default'] = array_merge(
    isset($databases[$name]['default']) ? $databases[$name]['default'] : array(),
    array(
      'driver' => 'mysql',
      'database' => $db_info['name'],
      'username' => $db_info['user'],
      'password' => $db_info['pass'],
      'host' => '',
      'port' => $db_info['port'],
    )
  );

  /**
   * Load the db control cache and check if it contains the name of a preferred
   * database server. If so, move that server to the front of the list.
   * 
   * Use DNS to look up our database server.
   *
   * Use pdnsd on our local server to do the lookup; if pdnsd is down use the
   * DNS server directly ("dns-master", defined in /etc/hosts); if dns-master
   * goes down pdnsd will use its last cached result
   */

  require_once("/usr/share/php/Net/DNS2_wrapper.php");
  try {
    $resolver = new Net_DNS2_Resolver(array('nameservers' => array('127.0.0.1', 'dns-master')));
    $response = $resolver->query("cluster-{$db_info['db_cluster_id']}.mysql", 'CNAME');
    $cached_id = $response->answer[0]->cname;
  }
  catch (Net_DNS2_Exception $e) {
    $cached_id = "";
  }

  if (!empty($cached_id)) {
    if (isset($db_servers[$cached_id])) {
      $db_servers[$cached_id] = -1;
      asort($db_servers, SORT_NUMERIC);
    }
    else {
      syslog(LOG_WARNING, "AH_CRITICAL: unrecognized server name in $filename: $cached_id");
    }
  }

  // Loop through each database server and try to connect to it. Stop once a
  // successful connection is made.
  $connected_id = '';
  $checked_for_lock_record = FALSE;
  foreach ($db_servers as $server => $value) {
    // If we have a cached server, and this isn't it, we might be locked to that
    // cached server.  Check to see if we are locked to the cached server, and give
    // up all hope of finding a suitable database if that's the case.
    if (!empty($cached_id) && !$checked_for_lock_record && $server != $cached_id) {
      try {
        $resolver->query("lock.cluster-{$db_info['db_cluster_id']}.mysql", 'TXT');
        break;
      } catch (Net_DNS2_Exception $e) {
        // We are not locked -- don't do this again.
        $checked_for_lock_record = TRUE;
      }
    }

    $databases[$name]['default']['host'] = $server;
    Database::parseConnectionInfo();
    // Try to connect to each server three times, with an increasing delay
    // between each attempt.
    $attempt = 1;
    $max_attempts = 3;
    while ($attempt <= $max_attempts) {
      try {
        Database::getConnection('default');
        // Quit the entire loop as soon as a successful connection is made.
        $connected_id = $server;
        break 2;
      }      
      catch (Exception $e) {
        if ( preg_match('/\[1049\][\s]+Unknown database/',$e->getMessage()) ) {  //SQLSTATE[42000] [1049] Unknown database 's6'
          $connected_id = NULL;
          break 2;
        }
        if ($attempt < $max_attempts) {
          // Wait 0.5 seconds before the second attempt, and 1 second before
          // the third attempt.
          usleep($attempt * 500000);
        }
      }
      $attempt += 1;
    }
    // If we failed to connect to this server, log a warning message before
    // moving on to the next one.
    syslog(LOG_WARNING, "AH_WARNING: The connection to database {$db_info['name']} on $server failed after $max_attempts attempts and is now failing over. Error was: " . $e->getMessage());
  }

  // If no connection was made, show an error page and exit. For command line
  // scripts, however, throw an exception instead so that the calling code can
  // catch it.
  if (empty($connected_id)) {
    $message = "Failed to connect to any database servers for database {$db_info['name']}.";
    syslog(LOG_ERR, $message);
    if (acquia_hosting_is_cli()) {
      throw new Exception($message);
    }
    else {
      acquia_hosting_db_fail_fast();
    }
  }

  // If the preferred database server changed, write this to the cache.
  if (empty($cached_id) || $cached_id != $connected_id ) {
    // Store the ID of a preferred database server in the cache.
    $update_cmd = "nohup sudo -u dnsuser /usr/local/sbin/ah-nsupdate cluster-{$db_info['db_cluster_id']} $connected_id > /dev/null &";
    exec(escapeshellcmd($update_cmd));
  }

  // Return the database info that is actually to be used.
  return $databases[$name]['default'];
}


/**
 * At the moment Drupal 7 can't readily display the miantenance
 * page if called from settings.php, so just emit this minimal
 * error page.
 *
 * @todo an image or some styling if we care.
 */
function acquia_hosting_db_fail_fast() {
header($_SERVER['SERVER_PROTOCOL'] .' 503 Service temporarily unavailable');

echo <<<EOF
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<title>Error</title>
</head>
<body>
The website encountered an unexpected error. Please try again later.
<!-- DB not found -->
</body>
</html>
EOF;
exit;
}

[ericduran]

One thing I've realized we should do in the generated settings.php is to use the servers IP address instead of no host or localhost for the database host.

We should do this so if someone runs drush from their host machine it will still work without specifying a remote a alias.

[scottrigby]

@ericduran Good idea... That'll be in my settings pr
On Jun 25, 2014 9:06 AM, "Eric J. Duran" notifications@github.com wrote:

One thing I've realized we should do in the generated settings.php is to
use the servers IP address instead of no host or localhost for the database
host.

We should do this so if someone runs drush from their host machine it will
still work without specifying a remote a alias.

—
Reply to this email directly or view it on GitHub
#36 (comment).

[scottrigby]

@ericduran So I put just the $databases array in #45 . We should take a look at each piece of the file above, and see what else we want or need to include (and add those as follow-up PRs).

[ericduran]

This is now all fixed, with #84