Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Config autogeneration #42

Merged
merged 25 commits into from

2 participants

David Crane Joshua Timberman
David Crane

COOK-2390 - Recipes to auto-generate many postgresql.conf settings, following "initdb" and "pgtune"

I've written two recipes to auto-generate many postgresql.conf settings. I derived them from the source code of a couple important PostgreSQL utilities:

initdb – initializes the database cluster’s default locale and character set encoding. (See the original initdb source code.)

pgtune – takes the wimpy default postgresql.conf and expands the database
server to be as powerful as the hardware it's being deployed on. (See the original pgtune python script.)

David Crane davidc-donorschoose Careful about Debian minor version numbers 92b9ecc
David Crane davidc-donorschoose Restored helpful commentary to the postgresql.conf template, still us…
…ing the key in node['postgresql']['config'] attributes as before.
78e2dc1
David Crane davidc-donorschoose Optional postgresql::config recipe to install useful administration t…
…ools to support postgresql::server
3404084
David Crane davidc-donorschoose Corrected postgresql-contrib package name in debian family. 94ac0a8
David Crane davidc-donorschoose Document postgres "encrypted password" for chef-solo 7484863
David Crane davidc-donorschoose Removed obsolete code for installing server_redhat packages with expl…
…icit names.

I suspect this resulted from a merge accident. This code was originally removed
in the 2012-08-06 commit for "[COOK-1532] - specify packages with attributes",
but it reappeared in the 2012-11-05 commit for "Support for postgresql 9.0.x on
opensuse 11.4".

Clearly, the old code badly breaks the redhat family for postgresql-9.X, and
since it probably has no effect in opensuse, I just decided to remove it.
588eb74
David Crane davidc-donorschoose Run initdb for any platform version on first install d622edf
David Crane davidc-donorschoose Documented optional postgresql::contrib recipe. 3c7b43d
David Crane davidc-donorschoose Removed unnecessary attempt to include_recipe 'postgresql::ppa_pitti_…
…postgresql'
cfb6b3f
David Crane davidc-donorschoose PostgreSQL 9.0 and 9.1 changes to add/remove/modify parameter comment…
…ary.
f37a144
David Crane davidc-donorschoose PostgreSQL 9.2 changes to add/remove/modify parameter commentary. 465ab62
David Crane davidc-donorschoose Optional recipe to enable the PGDG yum repo for newer versions of Pos…
…tgreSQL.

This is the same approach used for the PostgreSQL backport PPA repository.
(Note yum_pgdg_postgresql.rb is to redhat as ppa_pitti_postgresql.rb is to debian.)
8a89bb3
David Crane davidc-donorschoose Validates attributes related to installing recent PostgreSQL versions
and logs chef WARN messages about probable mistakes and fixes.
I attempted to present the redhat family (yum_pgdg_postgresql.rb) and the
debian family (ppa_pitti_postgresql.rb) correctly, since they use different
naming conventions.
b933550
David Crane davidc-donorschoose Merge remote branch 'upstream/master' into track-upstream.
This is opscode release 2.1.0 that was tagged 2012-12-13.
4998969
David Crane davidc-donorschoose Another PostgreSQL 9.2 change to add a new parameter commentary. debbc3a
David Crane davidc-donorschoose The redhat family only needs "postgresql92" (not "postgresql92-devel"…
…) for client package list.
80f8382
David Crane davidc-donorschoose Enable PostgreSQL 9.0 installation from PGDG yum repository. fd2b779
David Crane davidc-donorschoose === REVERT to release v2.1.0 of opscode-cookbooks/postgresql.
(Reverted my COOK-2051 changes between Dec 12 and Jan 6)
5269fa1
David Crane davidc-donorschoose === MERGE from release v2.2.0 of opscode-cookbooks/postgresql.
(Merged my COOK-2230, COOK-2231 and COOK-2233 changes from Jan 15)
49c094d
David Crane davidc-donorschoose Merge branch 'track_upstream' 1968bc7
David Crane davidc-donorschoose Recipe config_initdb to take locale and timezone settings from system 2647b9b
David Crane davidc-donorschoose Recipe config_pgtune to tune performance for database workload & RAM 18d399f
David Crane davidc-donorschoose Documented how optional config_* recipes autogenerate postgresql.conf 19a261e
David Crane davidc-donorschoose Better example for config_pgtune JSON attributes 98f4a51
David Crane davidc-donorschoose Refactored workhorse methods into libraries/default.rb 160410c
Joshua Timberman jtimberman merged commit 8518e10 into from
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Dec 12, 2012
  1. David Crane
Commits on Dec 13, 2012
  1. David Crane

    Restored helpful commentary to the postgresql.conf template, still us…

    davidc-donorschoose authored
    …ing the key in node['postgresql']['config'] attributes as before.
  2. David Crane

    Optional postgresql::config recipe to install useful administration t…

    davidc-donorschoose authored
    …ools to support postgresql::server
  3. David Crane
Commits on Dec 14, 2012
  1. David Crane
  2. David Crane

    Removed obsolete code for installing server_redhat packages with expl…

    davidc-donorschoose authored
    …icit names.
    
    I suspect this resulted from a merge accident. This code was originally removed
    in the 2012-08-06 commit for "[COOK-1532] - specify packages with attributes",
    but it reappeared in the 2012-11-05 commit for "Support for postgresql 9.0.x on
    opensuse 11.4".
    
    Clearly, the old code badly breaks the redhat family for postgresql-9.X, and
    since it probably has no effect in opensuse, I just decided to remove it.
  3. David Crane
  4. David Crane
  5. David Crane
  6. David Crane
Commits on Dec 17, 2012
  1. David Crane
  2. David Crane

    Optional recipe to enable the PGDG yum repo for newer versions of Pos…

    davidc-donorschoose authored
    …tgreSQL.
    
    This is the same approach used for the PostgreSQL backport PPA repository.
    (Note yum_pgdg_postgresql.rb is to redhat as ppa_pitti_postgresql.rb is to debian.)
  3. David Crane

    Validates attributes related to installing recent PostgreSQL versions

    davidc-donorschoose authored
    and logs chef WARN messages about probable mistakes and fixes.
    I attempted to present the redhat family (yum_pgdg_postgresql.rb) and the
    debian family (ppa_pitti_postgresql.rb) correctly, since they use different
    naming conventions.
  4. David Crane

    Merge remote branch 'upstream/master' into track-upstream.

    davidc-donorschoose authored
    This is opscode release 2.1.0 that was tagged 2012-12-13.
  5. David Crane
Commits on Jan 6, 2013
  1. David Crane
  2. David Crane
Commits on Feb 2, 2013
  1. David Crane

    === REVERT to release v2.1.0 of opscode-cookbooks/postgresql.

    davidc-donorschoose authored
    (Reverted my COOK-2051 changes between Dec 12 and Jan 6)
  2. David Crane

    === MERGE from release v2.2.0 of opscode-cookbooks/postgresql.

    davidc-donorschoose authored
    (Merged my COOK-2230, COOK-2231 and COOK-2233 changes from Jan 15)
Commits on Feb 4, 2013
  1. David Crane
Commits on Feb 11, 2013
  1. David Crane
  2. David Crane
  3. David Crane
Commits on Feb 13, 2013
  1. David Crane
Commits on Feb 21, 2013
  1. David Crane
This page is out of date. Refresh to see the latest.
94 README.md
View
@@ -87,6 +87,9 @@ be dynamically rendered from the attributes. The helpful commentary
will no longer be present. You should consult the PostgreSQL
documentation for specific configuration details.
+See __Recipes__ `config_initdb` and `config_pgtune` below to
+auto-generate many postgresql.conf settings.
+
For values that are "on" or "off", they should be specified as literal
`true` or `false`. String values will be used with single quotes. Any
configuration option set to the literal `nil` will be skipped
@@ -180,6 +183,96 @@ database, and manages the postgresql service. You should include the
`postgresql::server` recipe, which will include this on RHEL/Fedora
platforms.
+config\_initdb
+--------------
+
+Takes locale and timezone settings from the system configuration.
+This recipe creates `node.default['postgresql']['config']` attributes
+that conform to the system's locale and timezone. In addition, this
+recipe creates the same error reporting and logging settings that
+`initdb` provided: a rotation of 7 days of log files named
+postgresql-Mon.log, etc.
+
+The default attributes created by this recipe are easy to override with
+normal attributes because of Chef attribute precedence. For example,
+suppose a DBA wanted to keep log files indefinitely, rolling over daily
+or when growing to 10MB. The Chef installation could include the
+`postgresql::config_initdb` recipe for the locale and timezone settings,
+but customize the logging settings with these node JSON attributes:
+
+ "postgresql": {
+ "config": {
+ "log_rotation_age": "1d",
+ "log_rotation_size": "10MB",
+ "log_filename": "postgresql-%Y-%m-%d_%H%M%S.log"
+ }
+ }
+
+Credits: This `postgresql::config_initdb` recipe is based on algorithms
+in the [source code](http://doxygen.postgresql.org/initdb_8c_source.html)
+for the PostgreSQL `initdb` utility.
+
+config\_pgtune
+--------------
+
+Performance tuning.
+Takes the wimpy default postgresql.conf and expands the database server
+to be as powerful as the hardware it's being deployed on. This recipe
+creates a baseline configuration of `node.default['postgresql']['config']`
+attributes in the right general range for a dedicated Postgresql system.
+Most installations won't need additional performance tuning.
+
+The only decision you need to make is to choose a `db_type` from the
+following database workloads. (See the recipe code comments for more
+detailed descriptions.)
+
+ * "dw" -- Data Warehouse
+ * "oltp" -- Online Transaction Processing
+ * "web" -- Web Application
+ * "mixed" -- Mixed DW and OLTP characteristics
+ * "desktop" -- Not a dedicated database
+
+This recipe uses a performance model with three input parameters.
+These node attributes are completely optional, but it is obviously
+important to choose the `db_type` correctly:
+
+ * `node['postgresql']['config_pgtune']['db_type']` --
+ Specifies database type from the list of five choices above.
+ If not specified, the default is "mixed".
+
+ * `node['postgresql']['config_pgtune']['max_connections']` --
+ Specifies maximum number of connections expected.
+ If not specified, it depends on database type:
+ "web":200, "oltp":300, "dw":20, "mixed":80, "desktop":5
+
+ * `node['postgresql']['config_pgtune']['total_memory']` --
+ Specifies total system memory in kB. (E.g., "49416564kB".)
+ If not specified, it will be taken from Ohai automatic attributes.
+ This could be used to tune a system that isn't a dedicated database.
+
+The default attributes created by this recipe are easy to override with
+normal attributes because of Chef attribute precedence. For example, if
+you are running application benchmarks to try different buffer cache
+sizes, you would experiment with this node JSON attribute:
+
+ "postgresql": {
+ "config": {
+ "shared_buffers": "3GB"
+ }
+ }
+
+Note that the recipe uses `max_connections` in its computations. If
+you want to override that setting, you should specify
+`node['postgresql']['config_pgtune']['max_connections']` instead of
+`node['postgresql']['config']['max_connections']`.
+
+Credits: This `postgresql::config_pgtune` recipe is based on the
+[pgtune python script](https://github.com/gregs1104/pgtune)
+developed by
+[Greg Smith](http://notemagnet.blogspot.com/2008/11/automating-initial-postgresqlconf.html)
+and
+[other pgsql-hackers](http://www.postgresql.org/message-id/491C6CDC.8090506@agliodbs.com).
+
contrib
-------
@@ -273,6 +366,7 @@ License and Author
- Author:: Joshua Timberman (<joshua@opscode.com>)
- Author:: Lamont Granquist (<lamont@opscode.com>)
- Author:: Chris Roberts (<chrisroberts.code@gmail.com>)
+- Author:: David Crane (<davidc@donorschoose.org>)
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
294 libraries/default.rb
View
@@ -0,0 +1,294 @@
+#
+# Cookbook Name:: postgresql
+# Library:: default
+# Author:: David Crane (<davidc@donorschoose.org>)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+module Opscode
+ module PostgresqlHelpers
+
+#######
+# Function to truncate value to 4 significant bits, render human readable.
+# Used in recipes/config_initdb.rb to set this attribute:
+#
+# The memory settings (shared_buffers, effective_cache_size, work_mem,
+# maintenance_work_mem and wal_buffers) will be rounded down to keep
+# the 4 most significant bits, so that SHOW will be likely to use a
+# larger divisor. The output is actually a human readable string that
+# ends with "GB", "MB" or "kB" if over 1023, exactly what Postgresql
+# will expect in a postgresql.conf setting. The output may be up to
+# 6.25% less than the original value because of the rounding.
+def binaryround(value)
+
+ # Keep a multiplier which grows through powers of 1
+ multiplier = 1
+
+ # Truncate value to 4 most significant bits
+ while value >= 16
+ value = (value / 2).floor
+ multiplier = multiplier * 2
+ end
+
+ # Factor any remaining powers of 2 into the multiplier
+ while value == 2*((value / 2).floor)
+ value = (value / 2).floor
+ multiplier = multiplier * 2
+ end
+
+ # Factor enough powers of 2 back into the value to
+ # leave the multiplier as a power of 1024 that can
+ # be represented as units of "GB", "MB" or "kB".
+ if multiplier >= 1024*1024*1024
+ while multiplier > 1024*1024*1024
+ value = 2*value
+ multiplier = (multiplier/2).floor
+ end
+ multiplier = 1
+ units = "GB"
+
+ elsif multiplier >= 1024*1024
+ while multiplier > 1024*1024
+ value = 2*value
+ multiplier = (multiplier/2).floor
+ end
+ multiplier = 1
+ units = "MB"
+
+ elsif multiplier >= 1024
+ while multiplier > 1024
+ value = 2*value
+ multiplier = (multiplier/2).floor
+ end
+ multiplier = 1
+ units = "kB"
+
+ else
+ units = ""
+ end
+
+ # Now we can return a nice human readable string.
+ return "#{multiplier * value}#{units}"
+end
+
+#######
+# Locale Configuration
+
+# Function to test the date order.
+# Used in recipes/config_initdb.rb to set this attribute:
+# node.default['postgresql']['config']['datestyle']
+def locale_date_order
+ # Test locale conversion of mon=11, day=22, year=33
+ testtime = DateTime.new(2033,11,22,0,0,0,"-00:00")
+ #=> #<DateTime: 2033-11-22T00:00:00-0000 ...>
+
+ # %x - Preferred representation for the date alone, no time
+ res = testtime.strftime("%x")
+
+ if res.nil?
+ return 'mdy'
+ end
+
+ posM = res.index("11")
+ posD = res.index("22")
+ posY = res.index("33")
+
+ if (posM.nil? || posD.nil? || posY.nil?)
+ return 'mdy'
+ elseif (posY < posM && posM < posD)
+ return 'ymd'
+ elseif (posD < posM)
+ return 'dmy'
+ else
+ return 'mdy'
+ end
+end
+
+#######
+# Timezone Configuration
+require 'find'
+
+# Function to determine where the system stored shared timezone data.
+# Used in recipes/config_initdb.rb to detemine where it should have
+# select_default_timezone(tzdir) search.
+def pg_TZDIR()
+ # System time zone conversions are controlled by a timezone data file
+ # identified through environment variables (TZ and TZDIR) and/or file
+ # and directory naming conventions specific to the Linux distribution.
+ # Each of these timezone names will have been loaded into the PostgreSQL
+ # pg_timezone_names view by the package maintainer.
+ #
+ # Instead of using the timezone name configured as the system default,
+ # the PostgreSQL server uses ones named in postgresql.conf settings
+ # (timezone and log_timezone). The initdb utility does initialize those
+ # settings to the timezone name that corresponds to the system default.
+ #
+ # The system's timezone name is actually a filename relative to the
+ # shared zoneinfo directory. That is usually /usr/share/zoneinfo, but
+ # it was /usr/lib/zoneinfo in older distributions and can be anywhere
+ # if specified by the environment variable TZDIR. The tzset(3) manpage
+ # seems to indicate the following precedence:
+ tzdir = nil
+ if ::File.directory?("/usr/lib/zoneinfo")
+ tzdir = "/usr/lib/zoneinfo"
+ else
+ share_path = [ ENV['TZDIR'], "/usr/share/zoneinfo" ].compact.first
+ if ::File.directory?(share_path)
+ tzdir = share_path
+ end
+ end
+ return tzdir
+end
+
+#######
+# Function to support select_default_timezone(tzdir), which is
+# used in recipes/config_initdb.rb.
+def validate_zone(tzname)
+ # PostgreSQL does not support leap seconds, so this function tests
+ # the usual Linux tzname convention to avoid a misconfiguration.
+ # Assume that the tzdata package maintainer has kept all timezone
+ # data files with support for leap seconds is kept under the
+ # so-named "right/" subdir of the shared zoneinfo directory.
+ #
+ # The original PostgreSQL initdb is not Unix-specific, so it did a
+ # very complicated, thorough test in its pg_tz_acceptable() function
+ # that I could not begin to understand how to do in ruby :).
+ #
+ # Testing the tzname is good enough, since a misconfiguration
+ # will result in an immediate fatal error when the PostgreSQL
+ # service is started, with pgstartup.log messages such as:
+ # LOG: time zone "right/US/Eastern" appears to use leap seconds
+ # DETAIL: PostgreSQL does not support leap seconds.
+
+ if tzname.index("right/") == 0
+ return false
+ else
+ return true
+ end
+end
+
+# Function to support select_default_timezone(tzdir), which is
+# used in recipes/config_initdb.rb.
+def scan_available_timezones(tzdir)
+ # There should be an /etc/localtime zoneinfo file that is a link to
+ # (or a copy of) a timezone data file under tzdir, which should have
+ # been installed under the "share" directory by the tzdata package.
+ #
+ # The initdb utility determines which shared timezone file is being
+ # used as the system's default /etc/localtime. The timezone name is
+ # the timezone file path relative to the tzdir.
+
+ bestzonename = nil
+
+ if (tzdir.nil?)
+ Chef::Log.error("The zoneinfo directory not found (looked for /usr/share/zoneinfo and /usr/lib/zoneinfo)")
+ elsif !::File.exists?("/etc/localtime")
+ Chef::Log.error("The system zoneinfo file not found (looked for /etc/localtime)")
+ elsif ::File.directory?("/etc/localtime")
+ Chef::Log.error("The system zoneinfo file not found (/etc/localtime is a directory instead)")
+ elsif ::File.symlink?("/etc/localtime")
+ # PostgreSQL initdb doesn't use the symlink target, but this
+ # certainly will make sense to any system administrator. A full
+ # scan of the tzdir to find the shortest filename could result
+ # "US/Eastern" instead of "America/New_York" as bestzonename,
+ # in spite of what the sysadmin had specified in the symlink.
+ # (There are many duplicates under tzdir, with the same timezone
+ # content appearing as an average of 2-3 different file names.)
+ path = ::File.readlink("/etc/localtime")
+ bestzonename = path.gsub("#{tzdir}/","")
+ else # /etc/localtime is a file, so scan for it under tzdir
+ localtime_content = File.read("/etc/localtime")
+
+ Find.find(tzdir) do |path|
+ # Only consider files (skip directories or symlinks)
+ if !::File.directory?(path) && !::File.symlink?(path)
+ # Ignore any file named "posixrules" or "localtime"
+ if ::File.basename(path) != "posixrules" && ::File.basename(path) != "localtime"
+ # Do consider if content exactly matches /etc/localtime.
+ if localtime_content == File.read(path)
+ tzname = path.gsub("#{tzdir}/","")
+ if validate_zone(tzname)
+ if (bestzonename.nil? ||
+ tzname.length < bestzonename.length ||
+ (tzname.length == bestzonename.length &&
+ (tzname <=> bestzonename) < 0)
+ )
+ bestzonename = tzname
+ end
+ end
+ end
+ end
+ end
+ end
+ end
+
+ return bestzonename
+end
+
+# Function to support select_default_timezone(tzdir), which is
+# used in recipes/config_initdb.rb.
+def identify_system_timezone(tzdir)
+ resultbuf = scan_available_timezones(tzdir)
+
+ if !resultbuf.nil?
+ # Ignore Olson's rather silly "Factory" zone; use GMT instead
+ if (resultbuf <=> "Factory") == 0
+ resultbuf = nil
+ end
+
+ else
+ # Did not find the timezone. Fallback to use a GMT zone. Note that the
+ # Olson timezone database names the GMT-offset zones in POSIX style: plus
+ # is west of Greenwich.
+ testtime = DateTime.now
+ std_ofs = testtime.strftime("%:z").split(":")[0].to_i
+
+ resultbuf = [
+ "Etc/GMT",
+ (-std_ofs > 0) ? "+" : "",
+ (-std_ofs).to_s
+ ].join('')
+ end
+
+ return resultbuf
+end
+
+#######
+# Function to determine the name of the system's default timezone.
+# Used in recipes/config_initdb.rb to set these attributes:
+# node.default['postgresql']['config']['log_timezone']
+# node.default['postgresql']['config']['timezone']
+def select_default_timezone(tzdir)
+
+ system_timezone = nil
+
+ # Check TZ environment variable
+ tzname = ENV['TZ']
+ if !tzname.nil? && !tzname.empty? && validate_zone(tzname)
+ system_timezone = tzname
+
+ else
+ # Nope, so try to identify system timezone from /etc/localtime
+ tzname = identify_system_timezone(tzdir)
+ if validate_zone(tzname)
+ system_timezone = tzname
+ end
+ end
+
+ return system_timezone
+end
+
+# End the Opscode::PostgresqlHelper module
+ end
+end
148 recipes/config_initdb.rb
View
@@ -0,0 +1,148 @@
+#
+# Cookbook Name:: postgresql
+# Recipe:: config_initdb
+# Author:: David Crane (<davidc@donorschoose.org>)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#######
+# Load the locale_date_order() and select_default_timezone(tzdir)
+# methods from libraries/default.rb
+::Chef::Recipe.send(:include, Opscode::PostgresqlHelpers)
+
+#######
+# This recipe is derived from the setup_config() source code in the
+# PostgreSQL initdb utility. It determines postgresql.conf settings that
+# conform to the system's locale and timezone configuration, and also
+# sets the error reporting and logging settings.
+#
+# See http://doxygen.postgresql.org/initdb_8c_source.html for the
+# original initdb source code.
+#
+# By examining the system configuration, this recipe will set the
+# following node.default['postgresql']['config'] attributes:
+#
+# - Locale and Formatting -
+# * datestyle
+# * lc_messages
+# * lc_monetary
+# * lc_numeric
+# * lc_time
+# * default_text_search_config
+#
+# - Timezone Conversion -
+# * log_timezone
+# * timezone
+#
+# In addition, this recipe will recommend the same error reporting and
+# logging settings that initdb provided. These settings do differ from
+# the PostgreSQL default settings, which would log to stderr only. The
+# initdb settings rotate 7 days of log files named postgresql-Mon.log,
+# etc. through these node.default['postgresql']['config'] attributes:
+#
+# - Where to Log -
+# * log_destination = 'stderr'
+# * log_directory = 'pg_log'
+# * log_filename = 'postgresql-%a.log'
+# (Default was: postgresql-%Y-%m-%d_%H%M%S.log)
+# * logging_collector = true # on
+# (Turned on to capture stderr logging and redirect into log files)
+# (Default was: false # off)
+# * log_rotation_age = 1d
+# * log_rotation_size = 0
+# (Default was: 10MB)
+# * log_truncate_on_rotation = true # on
+# (Default was: false # off)
+
+#######
+# Locale Configuration
+
+# See libraries/default.rb for the locale_date_order() method.
+node.default['postgresql']['config']['datestyle'] = "iso, #{locale_date_order()}"
+
+# According to the locale(1) manpage, the locale settings are determined
+# by environment variables according to the following precedence:
+# LC_ALL > (LC_MESSAGES, LC_MONETARY, LC_NUMERIC, LC_TIME) > LANG.
+
+node.default['postgresql']['config']['lc_messages'] =
+ [ ENV['LC_ALL'], ENV['LC_MESSAGES'], ENV['LANG'] ].compact.first
+
+node.default['postgresql']['config']['lc_monetary'] =
+ [ ENV['LC_ALL'], ENV['LC_MONETARY'], ENV['LANG'] ].compact.first
+
+node.default['postgresql']['config']['lc_numeric'] =
+ [ ENV['LC_ALL'], ENV['LC_NUMERIC'], ENV['LANG'] ].compact.first
+
+node.default['postgresql']['config']['lc_time'] =
+ [ ENV['LC_ALL'], ENV['LC_TIME'], ENV['LANG'] ].compact.first
+
+node.default['postgresql']['config']['default_text_search_config'] =
+ case ENV['LANG']
+ when /da_.*/
+ 'pg_catalog.danish'
+ when /nl_.*/
+ 'pg_catalog.dutch'
+ when /en_.*/
+ 'pg_catalog.english'
+ when /fi_.*/
+ 'pg_catalog.finnish'
+ when /fr_.*/
+ 'pg_catalog.french'
+ when /de_.*/
+ 'pg_catalog.german'
+ when /hu_.*/
+ 'pg_catalog.hungarian'
+ when /it_.*/
+ 'pg_catalog.italian'
+ when /no_.*/
+ 'pg_catalog.norwegian'
+ when /pt_.*/
+ 'pg_catalog.portuguese'
+ when /ro_.*/
+ 'pg_catalog.romanian'
+ when /ru_.*/
+ 'pg_catalog.russian'
+ when /es_.*/
+ 'pg_catalog.spanish'
+ when /sv_.*/
+ 'pg_catalog.swedish'
+ when /tr_.*/
+ 'pg_catalog.turkish'
+ else
+ nil
+ end
+
+#######
+# Timezone Configuration
+
+# Determine the name of the system's default timezone and specify node
+# defaults for the postgresql.cof settings. If the timezone cannot be
+# identified, do as initdb would do: leave it unspecified so PostgreSQL
+# uses it's internal default of GMT.
+tzdirpath = pg_TZDIR() # See libraries/default.rb
+default_timezone = select_default_timezone(tzdirpath) # See libraries/default.rb
+if !default_timezone.nil?
+ node.default['postgresql']['config']['log_timezone'] = default_timezone
+ node.default['postgresql']['config']['timezone'] = default_timezone
+end
+
+#######
+# - Where to Log -
+node.default['postgresql']['config']['log_destination'] = 'stderr'
+node.default['postgresql']['config']['log_directory'] = 'pg_log'
+node.default['postgresql']['config']['log_filename'] = 'postgresql-%a.log'
+node.default['postgresql']['config']['logging_collector'] = true # on
+node.default['postgresql']['config']['log_rotation_age'] = '1d'
+node.default['postgresql']['config']['log_rotation_size'] = 0
+node.default['postgresql']['config']['log_truncate_on_rotation'] = true # on
280 recipes/config_pgtune.rb
View
@@ -0,0 +1,280 @@
+#
+# Cookbook Name:: postgresql
+# Recipe:: config_pgtune
+# Author:: David Crane (<davidc@donorschoose.org>)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#######
+# Load the binaryround(value) method from libraries/default.rb
+::Chef::Recipe.send(:include, Opscode::PostgresqlHelpers)
+
+#######
+# This recipe is based on Greg Smith's pgtune script (the Feb 1, 2012
+# version at https://github.com/gregs1104/pgtune). Introduction: pgtune
+# takes the wimpy default postgresql.conf and expands the database
+# server to be as powerful as the hardware it's being deployed on.
+#
+# The default postgresql.conf aims at a system with approximately 128MB
+# of RAM. This recipe recommends a baseline configuration in the right
+# general range for a dedicated Postgresql system.
+#
+# This recipe takes three optional parameters that may be passed in as
+# node['postgresql']['config_pgtune'] attributes:
+# * db_type -- Specifies database type as one of: dw, oltp,
+# web, mixed, desktop. If not specified, the default is mixed.
+# * max_connections -- Specifies number of maximum connections
+# expected. If not specified, it depends on database type.
+# * total_memory -- Specifies total system memory. If not specified,
+# it will be detected from the Ohai automatic attributes.
+#
+# Using those inputs, this recipe will compute and set the following
+# node.default['postgresql']['config'] attributes:
+# * max_connections
+# * shared_buffers
+# * effective_cache_size
+# * work_mem
+# * maintenance_work_mem
+# * checkpoint_segments
+# * checkpoint_completion_target
+# * wal_buffers
+# * default_statistics_target
+#
+# This recipe deviates from the original pgtune script for 2 settings:
+# shared_buffers is capped for large memory systems (which Greg
+# mentioned in a TODO.rst) and wal_buffers will auto-tune starting with
+# 9.1 (which is a feature that Greg built into Postgresql).
+
+#######
+# These are the workload characteristics of the five database types
+# that can be specified as node['postgresql']['config_pgtune']['db_type']:
+#
+# dw -- Data Warehouse
+# * Typically I/O- or RAM-bound
+# * Large bulk loads of data
+# * Large complex reporting queries
+# * Also called "Decision Support" or "Business Intelligence"
+#
+# oltp -- Online Transaction Processing
+# * Typically CPU- or I/O-bound
+# * DB slightly larger than RAM to 1TB
+# * 20-40% small data write queries
+# * Some long transactions and complex read queries
+#
+# web -- Web Application
+# * Typically CPU-bound
+# * DB much smaller than RAM
+# * 90% or more simple queries
+#
+# mixed -- Mixed DW and OLTP characteristics
+# * A wide mixture of queries
+#
+# desktop -- Not a dedicated database
+# * A general workstation, perhaps for a developer
+
+# Parse out db_type option, or use default.
+db_type = 'mixed'
+
+if (node['postgresql'].attribute?('config_pgtune') && node['postgresql']['config_pgtune'].attribute?('db_type'))
+ db_type = node['postgresql']['config_pgtune']['db_type']
+ if (!(["dw","oltp","web","mixed","desktop"].include?(db_type)))
+ Chef::Application.fatal!([
+ "Bad value (#{db_type})",
+ "for node['postgresql']['config_pgtune']['db_type'] attribute.",
+ "Valid values are one of dw, oltp, web, mixed, desktop."
+ ].join(' '))
+ end
+end
+
+# Parse out max_connections option, or use a value based on db_type.
+con =
+{ "web" => 200,
+ "oltp" => 300,
+ "dw" => 20,
+ "mixed" => 80,
+ "desktop" => 5
+}.fetch(db_type)
+
+if (node['postgresql'].attribute?('config_pgtune') && node['postgresql']['config_pgtune'].attribute?('max_connections'))
+ max_connections = node['postgresql']['config_pgtune']['max_connections']
+ if (max_connections.match(/\A[1-9]\d*\Z/) == nil)
+ Chef::Application.fatal!([
+ "Bad value (#{max_connections})",
+ "for node['postgresql']['config_pgtune']['max_connections'] attribute.",
+ "Valid values are non-zero integers only."
+ ].join(' '))
+ end
+ con = max_connections.to_i
+end
+
+# Parse out total_memory option, or use value detected by Ohai.
+total_memory = node['memory']['total']
+
+# Override max_connections with a node attribute if DevOps desires.
+# For example, on a system *not* dedicated to Postgresql.
+if (node['postgresql'].attribute?('config_pgtune') && node['postgresql']['config_pgtune'].attribute?('total_memory'))
+ total_memory = node['postgresql']['config_pgtune']['total_memory']
+ if (total_memory.match(/\A[1-9]\d*kB\Z/) == nil)
+ Chef::Application.fatal!([
+ "Bad value (#{total_memory})",
+ "for node['postgresql']['config_pgtune']['total_memory'] attribute.",
+ "Valid values are non-zero integers followed by kB (e.g., 49416564kB)."
+ ].join(' '))
+ end
+end
+
+# Ohai reports node[:memory][:total] in kB, as in "921756kB"
+mem = total_memory.split("kB")[0].to_i / 1024 # in MB
+
+#######
+# RAM-related settings computed as in Greg Smith's pgtune script.
+# Remember that con and mem were either chosen above based on the
+# db_type or the actual total memory, or were passed in attributes.
+
+# (1) max_connections
+# Sets the maximum number of concurrent connections.
+node.default['postgresql']['config']['max_connections'] = con
+
+# The calculations for the next four settings would not be optimal
+# for low memory systems. In that case, the calculation is skipped,
+# leaving the built-in Postgresql settings, which are actually
+# intended for those low memory systems.
+if (mem >= 256)
+
+ # (2) shared_buffers
+ # Sets the number of shared memory buffers used by the server.
+ shared_buffers =
+ { "web" => mem/4,
+ "oltp" => mem/4,
+ "dw" => mem/4,
+ "mixed" => mem/4,
+ "desktop" => mem/16
+ }.fetch(db_type)
+
+ # Robert Haas has advised to cap the size of shared_buffers based on
+ # the memory architecture: 2GB on 32-bit and 8GB on 64-bit machines.
+ # http://rhaas.blogspot.com/2012/03/tuning-sharedbuffers-and-walbuffers.html
+ case node['kernel']['machine']
+ when "i386" # 32-bit machines
+ if shared_buffers > 2*1024
+ shared_buffers = 2*1024
+ end
+ when "x86_64" # 64-bit machines
+ if shared_buffers > 8*1024
+ shared_buffers = 8*1024
+ end
+ end
+
+ node.default['postgresql']['config']['shared_buffers'] = binaryround(shared_buffers*1024*1024)
+
+ # (3) effective_cache_size
+ # Sets the planner's assumption about the size of the disk cache.
+ # That is, the portion of the kernel's disk cache that will be
+ # used for PostgreSQL data files.
+ effective_cache_size =
+ { "web" => mem * 3 / 4,
+ "oltp" => mem * 3 / 4,
+ "dw" => mem * 3 / 4,
+ "mixed" => mem * 3 / 4,
+ "desktop" => mem / 4
+ }.fetch(db_type)
+
+ node.default['postgresql']['config']['effective_cache_size'] = binaryround(effective_cache_size*1024*1024)
+
+ # (4) work_mem
+ # Sets the maximum memory to be used for query workspaces.
+ work_mem =
+ { "web" => mem / con,
+ "oltp" => mem / con,
+ "dw" => mem / con / 2,
+ "mixed" => mem / con / 2,
+ "desktop" => mem / con / 6
+ }.fetch(db_type)
+
+ node.default['postgresql']['config']['work_mem'] = binaryround(work_mem*1024*1024)
+
+ # (5) maintenance_work_mem
+ # Sets the maximum memory to be used for maintenance operations.
+ # This includes operations such as VACUUM and CREATE INDEX.
+ maintenance_work_mem =
+ { "web" => mem / 16,
+ "oltp" => mem / 16,
+ "dw" => mem / 8,
+ "mixed" => mem / 16,
+ "desktop" => mem / 16
+ }.fetch(db_type)
+
+ # Cap maintenence RAM at 1GB on servers with lots of memory
+ if (maintenance_work_mem > 1*1024)
+ maintenance_work_mem = 1*1024
+ end
+
+ node.default['postgresql']['config']['maintenance_work_mem'] = binaryround(maintenance_work_mem*1024*1024)
+
+end
+
+#######
+# Checkpoint-related parameters that affect transaction rate and
+# maximum tolerable recovery playback time.
+
+# (6) checkpoint_segments
+# Sets the maximum distance in log segments between automatic WAL checkpoints.
+checkpoint_segments =
+{ "web" => 8,
+ "oltp" => 16,
+ "dw" => 64,
+ "mixed" => 16,
+ "desktop" => 3
+}.fetch(db_type)
+
+node.default['postgresql']['config']['checkpoint_segments'] = checkpoint_segments
+
+# (7) checkpoint_completion_target
+# Time spent flushing dirty buffers during checkpoint, as fraction
+# of checkpoint interval.
+checkpoint_completion_target =
+{ "web" => "0.7",
+ "oltp" => "0.9",
+ "dw" => "0.9",
+ "mixed" => "0.9",
+ "desktop" => "0.5"
+}.fetch(db_type)
+
+node.default['postgresql']['config']['checkpoint_completion_target'] = checkpoint_completion_target
+
+# (8) wal_buffers
+# Sets the number of disk-page buffers in shared memory for WAL.
+# Starting with 9.1, wal_buffers will auto-tune if set to the -1 default.
+# For 8.X and 9.0, it needed to be specified, which pgtune did as follows.
+if node['postgresql']['version'].to_f < 9.1
+ wal_buffers = 512 * checkpoint_segments
+ # The pgtune seems to use 1kB units for wal_buffers
+ node.default['postgresql']['config']['wal_buffers'] = binaryround(wal_buffers*1024)
+else
+ node.default['postgresql']['config']['wal_buffers'] = "-1"
+end
+
+# (9) default_statistics_target
+# Sets the default statistics target. This applies to table columns
+# that have not had a column-specific target set via
+# ALTER TABLE SET STATISTICS.
+default_statistics_target =
+{ "web" => 100,
+ "oltp" => 100,
+ "dw" => 500,
+ "mixed" => 100,
+ "desktop" => 100
+}.fetch(db_type)
+
+node.default['postgresql']['config']['default_statistics_target'] = default_statistics_target
Something went wrong with that request. Please try again.