recommended-production-settings.md

title	summary	toc	toc_not_nested
Recommended Production Settings	Recommended settings for production deployments.	false	true

This page provides recommended settings for production deployments.

Cluster Topology

When running a cluster with more than one node, each replica will be on a different node and a majority of replicas must remain available for the cluster to make progress. Therefore:

• Run at least three nodes to ensure that a majority of replicas (2/3) remains available when a node goes down.

• Run one node per machine. Since CockroachDB replicates across nodes, running more than one node per machine increases the risk of data unavailability if a machine fails.

• If a machine has multiple disks or SSDs, it's more efficient to run one node with multiple --store flags than one node per disk, because this informs CockroachDB about the relationship between the stores and ensures that data will be replicated across different machines instead of being assigned to different disks of the same machine. For more details about stores, see Start a Node.

• Configurations with odd numbers of replicas are more robust than those with even numbers. Clusters of three and four nodes can each tolerate one node failure and still reach a majority (2/3 and 3/4 respectively), so the fourth replica doesn't add any extra fault-tolerance. To survive two simultaneous failures, you must have five replicas.

• When replicating across datacenters, it's recommended to use datacenters on a single continent to ensure performance. Inter-continent scenarios will improve in performance soon.

For details about controlling the number and location of replicas, see Configure Replication Zones.

Clock Synchronization

CockroachDB needs moderately accurate time to preserve data consistency, so it's important to run NTP or other clock synchronization software on each machine. If clocks drift too far apart, nodes will self-terminate, but this mechanism is not fail-safe and data consistency guarantees may be lost.

Cache Size

If you run multiple applications on the same machine as a CockroachDB node, you might consider manually setting the cache size instead of using the default 25% of available memory.

To manually set the limit of the cache size, start the node using the --cache flag. For example, the following command limits a node's cache to 5GB:

$ cockroach start --cache=5GB <other start flags>

File Descriptors Limit

CockroachDB can use a large number of open file descriptors, often more than is available by default. Therefore, please note the following recommendations.

For each CockroachDB node:

• At a minimum, the file descriptors limit must be 2256 (2000 per store plus 256 for networking). If the limit is below this threshold, the node will not start.
• It is recommended to set the file descriptors limit to unlimited; otherwise, the recommended limit is at least 15000 (10000 per store plus 5000 for networking). This higher limit ensures performance and accommodates cluster growth.
• When the file descriptors limit is not high enough to allocate the recommended amounts, CockroachDB allocates 10000 per store and the rest for networking; if this would result in networking getting less than 256, CockroachDB instead allocates 256 for networking and evenly splits the rest across stores.

Increase the File Descriptors Limit

<script> $(document).ready(function(){ //detect os and display corresponding tab by default if (navigator.appVersion.indexOf("Mac")!=-1) { $('#os-tabs').find('button').removeClass('current'); $('#mac').addClass('current'); toggleMac(); } if (navigator.appVersion.indexOf("Linux")!=-1) { $('#os-tabs').find('button').removeClass('current'); $('#linux').addClass('current'); toggleLinux(); } if (navigator.appVersion.indexOf("Win")!=-1) { $('#os-tabs').find('button').removeClass('current'); $('#windows').addClass('current'); toggleWindows(); } var install_option = $('.install-option'), install_button = $('.install-button'); install_button.on('click', function(e){ e.preventDefault(); var hash = $(this).prop("hash"); install_button.removeClass('current'); $(this).addClass('current'); install_option.hide(); $(hash).show(); }); //handle click event for os-tab buttons $('#os-tabs').on('click', 'button', function(){ $('#os-tabs').find('button').removeClass('current'); $(this).addClass('current'); if($(this).is('#mac')){ toggleMac(); } if($(this).is('#linux')){ toggleLinux(); } if($(this).is('#windows')){ toggleWindows(); } }); function toggleMac(){ $(".mac-button:first").trigger('click'); $("#macinstall").show(); $("#linuxinstall").hide(); $("#windowsinstall").hide(); } function toggleLinux(){ $(".linux-button:first").trigger('click'); $("#linuxinstall").show(); $("#macinstall").hide(); $("#windowsinstall").hide(); } function toggleWindows(){ $("#windowsinstall").show(); $("#macinstall").hide(); $("#linuxinstall").hide(); } }); </script>

Mac Linux Windows

• Yosemite and later
• Older versions

Yosemite and later

To adjust the file descriptors limit for a single process in Mac OS X Yosemite and later, you must create a property list configuration file with the hard limit set to the recommendation mentioned above. Note that CockroachDB always uses the hard limit, so it's not technically necessary to adjust the soft limit, although we do so in the steps below.

For example, for a node with 3 stores, we would set the hard limit to at least 35000 (10000 per store and 5000 for networking) as follows:

1. Check the current limits:

   $ launchctl limit maxfiles
   maxfiles    10240          10240      

   The last two columns are the soft and hard limits, respectively. If unlimited is listed as the hard limit, note that the hidden default limit for a single process is actually 10240.

2. Create /Library/LaunchDaemons/limit.maxfiles.plist and add the following contents, with the final strings in the ProgramArguments array set to 35000:

   <?xml version="1.0" encoding="UTF-8"?>
   <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
     <plist version="1.0">
       <dict>
         <key>Label</key>
           <string>limit.maxfiles</string>
         <key>ProgramArguments</key>
           <array>
             <string>launchctl</string>
             <string>limit</string>
             <string>maxfiles</string>
             <string>35000</string>
             <string>35000</string>
           </array>
         <key>RunAtLoad</key>
           <true/>
         <key>ServiceIPC</key>
           <false/>
       </dict>
     </plist>

   Make sure the plist file is owned by root:wheel and has permissions -rw-r--r--. These permissions should be in place by default.

3. Restart the system for the new limits to take effect.

4. Check the current limits:

   $ launchctl limit maxfiles
   maxfiles    35000          35000      

Older versions

To adjust the file descriptors limit for a single process in OS X versions earlier than Yosemite, edit /etc/launchd.conf and increase the hard limit to the recommendation mentioned above. Note that CockroachDB always uses the hard limit, so it's not technically necessary to adjust the soft limit, although we do so in the steps below.

For example, for a node with 3 stores, we would set the hard limit to at least 35000 (10000 per store and 5000 for networking) as follows:

1. Check the current limits:

   $ launchctl limit maxfiles
   maxfiles    10240          10240      

   The last two columns are the soft and hard limits, respectively. If unlimited is listed as the hard limit, note that the hidden default limit for a single process is actually 10240.

2. Edit (or create) /etc/launchd.conf and add a line that looks like the following, with the last value set to the new hard limit:

   limit maxfiles 35000 35000
   

3. Save the file, and restart the system for the new limits to take effect.

4. Verify the new limits:

   $ launchctl limit maxfiles
   maxfiles    35000          35000      

• Standard
• With Systemd

Standard

To adjust the file descriptors limit for a single process on Linux, enable PAM user limits and set the hard limit to the recommendation mentioned above. Note that CockroachDB always uses the hard limit, so it's not technically necessary to adjust the soft limit, although we do so in the steps below.

For example, for a node with 3 stores, we would set the hard limit to at least 35000 (10000 per store and 5000 for networking) as follows:

1. Make sure the following line is present in both /etc/pam.d/common-session and /etc/pam.d/common-session-noninteractive:

   session    required   pam_limits.so

2. Edit /etc/security/limits.conf and append the following lines to the file:

   *              soft     nofile          35000
   *              hard     nofile          35000

   Note that * can be replaced with the username that will be running the CockroachDB server.

3. Save and close the file.

4. Restart the system for the new limits to take effect.

5. Verify the new limits:

   $ ulimit -a

With Systemd

Alternately, if you're using Systemd:

1. Edit the service definition to configure the maximum number of open files:

   [Service]
   ...
   LimitNOFILE=35000

2. Reload Systemd for the new limit to take effect:

   $ systemctl daemon-reload

CockroachDB does not yet provide a native Windows binary. Once that's available, we will also provide documentation on adjusting the file descriptors limit on Windows.

Attributions

This section, "File Descriptors Limit", is a derivative of Open File Limits by Riak, used under Creative Commons Attribution 3.0 Unported License.