Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Batteries-included CouchDB build system
Ruby C
branch: master

Upgrade to Erlang/OTP R16B03-1

This is due to Russell Branca's notes, attached below.

There has been some discussion on what versions of Erlang CouchDB
should support, and what versions of Erlang are detrimental to
use. Sadly there were some pretty substantial problems in the R15 line
and even parts of R16 that are landmines for CouchDB. This post will
describe the current state of things and make some potential
recommendations on approach.

It was discovered by Basho that R15* and R16B are susceptible to
scheduler collapse. There's quite a bit of discussion and
information in several threads [1] [2] [3] [4] [5].

So what is scheduler collapse? Erlang schedulers can be put to sleep
when there is not sufficient work to occupy all schedulers, which
saves on CPU and power consumption. When the schedulers that are still
running go through enough reductions to pass the work balancing
threshold, they can trigger a rebalance of work that will wake up
sleeping schedulers. The other mechanism for sharing scheduler load is
work stealing. A scheduler that does not have any work to do can
steal work from other schedulers. However a scheduler that has gone
to sleep cannot steal work, it has to be woken up separately.

Now the real problem of scheduler collapse occurs when you take
sleeping schedulers and long running NIFs and BIFs that do not report
an appropriate amount of reductions. When you have NIFs and BIFs that
don't report an appropriate amount of reductions, you can get into a
situation where a long running function call will only show up as
taking one reduction, and never hit the work balance threshold,
causing that scheduler to be blocked during the operation and no
additional schedulers getting woken up.

I keep mentioning "NIFs and BIFs" because it's important to note that
it is _not_ just user defined NIFs that are problematic, but also a
number of Erlang BIFs that don't properly report reductions.
Particularly relevant to CouchDB are the BIFs `term_to_binary` and
`binary_to_term` which do _not_ behave properly, and each report a
single reduction count, regardless of the size of the value passed to
them. Given that every write CouchDB makes goes through
`term_to_binary`, this is definitely not good.

This problem is systemic to all versions of R15 and R16B. In R16B01,
two changes were made to alleviate the problem. First, in `OTP-11163`
`term_to_binary` now uses an appropriate amount of reductions and will
yield back to the scheduler. The second important change was the
introduction of the `+sfwi` (Scheduler Forced Wakeup Interval) flag
[6] which allows you to specify a time interval for a new watchdog
process to check scheduler run queues and wake up sleeping schedulers
if need be. These two changes help significantly, although from what I
understand, they do not fully eliminate scheduler collapse.

*NOTE*: the `+sfwi` is _not_ enabled by default, you must specify a
greater than zero time interval to enable this. *WE NEED TO ENABLE
THIS SETTING.* We should figure out a way to conditionally add this
to vm.args or some such.

On a side note, Basho runs R15B01 because they backported the `+sfwi`
feature to R15B01 [7] [8]. They recommend running with `+sfwi 500` for
a 500ms interval. It might be worth testing out different values, but
500 seems like a good starting point. For Riak 2.0, they will be
building against R16B03-1 and 17.0 as their set of patches to R16B02
landed in R16B03-1 [9] [10].

So R16B01 sorted out the scheduler collapse issues, but unfortunately
it also broke monitors, which immediately disqualifies this release as
something we should recommend to users. The issues was fixed in
`OTP-11225` in R16B02.

I don't know of any catastrophic problems on the order of those
described above in either of these releases. Basho fixed a number of
unrelated bugs in R16B02 [9] [10] that have since landed in R16B03-1,
which indicates we should probably prefer R16B03-1 over R16B02. R16B03
is also disqualified because it broke SSL and `erl_syntax`, resulting
in the patched R16B03-1.

R14B01, R14B03, and R14B04 are known good stable releases of Erlang,
and in my opinion the only known stable releases > R13 that don't
present issues for CouchDB (I think R16B02/R16B03-1 are too new to
declare stable yet). As for R14B02, there are some bad `ets` issues
with that release.

It's worth pointing out that there are two known bugs in R14B01, as
Robert Newson explains:

```
There are two bugs in R14B01 that we do encounter, however. 1) Another
32/64 bit oops causes the vm to attempt to allocate huge amounts of
ram (terabytes, or more) if it ever tries to allocate more than 2gib
of ram at once. When this happens, the vm dies and is restarted. It’s
annoying, but infrequent. 2) Sometimes when closing a file, the
underlying file descriptor is *not* closed, though the erlang process
exits. This is rare but still quite annoying.
```

The 17.0 release brings in a number of interesting changes to help the
scheduler collapse situation. `OTP-11648` improves reduction cost and
yielding of `term_to_binary`. It also utilizes `OTP-11388` which
allows for NIFs and BIFs to have more control over when and how they
are garbage collected (we should do some investigation on the
usefulness of this for NIFs like Jiffy).

The 17.0 release also updates `binary_to_term` in `OTP-11535` to
behave properly with reductions and yielding similar to
`term_to_binary`. This marks the 17.0 release as an important one for
CouchDB as now `term_to_binary` and `binary_to_term` both behave
properly.

One other interesting item introduced in the 17.0 release is the
concept of dirty schedulers [12] [13]. This is an experimental feature
providing CPU and I/O schedulers specifically for NIFs that are known
to take longer that 1ms to run. In general, we want to make sure the
NIFs we use will yield and report reductions properly, but for
situations where that isn't feasible, we may want to look into using
dirty schedulers down the road when it's a non experimental feature.

In my opinion we need to take the Erlang release issues more seriously
than we currently do and provide strong recommendations to users on
what versions of Erlang we support. I suggest we loosely take an
approach similar to Debian, and make three recommendations:

  * OldStable: [R14B01, R14B03, R14B04 (NOTE: _not_ R14B02)]
  * Unstable: [R16B03-1 recommended, R16B02 acceptable]
  * Experimental: [17.0]

I'm not suggesting permanently having three Erlang releases
recommended like this, but it currently seems appropriate. I think
long term we should target 17.x as our preferred Erlang release, and
then make a CouchDB 3.0 release that is backwards incompatible with
anything less than 17.0 so that we can switch over to using maps.

The narrowness of the acceptable releases list is going to cause some
problems. Debian Wheezy runs R15B01, which as established above, is
not good to run with unless you have the `+sfwi` patch, and I'm sure
there are many other distros running R15 and R16B or R16B01. I think
it would be useful to users to have a set of packages with a proper
Erlang CouchDB release allowing us to bless specific versions of
Erlang and bundle it together, but I know this idea goes against the
recent change in stance on working with distributions, and I don't
know the ASF stance on this issue well enough to comment on the
legality of it. That said, it does seem like the logical approach
until we get a range of stable releases spread out through the
distros.

We need to make sure that all NIFs we use that could potentially take
longer than 1ms to run properly yield and report reductions. For
Jiffy, there is already a good start on this work [11]. We'll want to
look into what needs to be done for the rest of the NIFs.

There's quite a bit of information here, and plenty more in the
footnotes, so I hope this gives a good overview of the current state
of Erlang releases and helps us to make informed decisions on what
approach to take with Erlang releases.

[1] http://comments.gmane.org/gmane.comp.lang.erlang.bugs/3564

[2] http://erlang.org/pipermail/erlang-questions/2013-April/073490.html

[3] http://erlang.org/pipermail/erlang-questions/2012-October/069503.html

[4] http://erlang.org/pipermail/erlang-questions/2012-October/069585.html

[5] http://permalink.gmane.org/gmane.comp.lang.erlang.bugs/3573

[6] http://erlang.org/pipermail/erlang-patches/2013-June/004109.html

[7] https://gist.github.com/evanmcc/a599f4c6374338ed672e

[8] http://data.story.lu/2013/06/23/riak-1-3-2-released

[9] basho/otp@erlang:maint...OTP_R16B02_basho4

[10] https://groups.google.com/forum/#!topic/nosql-databases/XpFKVeUBdn0

[11] davisp/jiffy#49

[12] erlang/otp@c1c03ae

[13] http://www.erlang.org/doc/man/erl_nif.html#dirty_nifs
latest commit 849fa7ec89
Jason Smith authored

README.md

Build CouchDB

Build CouchDB is a wrapper or master project which pulls in, from official sources, CouchDB plus all of its dependencies. It is the most straightforward and reliable procedure to build official CouchDB releases from source.

Build CouchDB builds an isolated, independent server. You do not need administrator access to run it. You can run several couches (for example, 0.10, 0.11, 1.0 releases) side-by-side.

Supported Platforms

Build CouchDB is developed and tested on the following operating systems:

  • Red Hat Enterprise Linux Server release 5.5 (Tikanga)
  • CentOS 5.5
  • Debian GNU/Linux 5.0 (Lenny)
  • Ubuntu
    • 9.10 (Karmic Koala)
    • 10.04 LTS (Lucid Lynx)
    • 10.10 (Maverik Meerkat)
    • 11.04 (Natty Narwhal)
    • 11.10 (Oneiric Ocelot)
  • Fedora 13
  • Mac OS X
  • OpenSUSE 11.3
  • Scientific Linux 5.3
  • Solaris 10, OpenSolaris

The following systems are planned for support in the near future:

  • MS Windows Vista, Windows 7

Requirements

You need only a few packages provided by the operating system. Copy and paste the commands below.

On Fedora:

sudo yum install gcc gcc-c++ make libtool zlib-devel openssl-devel rubygem-rake

On Red Hat Enterprise Linux:

The procedure is the same as Fedora, but also install the ruby-rdoc package.

On Debian, first install sudo and add yourself to /etc/sudoers.

su -
apt-get install sudo
visudo

On Ubuntu and Debian:

sudo apt-get install help2man make gcc zlib1g-dev libssl-dev rake texinfo flex dctrl-tools libsctp-dev libxslt1-dev libcap2-bin ed

On OpenSUSE:

sudo zypper install flex lksctp-tools-devel zip \
            rubygem-rake gcc-c++ make m4 zlib-devel \
            libopenssl-devel libtool automake

On Scientific Linux

sudo yum install --enablerepo=dag gcc gcc-c++ libtool zlib-devel openssl-devel \
            autoconf213

On Solaris

This build only supports the OpenCSW toolchain. If you do not use OpenCSW, I wish you the best. If you have success, let me know!

The SunStudio tools are required:

sudo pkg install ss-dev

Also, OpenCSW packages are needed.

pkgadd -d http://mirror.opencsw.org/opencsw/pkg_get.pkg # Answer all questions affirmatively

Add CSW to your path. This must always be in the PATH. Every time you log in, you must set the correct $PATH (or make it automatic in .profile).

PATH=/opt/csw/bin:$PATH

Change the package archive (ibiblio URL is down) by running vi /opt/csw/etc/pkg-get.conf and setting url=ftp://ftp.ibiblio.org/pub/mirrors/opencsw/current. Save and exit, then run:

pkg-get updatecatalog

Finally, install Rake from OpenCSW:

sudo pkg-get install ruby rake # Also perhaps "git"

On Mac OS X

Install Xcode from Mac App Store. Launch XCode.app, then go to Perferences, select Downloads tab and Install Command Line Tools (You will need Apple Developer ID to download CLT).

Getting the Code

You will need the Git tool. Check out the code and pull in the third-party submodules.

git clone git://github.com/jhs/build-couchdb.git
cd build-couchdb
git submodule init
git submodule update

How to Build CouchDB

Just run Rake.

rake

CouchDB and all its dependencies will install in the build/. To uninstall, simply delete that directory.

Usage

It's CouchDB! Just type couchdb. (But remember the path)

$ build/bin/couchdb
Apache CouchDB 0.12.0aa63efb6-git (LogLevel=info) is starting.
Apache CouchDB has started. Time to relax.
[info] [<0.33.0>] Apache CouchDB has started on http://127.0.0.1:5984/

You can of course call it by absolute path. If your username is amit and you checked out the code in your home directory, you would run:

/home/amit/build-couchdb/build/bin/couchdb

Conveniently Add CouchDB to the PATH

The build process creates a small shell script, build/env.sh. The script will add the build to your shell's $PATH. This will only affect that shell session, other terminals or shell sessions will not change. (This is on purpose, to isolate CouchDB, so that it is easy to remove, or so multiple versions can be installed side-by-side.)

Simply source the script when you want to use CouchDB.

. build/env.sh

Your working directory needn't be anywhere special when sourcing the file. It can be processed from anywhere. The idea is, when you are working, you realize you need couchdb, just type . ~/my/stuff/code/build-couchdb/build/env.sh or whatever and it will work.

You can source the file as often as you like. Subsequent execution will not do anything.

. build/env.sh
. build/env.sh # Sourcing with wild abandon!

If the file is read from a script or in a pipeline, it will execute silently (by detecting whether it is connected to a TTY terminal).

Cheat Codes

Build CouchDB supports some fancy tricks by entering cheat codes to the Rake command.

Build any Git branch or tag of Erlang/OTP

Add a erl_checkout parameter with the commit id, branch name, or tag name.

For example, to build with Erlang R13B04 release:

rake erl_checkout="OTP_R13B04"

Build any Git branch or tag of CouchDB

Add a git parameter with the repository URL, then a space, then the branch, tag, or commit hash. (Remember to quote all of those so Rake sees the space.)

CouchDB Plugins

Any CouchDB plugin can be loaded remotely from Git, built, and installed into the final CouchDB system.

rake plugin="git://github.com/couchbase/geocouch origin/couchdb1.2.x"
# (Or perhaps origin/couchdb_1.1.x)

Multiple plugins can be processed together:

rake plugins="git://github.com/vmx/couchdb origin/gc-separate,git://github.com/somebody/whatever some_tag"

(Both plugin and plugins supports comma-separated lists; use whichever you remember better.)

Install CouchDB somewhere besides build/.

Add an install parameter to place the final couchdb binaries anywhere.

Build CouchDB makes it simple to install several couchdb versions side-by-side.

rake install=$PWD/stable
rake git="git://git.apache.org/couchdb.git trunk" install=$PWD/trunk
for tag in 1.0.1 11.0 11.1; do
    rake git="git://git.apache.org/couchdb.git tags/$tag" install=$PWD/$tag
done

Note that install needs to be an absolute path.

For side-by-side installs there is a small shortcut to avoid rebuilding Erlang: use the couchdb_build variable instead, which will install CouchDB separately from its dependencies. Just remember never to move or delete the dependencies!

rake install=/dependencies/go/here couchdb_build=/but/couch/goes/here

Support "unclean" builds.

Build CouchDB confirms that the Git checkout looks good before attempting a build. If you see this error message, then Build CouchDB is suspicious of your checkout:

This checkout is not clean:
<list of changed files>

Heed this warning. Why is your checkout unclean? Shouldn't you build from a nice, clean checkout, with no funny business?

Nevertheless, if you wish to proceed, add an unclean parameter to Rake:

rake unclean=1

Get a manifest of all the components

To get a better idea of exactly what is going on, add a manifest parameter.

rake manifest=1

That will produce additional files in build/manifest which indicate which package (icu, erlang, spidermonkey, etc) owns which files within build. A trick I do a lot is cat build/manifest/couchdb | xargs rm to "uninstall" only couchdb so I can try a rebuild.

I have no idea how manifest interacts with install as I have never used them together.

Do not strip down Erlang/OTP

Build CouchDB strips many modules out of the Erlang platform to reduce disk usage. (You can see which ones at the top of tasks/erlang.rake.) To indicate that a package should be kept, set the otp_keep variable to space-separated library names.

rake otp_keep="compiler eunit"

For example, if you will use this project with Rebar, then you need the tools package.

rake otp_keep="tools"

Or, you can keep everything this way:

rake otp_keep="*"

How to build only Erlang, couchjs, and OTP so you can build your own CouchDB elsewhere

There is a special shortcut task to build everything CouchDB needs (i.e. its dependencies).

rake couchdb:deps otp_keep="*"

Be careful not to build the couchdb target because after it completes, it will delete Erlang components needed for building (but not running). Next, there is a simple task which outputs a sh script used to configure any CouchDB checkout.

rake --silent environment:configure

The output will look similar to this:

export PATH="/Users/jhs/src/build-couchdb/build/bin:$PATH"
LDFLAGS='-R/Users/jhs/src/build-couchdb/build/lib -L/Users/jhs/src/build-couchdb/build/lib' CFLAGS='-I/Users/jhs/src/build-couchdb/build/include/js -I/Users/jhs/src/build-couchdb/build/lib/erlang/usr/include' ./configure

In the CouchDB source, paste the above code after running ./bootstrap. Next, you can run make or make dev, or anything.

vim: tw=80

Something went wrong with that request. Please try again.