Permalink
Commits on Oct 3, 2017
  1. Merge remote-tracking branch 'origin/5.3-stable'

    Eric Wong committed Oct 3, 2017
    * origin/5.3-stable:
      unicorn 5.3.1
      fix GC issue on rb_global_variable array
  2. unicorn 5.3.1

    Eric Wong committed Oct 3, 2017
    This release fixes an occasional GC problem introduced in v5.3.0
    to reduce global variable overhead (commit 979ebcf)
    
    Thanks to Xuanzhong Wei for the patch which lead to this release:
    
    https://bogomips.org/unicorn-public/20171003182054.76392-1-azrlew@gmail.com/T/#u
    https://bogomips.org/unicorn-public/20171003145718.30404-1-azrlew@gmail.com/T/#u
    
    Xuanzhong Wei (1):
          fix GC issue on rb_global_variable array
  3. fix GC issue on rb_global_variable array

    azrle committed with Eric Wong Oct 3, 2017
    We need to add the array to ruby's global_list right after created it;
    otherwise it probably gets GCed.
    
    (cherry picked from commit e85de00)
  4. fix GC issue on rb_global_variable array

    azrle committed with Eric Wong Oct 3, 2017
    We need to add the array to ruby's global_list right after created it;
    otherwise it probably gets GCed.
Commits on Apr 8, 2017
  1. reduce method calls with String#start_with?

    Eric Wong committed Apr 8, 2017
    These three cold call sites instruction sequence size by a few
    hundred bytes combined since we no longer support Ruby 1.8.6.
    The "?/" shorthand is esoteric and no longer avoids allocation
    in Ruby 1.9+ (not that this is hot code).
Commits on Apr 1, 2017
  1. unicorn 5.3.0

    Eric Wong committed Apr 1, 2017
    A couple of portability fixes from Dylan Thacker-Smith and
    Jeremy Evans since 5.3.0.pre1 over a week ago, but this looks
    ready for a stable release, today.
    
    When I started this over 8 years ago, I wondered if this would
    just end up being an April Fools' joke.  Guess not.  I guess I
    somehow tricked people into using a terribly marketed web server
    that cannot talk directly to untrusted clients :x  Anyways,
    unicorn won't be able to handle slow clients 8 years from now,
    either, or 80 years from now.  And I vow never to learn to use
    new-fangled things like epoll, kqueue, or threads :P
    
    Anyways, this is a largish release with several new features,
    and no backwards incompatibilities.
    
    Simon Eskildsen contributed heavily using TCP_INFO under Linux
    to implement the (now 5 year old) check_client_connection feature:
    
      https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-check_client_connection
      https://bogomips.org/unicorn-public/?q=s:check_client_connection&d:..20170401&x=t
    
    This also led to FreeBSD and OpenBSD portability improvements in
    one of our dependencies, raindrops:
    
       https://bogomips.org/raindrops-public/20170323024829.GA5190@dcvr/T/#u
    
    Jeremy Evans contributed several new features.  First he
    implemented after_worker_exit to aid debugging:
    
      https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_exit
      https://bogomips.org/unicorn-public/?q=s:after_worker_exit&d:..20170401&x=t#t
    
    And then security-related features to isolate workers.  Workers
    may now chroot to drop access to the master filesystem, and the
    new after_worker_ready configuration hook now exists to aid with
    chroot support in workers:
    
      https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_ready
      https://bogomips.org/unicorn/Unicorn/Worker.html#method-i-user
      https://bogomips.org/unicorn-public/?q=s:after_worker_ready&d:..20170401&x=t#t
      https://bogomips.org/unicorn-public/?q=s:chroot&d:..20170401&x=t#t
    
    Additionally, workers may run in a completely different VM space
    (nullifying preload_app and any CoW savings) with the new
    worker_exec option:
    
      https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-worker_exec
      https://bogomips.org/unicorn-public/?q=s:worker_exec&d:..20170401&x=t#t
    
    There are also several improvements to FreeBSD and OpenBSD
    support with the addition of these features.
    
    shortlog of changes since v5.2.0 (2016-10-31):
    
    Dylan Thacker-Smith (1):
          Check for Socket::TCP_INFO constant before trying to get TCP_INFO
    
    Eric Wong (30):
          drop rb_str_set_len compatibility replacement
          TUNING: document THP caveat for Linux users
          tee_input: simplify condition for IO#write
          remove response_start_sent
          http_request: freeze constant strings passed IO#write
          Revert "remove response_start_sent"
          t/t0012-reload-empty-config.sh: access ivars directly if needed
          t0011-active-unix-socket.sh: fix race condition in test
          new test for check_client_connection
          revert signature change to HttpServer#process_client
          support "struct tcp_info" on non-Linux and Ruby 2.2+
          unicorn_http: reduce rb_global_variable calls
          oob_gc: rely on opt_aref_with optimization on Ruby 2.2+
          http_request: reduce insn size for check_client_connection
          freebsd: avoid EINVAL when setting accept filter
          test-lib: expr(1) portability fix
          tests: keep disabled tests defined
          test_exec: SO_KEEPALIVE value only needs to be true
          doc: fix links to raindrops project
          http_request: support proposed Raindrops::TCP states on non-Linux
          ISSUES: expand on mail archive info + subscription disclaimer
          test_ccc: use a pipe to synchronize test
          doc: remove private email support address
          input: update documentation and hide internals.
          http_server: initialize @pid ivar
          gemspec: remove olddoc from build dependency
          doc: add version annotations for new features
          unicorn 5.3.0.pre1
          doc: note after_worker_exit is also 5.3.0+
          test_exec: SO_KEEPALIVE value only needs to be true (take #2)
    
    Jeremy Evans (7):
          Add after_worker_exit configuration option
          Fix code example in after_worker_exit documentation
          Add support for chroot to Worker#user
          Add after_worker_ready configuration option
          Add worker_exec configuration option
          Don't pass a block for fork when forking workers
          Check for SocketError on first ccc attempt
    
    Simon Eskildsen (1):
          check_client_connection: use tcp state on linux
Commits on Mar 27, 2017
  1. test_exec: SO_KEEPALIVE value only needs to be true (take #2)

    Eric Wong committed Mar 27, 2017
    We need to ensure the portability of the sd_listen_fds emulation
    test, too, which didn't get tested on my FreeBSD 10.3 install
    due to it being on Ruby 2.2
    
    Followup-to: 4ce6b00 ("test_exec: SO_KEEPALIVE value only needs to be true")
Commits on Mar 26, 2017
  1. Check for Socket::TCP_INFO constant before trying to get TCP_INFO

    dylanahsmith committed with Eric Wong Mar 26, 2017
    The ruby constant Socket::TCP_INFO is only defined if TCP_INFO is defined
    in C, so we can just check for the presence of that ruby constant instead
    of rescuing SocketError from the call to getsockopt.
Commits on Mar 24, 2017
  1. Check for SocketError on first ccc attempt

    jeremyevans committed with Eric Wong Mar 24, 2017
    On OpenBSD, getsockopt(2) does not support TCP_INFO.  With the current code,
    this results in a 500 for all clients if check_client_connection is enabled
    on OpenBSD.
    
    This patch rescues SocketError on the first getsockopt call, and
    if SocketError is raised, it doesn't check in the future.  This
    should be the same behavior as if TCP_INFO was supported but
    inspect did not return a string in the expected format.
  2. doc: note after_worker_exit is also 5.3.0+

    Eric Wong committed Mar 24, 2017
    Followup-to: 650e01a
      ("doc: add version annotations for new features")
  3. unicorn 5.3.0.pre1

    Eric Wong committed Mar 23, 2017
    A largish release with several new features.
    
    Simon Eskildsen contributed heavily using TCP_INFO under Linux
    to implement the (now 5 year old) check_client_connection feature:
    
      https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-check_client_connection
      https://bogomips.org/unicorn-public/?q=s:check_client_connection&d:..20170324&x=t
    
    This also led to FreeBSD and OpenBSD portability improvements in
    one of our dependencies, raindrops:
    
       https://bogomips.org/raindrops-public/20170323024829.GA5190@dcvr/T/#u
    
    Jeremy Evans contributed several new features.  First he
    implemented after_worker_exit to aid debugging:
    
      https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_exit
      https://bogomips.org/unicorn-public/?q=s:after_worker_exit&d:..20170324&x=t#t
    
    And then security-related features to isolate workers.  Workers
    may now chroot to drop access to the master filesystem, and the
    new after_worker_ready configuration hook now exists to aid with
    chroot support in workers:
    
      https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_ready
      https://bogomips.org/unicorn/Unicorn/Worker.html#method-i-user
      https://bogomips.org/unicorn-public/?q=s:after_worker_ready&d:..20170324&x=t#t
      https://bogomips.org/unicorn-public/?q=s:chroot&d:..20170324&x=t#t
    
    Additionally, workers may run in a completely different VM space
    (nullifying preload_app and any CoW savings) with the new
    worker_exec option:
    
      https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-worker_exec
      https://bogomips.org/unicorn-public/?q=s:worker_exec&d:..20170324&x=t#t
    
    There are also several improvements to FreeBSD and OpenBSD
    support with the addition of these features.
    
    34 changes since 5.2.0 (2016-10-31):
    
    Eric Wong (27):
          drop rb_str_set_len compatibility replacement
          TUNING: document THP caveat for Linux users
          tee_input: simplify condition for IO#write
          remove response_start_sent
          http_request: freeze constant strings passed IO#write
          Revert "remove response_start_sent"
          t/t0012-reload-empty-config.sh: access ivars directly if needed
          t0011-active-unix-socket.sh: fix race condition in test
          new test for check_client_connection
          revert signature change to HttpServer#process_client
          support "struct tcp_info" on non-Linux and Ruby 2.2+
          unicorn_http: reduce rb_global_variable calls
          oob_gc: rely on opt_aref_with optimization on Ruby 2.2+
          http_request: reduce insn size for check_client_connection
          freebsd: avoid EINVAL when setting accept filter
          test-lib: expr(1) portability fix
          tests: keep disabled tests defined
          test_exec: SO_KEEPALIVE value only needs to be true
          doc: fix links to raindrops project
          http_request: support proposed Raindrops::TCP states on non-Linux
          ISSUES: expand on mail archive info + subscription disclaimer
          test_ccc: use a pipe to synchronize test
          doc: remove private email support address
          input: update documentation and hide internals.
          http_server: initialize @pid ivar
          gemspec: remove olddoc from build dependency
          doc: add version annotations for new features
    
    Jeremy Evans (6):
          Add after_worker_exit configuration option
          Fix code example in after_worker_exit documentation
          Add support for chroot to Worker#user
          Add after_worker_ready configuration option
          Add worker_exec configuration option
          Don't pass a block for fork when forking workers
    
    Simon Eskildsen (1):
          check_client_connection: use tcp state on linux
Commits on Mar 23, 2017
  1. doc: add version annotations for new features

    Eric Wong committed Mar 8, 2017
    We will inevitably have people running old unicorn versions
    for many years to come; but they may be reading the latest
    documentation online.
    
    Annotate when the new features (will) appear to avoid misleading
    users on old versions.
  2. gemspec: remove olddoc from build dependency

    Eric Wong committed Mar 23, 2017
    It's a little less DRY, and there'll be no NEWS file generated,
    but it's one less thing to install, so perhaps that's worth it.
    The website at https://bogomips.org/unicorn/ will continue
    to use olddoc, of course,
  3. Merge remote-tracking branch 'origin/worker_exec'

    Eric Wong committed Mar 23, 2017
    * origin/worker_exec:
      Don't pass a block for fork when forking workers
      Add worker_exec configuration option
  4. Merge branch 'ccc-tcp-v3'

    Eric Wong committed Mar 23, 2017
    * ccc-tcp-v3:
      test_ccc: use a pipe to synchronize test
      http_request: support proposed Raindrops::TCP states on non-Linux
  5. http_server: initialize @pid ivar

    Eric Wong committed Mar 21, 2017
    This quiets down warnings when run with '-w'
  6. input: update documentation and hide internals.

    Eric Wong committed Mar 20, 2017
    rack 2.x exists nowadays still allows rewindable input as an
    option, and we will still enable it by default to avoid breaking
    any existing code.
    
    Hide the internal documentation since we do not want people
    depending on unicorn internals; so there's no reason to confuse
    or overwhelm people with documentation about it.  Arguably,
    TeeInput and StreamInput should not be documented publically at
    all, but I guess that ship has long sailed...
  7. doc: remove private email support address

    Eric Wong committed Mar 20, 2017
    Email was never private, and won't further burden myself or
    any future maintainers with trying to maintain someone elses'
    privacy.
    
    Offering private support is also unfair to readers on public
    lists who may get a watered down or improperly translated
    summary (if at all).
    
    Instead, encourage the use of anonymity tools and scrubbing of
    sensitive information when the sender deems necessary.
Commits on Mar 22, 2017
  1. test_ccc: use a pipe to synchronize test

    Eric Wong committed Mar 22, 2017
    Sleeping 1 second to test 100 requests is too long for some
    systems; and not long enough for others.
    
    We need to also finish reading the sleeper response to ensure
    the server actually got the second request in, before sending
    SIGQUIT to terminate it; as it's possible for the test client
    to connect and abort 100 clients before the server even
    increments the request counter for the 2nd request.
Commits on Mar 21, 2017
  1. ISSUES: expand on mail archive info + subscription disclaimer

    Eric Wong committed Mar 21, 2017
    Tis better to pull than push, or something like that.
  2. http_request: support proposed Raindrops::TCP states on non-Linux

    Eric Wong committed Mar 21, 2017
    raindrops 0.18+ will have Raindrops::TCP state hash for portable
    mapping of TCP states to their respective numeric values.  This
    was necessary because TCP state numbers (and even macro names)
    differ between FreeBSD and Linux (and possibly other OSes).
    
    Favor using the Raindrops::TCP state hash if available, but
    fall back to the hard-coded values since older versions of
    raindrops did not support TCP_INFO on non-Linux systems.
    
    While we're in the area, favor "const_defined?" over "defined?"
    to reduce the inline constant cache footprint for branches
    which are only evaluated once.
    
    Patches to implement Raindrops::TCP for FreeBSD are available at:
    
      https://bogomips.org/raindrops-public/20170316031652.17433-1-e@80x24.org/T/
Commits on Mar 15, 2017
  1. Merge remote-tracking branch 'origin/ccc-tcp-v3'

    Eric Wong committed Mar 15, 2017
    * origin/ccc-tcp-v3:
      http_request: reduce insn size for check_client_connection
      support "struct tcp_info" on non-Linux and Ruby 2.2+
      revert signature change to HttpServer#process_client
      new test for check_client_connection
      check_client_connection: use tcp state on linux
Commits on Mar 14, 2017
  1. doc: fix links to raindrops project

    Eric Wong committed Mar 14, 2017
    bogomips.org is dropping prefixes to reduce subjectAltName bloat
    in TLS certificates.
  2. test_exec: SO_KEEPALIVE value only needs to be true

    Eric Wong committed Mar 10, 2017
    On FreeBSD 10.3, the value of SO_KEEPALIVE returned by
    getsockopt is 8, even when set to '1' via setsockopt.
    Relax the test to only ensure the boolean value is
    interpreted as "true".
    
    Verified independently of Ruby using the following:
    --------8<---------
        #include <sys/types.h>
        #include <sys/socket.h>
        #include <stdio.h>
    
    static int err(const char *msg)
    {
    	perror(msg);
    	return 1;
    }
    
    int main(void)
    {
    	int sv[2];
    	int set = 1;
    	int got;
    	socklen_t len = (socklen_t)sizeof(int);
    	int rc;
    
    	rc = socketpair(PF_LOCAL, SOCK_STREAM, 0, sv);
    	if (rc) return err("socketpair failed");
    
    	rc = setsockopt(sv[0], SOL_SOCKET, SO_KEEPALIVE, &set, len);
    	if (rc) return err("setsockopt failed");
    
    	rc = getsockopt(sv[0], SOL_SOCKET, SO_KEEPALIVE, &got, &len);
    	if (rc) return err("getsockopt failed");
    
    	printf("got: %d\n", got);
    	return 0;
    }
  3. tests: keep disabled tests defined

    Eric Wong committed Mar 10, 2017
    Some versions of test-unit will fail if an unspecified test is
    attempted via "-n", so we need to define an empty test.
    
    We cannot use "skip", either, as that seems exclusive to
    minitest; and we won't use minitest since it has more
    incompatible changes than test-unit over the last 8 years.
    
    The memory leak test is gone since we're more versed in the
    Ruby C API nowadays, modern GCs + mallocs may be less
    predictable about releasing memory back to the OS.
  4. test-lib: expr(1) portability fix

    Eric Wong committed Mar 10, 2017
    GNU expr supports '+' to match one or more occurrences, but
    it seems the expr(1) on my FreeBSD installation does not.
  5. freebsd: avoid EINVAL when setting accept filter

    Eric Wong committed Mar 10, 2017
    Accept filters can only be set on listen sockets, and it also
    fails with EINVAL if it's already set.
    
    Untested, but I suppose changing the accept filter on a listening
    socket is not supported, either; since that could affect in-flight
    sockets.
  6. http_request: reduce insn size for check_client_connection

    Eric Wong committed Mar 14, 2017
    Unlike constants and instance variables, class variable access
    is not optimized in the mainline Ruby VM.  Use a constant
    instead, to take advantage of inline constant caching.
    
    This further reduces runtime instruction size by avoiding a
    branch by allocating the Raindrops::TCP_Info object up front.
    
    This reduces the method size by roughly 300 bytes on 64-bit.
Commits on Mar 13, 2017
  1. Don't pass a block for fork when forking workers

    jeremyevans committed with Eric Wong Mar 13, 2017
    This reduces the stack depth, making GC more efficient.
Commits on Mar 10, 2017
  1. Add worker_exec configuration option

    jeremyevans committed with Eric Wong Mar 8, 2017
    The worker_exec configuration option makes all worker processes
    exec after forking.  This initializes the worker processes with
    separate memory layouts, defeating address space discovery
    attacks on operating systems supporting address space layout
    randomization, such as Linux, MacOS X, NetBSD, OpenBSD, and
    Solaris.
    
    Support for execing workers is very similar to support for reexecing
    the master process.  The main difference is the worker's to_i and
    master pipes also need to be inherited after worker exec just as the
    listening sockets need to be inherited after reexec.
    
    Because execing working is similar to reexecing the master, this
    extracts a couple of methods from reexec (listener_sockets and
    close_sockets_on_exec), so they can be reused in worker_spawn.
Commits on Mar 8, 2017
  1. oob_gc: rely on opt_aref_with optimization on Ruby 2.2+

    Eric Wong committed Mar 8, 2017
    Maybe oob_gc probably isn't heavily used anymore, maybe
    some Ruby 2.2+ users will benefit from this constant
    reduction.
    
    Followup-to: fb2f10e ("reduce constants and optimize for Ruby 2.2")
  2. unicorn_http: reduce rb_global_variable calls

    Eric Wong committed Feb 28, 2017
    rb_global_variable registers the address of the variable which
    refers to the object, instead of the object itself.  This adds
    extra overhead to each global variable for our case, where the
    variable is frozen and never changed.
    
    Given there are currently 59 elements in this array, this saves
    58 singly-linked list entries and associated malloc calls and
    associated overhead in the current mainline Ruby 2.x
    implementation.  On 64-bit GNU libc malloc, this is already
    16 * 58 = 928 bytes; more than the extra object slot and array
    slack space used by the new mark array.
    
    Mainline Ruby 1.9+ currently has a rb_gc_register_mark_object
    public function which would suite our needs, too, but it is
    currently undocumented, and may not be available in the future.
  3. Merge remote-tracking branch 'origin/chroot'

    Eric Wong committed Mar 8, 2017
    * origin/chroot:
      Add after_worker_ready configuration option
      Add support for chroot to Worker#user
  4. support "struct tcp_info" on non-Linux and Ruby 2.2+

    Eric Wong committed Mar 8, 2017
    Ruby 2.2+ can show "struct tcp_info" as a string via
    Socket::Option#inspect, and we can attempt to parse it
    out to extract the information we need.
    
    Parsing this string is inefficient, but does not depend on the
    ordering of the tcp_info struct.
  5. revert signature change to HttpServer#process_client

    Eric Wong committed Mar 8, 2017
    We can force kgio_tryaccept to return an internal class
    for TCP objects by subclassing Kgio::TCPServer.
    
    This avoids breakage in any unfortunate projects which depend on
    our undocumented internal APIs, such as gctools
    <https://github.com/tmm1/gctools>