Permalink
Commits on Nov 7, 2018
  1. Provide ChannelPipeline.remove methods with promises. (#651)

    Lukasa committed Nov 7, 2018
    Motivation:
    
    ChannelPipeline.remove0 contains a comment that indicates that users
    were expected to be able to use the ChannelHandlerContext when both
    handlerRemoved and the promise for channel removal call out.
    
    This works when invoking remove() from outside the event loop, but if
    a handler invokes remove on itself then it will only be able to attach
    a callback to the future *after* the callout occurs. This means that a
    ChannelHandler removing itself from the pipeline cannot rely on there
    being a moment when it can still invoke the ChannelHandlerContext, but
    when it is no longer a formal part of the pipeline.
    
    This kind of behaviour is useful in some unusual cases, and it would
    be nice to support it.
    
    Modifications:
    
    - Added remove() methods that accept promises as input.
    - Rewrote the current remove() methods in terms of the new ones.
    - Added tests.
    
    Result:
    
    ChannelHandlers can perform cleanup with a valid ChannelHandlerContext
    but outside the Channel.
Commits on Sep 28, 2018
  1. Rest between runs of tests that join multicast groups. (#625)

    Lukasa authored and normanmaurer committed Sep 28, 2018
    Motivation:
    
    On some machines we've seen tests that rapidly join and leave multicast groups
    hitting spurious test failures. These test failures manifest as hitting ENOMEM on
    setsockopt(IP_ADD_MEMBERSHIP). The reason appears to be that the IGMP join/leave
    messages that the kernel needs to emit for these joins can be queued up waiting for
    a timer to fire, and if we move sufficiently quickly we can push too many
    messages into that queue.
    
    This can be avoided by avoiding the need to emit the IGMP join/leave messages
    at all. We can do that by forcing the multicast joins onto the loopback interface,
    as there's no value in the kernel emitting join/leaves on the loopback. Given that
    for these tests the join is necessary only to get the kernel to have IGMP state,
    there's no reason not to do this.
    
    Modifications:
    
    Forced all multicast joins onto the loopback address.
    
    Result:
    
    No weird ENOMEM crashes in test runs.
Commits on Sep 25, 2018
  1. Work around SE-0213 (#623)

    Lukasa committed Sep 25, 2018
    Motivation:
    
    In Swift 5, types that conform to ExpressibleByIntegerLiteral will now
    have their init(integerLiteral:) initializer invoked whenever the
    compiler sees code of the form `Type(integerLiteral)`, rather than
    before where the integer literal would be created as `Int` and then
    an alternative initializer would be called. This broke our tests, so
    we should fix it now that we're aware of it.
    
    Modifications:
    
    Force the type of the argument for the one broken test line, add new
    test lines that execute this code path.
    
    Result:
    
    Better testing, fewer compile errors.
Commits on Sep 24, 2018
  1. Support UDP multicast. (#618)

    Lukasa committed Sep 24, 2018
    Motivation:
    
    A large number of very useful protocols are implemented using multicast
    with UDP. As a result, it would be helpful to add support for joining and
    leaving IP multicast groups using SwiftNIO.
    
    Modifications:
    
    - Defines a MulticastChannel protocol for channels that support joining and
      leaving multicast groups.
    - Adds an implementation of MulticastChannel to DatagramChannel.
    - Adds a interfaceIndex property to NIONetworkInterface.
    - Adds if_nametoindex to the Posix enum.
    - Adds a isMulticast computed property to SocketAddress
    - Adds a demo multicast chat application.
    - Add a number of multicast-related socket options to SocketOptionProvider.
    
    Result:
    
    NIO users will be able to write channels that handle multicast UDP.
Commits on Sep 18, 2018
  1. Add SocketOptionChannel for wider socket options. (#589)

    Lukasa authored and weissi committed Sep 18, 2018
    Motivation:
    
    A small number of socket options have values that do not fit into a C
    int type. Our current ChannelOption based approach for setting these
    simply does not work, and cannot be extended to support the truly arbitrary
    types that the setsockopt/getsockopt functions allow here.
    
    This makes it impossible to use some socket options, which is hardly a
    good place to be.
    
    There were a number of ways we could have addressed this: we could have
    special-cased all socket options with non-integer types in ChannelOption,
    but I believe that would be too manual, and risk limiting users that need
    to set other socket options. We could also have added a ChannelOption
    that simply allows users to pass a buffer to write into or read from,
    but that is a very un-Swift-like API that feels pretty gross to hold.
    
    Ultimately, the nicest seemed to be a new protocol users could check
    for, and that would provide APIs that let users hold the correct concrete
    type. As with setsockopt/getsockopt, while this API is typed it is
    not type-safe: ultimately, the struct we have here is treated just as a
    buffer by setsockopt/getsockopt. We do not attempt to prevent users from
    shooting themselves in the foot here.
    
    This PR does not include an example use case in any server, as I will
    provide such an example in a subsequent multicast PR.
    
    Modifications:
    
    - Added a SocketOptionChannel protocol.
    - Conformed BaseSocketChannel to SocketOptionChannel.
    - Wrote some tests for this.
    
    Result:
    
    Users can set and get sockopts with data that is larger than a C int.
Commits on Sep 10, 2018
  1. Actually report errors when hitting blacklisted errnos. (#610)

    Lukasa authored and weissi committed Sep 10, 2018
    Motivation:
    
    It's not all that useful to print the value of the pointer from sterror.
    Instead we should aim to print the string. While we're here, let's avoid
    repeating too much code.
    
    Modifications:
    
    - Centralised the handling of blacklisted errnos.
    - Correctly printed the error string when a blacklisted errno is hit.
    - Updated the output of the assertion to include the function failing.
    
    Result:
    
    Better diagnostics when blacklisted errnos are hit.
Commits on Aug 14, 2018
  1. Add Cocoapods podspec generation. (#568)

    Lukasa authored and weissi committed Aug 14, 2018
    Motivation:
    
    A number of users have asked for the ability to install SwiftNIO
    via Cocoapods. This is an entirely reasonable request, so let's
    do it.
    
    Modifications:
    
    - Added a script that can generate, and optionally upload,
      podspecs.
    - Made a slight modification to the dependency script to provide
      target-specific dependencies.
    
    Result:
    
    Users can use cocoapods if they prefer.
Commits on Aug 10, 2018
  1. Avoid implicit conversion. (#566)

    Lukasa committed Aug 10, 2018
    Motivation:
    
    We were inadvertently doing an implicit integer conversion from
    ssize_t to unsigned int in our sendmmsg shim. This can sometimes
    generate warnings. The conversion itself is safe.
    
    Modifications:
    
    Make the conversion explicit.
    
    Result:
    
    No warnings.
Commits on Aug 9, 2018
Commits on Aug 7, 2018
  1. Make withUnsafeMutable{Readable,Writable}Bytes inlinable. (#558)

    Lukasa committed Aug 7, 2018
    Motivation:
    
    I spotted this while flamegraphing: apparently we missed the
    declaration for inlineability here.
    
    Modifications:
    
    Made these two functions inlineable.
    
    Result:
    
    Better optimizations.
Commits on Aug 2, 2018
  1. Make bad websocket parser states unrepresentable. (#547)

    Lukasa committed Aug 2, 2018
    Motivation:
    
    Apparently when I wrote the WebSocket parser I forgot that enums are
    great, and so I added a bunch of optional properties. That was silly.
    This patch changes the WSParser structure to use an enum with
    associated data to ensure that we only store state when we are supposed
    to, and to guarantee that the state is good.
    
    Modifications:
    
    Move all state to enum case associated data.
    
    Result:
    
    Easier to validate the correctness of the WSParser code.
Commits on Aug 1, 2018
  1. Deprecate and replace changeCapacity. (#539)

    Lukasa committed Aug 1, 2018
    Motivation:
    
    As described in #537, ByteBuffer.changeCapacity is not a very good
    function. We want to replace it with a better one, more similar to
    reserveCapacity on the standard library data structures, and ideally
    substantially faster.
    
    Modifications:
    
    - Deprecated ByteBuffer.changeCapacity.
    - Added ByteBuffer.reserveCapacity.
    - Changed the implementation of changeCapacity to call reserveCapacity
      when we're increasing the capacity.
    
    Result:
    
    Better APIs, better performance in the legacy ones, everything is
    better.
Commits on Jul 31, 2018
  1. More featureful error handling options for WebSockets (#528)

    Lukasa committed Jul 31, 2018
    Motivation:
    
    Currently if you hit a parse error in the WebSocketFrameDecoder, that
    handler takes it upon itself to return the error code and shut the connection
    down. That's not really its job: it's not a pattern that we use anywhere
    else, and it prevents users from overriding this handling with their own
    choices.
    
    We should stop doing that. Unfortunately, we can't totally stop it because
    it's a supported feature in the current release of NIO, but we can give
    users the option to opt-out, and also build our future solution at the same
    time.
    
    Modifications:
    
    1. Added support for disabling the automatic error handling in
        WebSocketFrameDecoder.
    2. Created a new WebSocketProtocolErrorHandler class that will be used for
        default error handling in 2.0.
    3. Added support to the WebSocketUpgrader to disable automatic error handling
        altogether.
    4. Changed the WebSocketUpgrader to use the WebSocketProtocolErrorHandler
        for default error handling when necessary.
    
    Result:
    
    Better separation of concerns, and users can override NIO's default
    error handling behaviour.
Commits on Jul 26, 2018
  1. Avoid hot-looping when autoRead is disabled. (#526)

    Lukasa committed Jul 26, 2018
    Motivation:
    
    On macOS it is possible to hot-loop on kevent when a channel has autoRead
    disabled but pending data in the receive buffer. This is because
    EVFILT_EXCEPT will fire when there is pending data in the receive buffer.
    
    For now we work around this by setting NOTE_LOWAT to Int.max, which will
    ensure that we don't actually get EVFILT_EXCEPT fired for pending data.
    
    Modifications:
    
    - When registering EVFILT_EXCEPT, set NOTE_LOWAT to Int.max.
    - Fix a test that definitely didn't test what it wanted to.
    
    Result:
    
    No hot-loops with pending data.
Commits on Jul 12, 2018
  1. Cleanup wrapErrorIsNullReturnCall. (#511)

    Lukasa committed Jul 12, 2018
    Motivation:
    
    Given that wrapErrorIsNullReturnCall exists purely to wrap syscalls that
    return NULL pointers on errors, there is no point allowing the return
    value to be nil. Additionally, it's currently weirdly specific about the
    return pointer type, so we can make the function generic.
    
    Modifications:
    
    - Make the return value of wrapErrorIsNullReturnCall generic.
    - Make the return value of wrapErrorIsNullReturnCall non-optional.
    
    Result:
    
    Easier to use wrapErrorIsNullReturnCall and any matching function
Commits on Jun 10, 2018
  1. Update http_parser (#471)

    Lukasa authored and normanmaurer committed Jun 10, 2018
    Motivation:
    
    Our copy of http_parser has become almost a year old now, and there are
    both some bugfixes that improve correctness and some performance
    improvements in http_parser that we should pick up.
    
    Modifications:
    
    - Updated http_parser
    - Added tests to confirm we don't suffer from
        nodejs/http-parser#432
    - Added tests that we didn't get broken by SOURCE verb addition.
    
    Result:
    
    Shiny new http_parser!
Commits on May 30, 2018
  1. Widen assertion on readEOF. (#453)

    Lukasa committed May 30, 2018
    Motivation:
    
    In Darwin it is possible for readEOF to be received before writable,
    if we race connection teardown with registering for writable. See
    #452 for a more in-depth
    explanation.
    
    Modifications:
    
    Widened the assertion.
    
    Result:
    
    People's code won't go bang.
Commits on May 24, 2018
  1. Don't provide promises when we don't care about results. (#437)

    Lukasa authored and weissi committed May 24, 2018
    Motivation:
    
    When we don't care about the result of an operation, we shouldn't provide
    a promise.
    
    While I'm here I also removed an unnecessary close() call in the
    AcceptHandler: the close() would be called again almost immediately.
    
    Modifications:
    
    Removed a close() call.
    Set `promise: nil` on a pair of close() calls.
    
    Result:
    
    Fewer promise allocations.
Commits on May 23, 2018
  1. Correctly account for writes of all datagrams. (#431)

    Lukasa committed May 23, 2018
    Motivation:
    
    The Darwin gathering datagram writes shim incorrectly placed the
    "bytes written" amount into the first slot in the array for every write.
    This had the effect of incorrectly accounting for the amount of bytes
    written for literally every element of a vector write. Not good.
    
    Modifications:
    
    Correctly account for each vector write element.
    
    Result:
    
    Lower risk of preconditions in incorrect situations, better accounting
    of written bytes.
Commits on May 19, 2018
  1. Link to swift-nio-http2. (#416)

    Lukasa authored and weissi committed May 19, 2018
Commits on May 18, 2018
  1. Add ChannelCore.removeHandlers. (#408)

    Lukasa authored and normanmaurer committed May 18, 2018
    Motivation:
    
    If ChannelPipeline.removeHandlers is internal it is extremely difficult
    to implement a correct ChannelCore. For that reason, we should make it
    available as an extension on ChannelCore.
    
    This is a minor layering violation, but that's ok: ChannelCore and
    Channel are not actually intended to be separate objects in most cases,
    they are just separate for code clarity reasons.
    
    Modifications:
    
    Add ChannelCore.removeHandlers as a public method.
    
    Result:
    
    Easier to implement ChannelCore.
Commits on May 8, 2018
  1. Make hopTo(eventLoop:) public. (#395)

    Lukasa committed May 8, 2018
    Motivation:
    
    When this method was originally introduced it was also intended as a
    helper for external users, but we never made it public.
    
    Modifications:
    
    Make hopTo(eventLoop:) public.
    
    Result:
    
    Users will have an easier time managing their threads.
Commits on May 1, 2018
  1. Track Future allocations in CI. (#376)

    Lukasa authored and weissi committed May 1, 2018
    Motivation:
    
    We don't want to regress the number of heap allocations we perform.
    
    Modifications:
    
    Added appropriate environment variables to Docker files.
    
    Result:
    
    Things go bang if we regress Future allocation counts.
  2. Add UnsafeEmbeddedAtomic and use it. (#373)

    Lukasa authored and normanmaurer committed May 1, 2018
    Motivation:
    
    Many uses of atomics restrict themselves to using atomic variables within
    objects that already have reference semantics (i.e. classes) and have
    appropriate lifetime management. In this case, the current NIO Atomic
    performs an unnecessary heap allocation in order to get itself a reference
    count that it can use to free the underlying heap memory.
    
    This is an unnecessary allocation that can often be elided, especially in
    situations where many such atomics may be needed. Thus, we should provide
    the ability to manually manage the lifetime of the heap-allocated memory for
    the atomic, and change some uses of it in our codebase to reduce overhead.
    
    Modifications:
    
    1. Moved most of the code of Atomic to a new struct, UnsafeEmbeddedAtomic.
    2. Reimplemented Atomic in terms of UnsafeEmbeddedAtomic.
    3. Updated EventLoopFuture to use an UnsafeEmbeddedAtomic for isFulfilled.
    
    Result:
    
    1/3 as many allocations per EventLoopFuture.
Commits on Apr 27, 2018
  1. Avoid delivering websocket frames twice on EOF. (#368)

    Lukasa committed Apr 27, 2018
    Motivation:
    
    Unfortunately as a result of #108 the WebSocketFrameDecoder can emit a
    WebSocketFrame more than once if the user closes the connection in
    channelRead, or any other callback while decode() is on the call stack.
    This is obviously less than ideal, as it can allow multiple delivery of
    frames.
    
    Modifications:
    
    Given WebSocketFrameDecoder a no-op implementation of decodeLast.
    
    Result:
    
    No multi-delivery of frames.
Commits on Apr 23, 2018
  1. Prevent spurious test failure in testShuttingDownFailsRegistration (#338

    Lukasa committed Apr 23, 2018
    )
    
    Motivation:
    
    There were two errors in testShuttingDownFailsRegistration that
    could cause spurious test failures.
    
    The first was that the test did not wait for connection establishment
    before it began shutting the selector down, due to a change made in
    
    The second was that the server side of the connection was not wedged open,
    and so it would be closed by the call to loop.closeGently. That closure
    could, if the main thread took long enough, lead to the client side
    reading EOF and being forcibly closed.
    
    Modifications:
    
    Added a promise to wait for connection establishment.
    Added support for wedging both the server and client side open.
    
    Result:
    
    Fewer test failures.
Commits on Apr 18, 2018
  1. Make sure we create correct socket family. (#334)

    Lukasa authored and normanmaurer committed Apr 18, 2018
    Motivation:
    
    If a server binds "localhost", and then we try to connect with an
    AF_INET socket, then on macOS at least some of the time we will have
    a protocol mismatch, as at least some of the time localhost will have
    resolved to ::1 before 127.0.0.1.
    
    That's bad.
    
    Modifications:
    
    Create a socket whose address family matches the server's.
    
    Result:
    
    This test will pass more often
  2. Synchronous connection failures must close channels. (#329)

    Lukasa authored and weissi committed Apr 18, 2018
    Motivation:
    
    When a connect() call returns an error other than EINPROGRESS, we
    currently leave the FD registered and don't close the channel. This is
    wrong, we should take this as a signal to close the socket immediately,
    rathern than letting the selector tell us this FD is dead: after all,
    we know it's dead.
    
    Modifications:
    
    Call `close0()` in synchronous connect failures.
    
    Result:
    
    Faster failures!
  3. Allow ChannelCore implementors to unwrap NIOAny. (#321)

    Lukasa committed Apr 18, 2018
    Motivation:
    
    When implementing a custom ChannelCore, you will probably need access
    to the data inside a NIOAny. It should be possible to unwrap that.
    
    Modifications:
    
    Added an extension to ChannelCore to allow unwrapping a NIOAny.
    Added @_versioned to almost all of NIOAny.
    
    Result:
    
    ChannelCore is implementable outside NIO
Commits on Apr 12, 2018
  1. Fix up HTTP message framing edge cases. (#298)

    Lukasa committed Apr 12, 2018
    Motivation:
    
    HTTP message framing has a number of edge cases that NIO currently does
    not tolerate. We should decide what our position is on each of these edge
    cases and handle it appropriately.
    
    Modifications:
    
    Provide an extensive test suite that codifies our expected handling of
    these edge cases. Fix divergences from this behaviour.
    
    Result:
    
    Better tolerance for the weird corners of HTTP.
  2. Don't call to user code before reconciling Channel state. (#310)

    Lukasa committed Apr 12, 2018
    Motivation:
    
    Callouts to user code allows the user to make calls that re-enter
    channel code. In the case of channel closure, we could call out to the
    user before the channel knew it was completely closed, which would
    trigger a safety assertion (in debug mode) or hit a fatalError
    (in release mode).
    
    Modifications:
    
    Reconciled channel state before we call out to user code.
    Added a test for this case.
    
    Result:
    
    Fewer crashes, better channel state management.
Commits on Apr 11, 2018
  1. Restore unary initializer for HTTPHeaders (#301)

    Lukasa committed Apr 11, 2018
    Motivation:
    
    The Swift compiler seems to get very nervous when variadic inits
    are used for types that have constructors with optional values. In
    general we're not worried about breaking downstream consumers'
    extensions when we update our code, but in this case we break the
    ability to conform HTTPHeaders to ExpressibleByDictionaryLiteral,
    which is a bit annoying. See https://bugs.swift.org/browse/SR-7415
    for more details.
    
    For 1.5.0 we should conform HTTPHeaders ourselves, but until that
    time we can restore anyone who was conforming HTTPHeaders to
    ExpressibleByDictionaryLiteral by restoring the old unary initializer
    and delegating it to the new one. This presents an effect that is
    equivalent to the old behaviour, but is new.
    
    As a side note, in general it's a bad idea to conform types that you
    do not own to standard library protocols. NIO reserves the right to
    add conformances to our types in any Semver Minor release, so having
    that conformance in your own code risks breakage without a Semver
    Major patch bump. Please be aware.
    
    Modifications:
    
    Restored the unary initializer for HTTPHeaders.
    
    Result:
    
    Less breakage, more happiness.
Commits on Apr 10, 2018
  1. Enable our one disabled integration test. (#295)

    Lukasa authored and weissi committed Apr 10, 2018
    Motivation:
    
    We have one integration test left disabled, but actually it basically
    already passes: it just expects the wrong format of error. We should
    have all our integration tests enabled.
    
    Modifications:
    
    Enabled the test.
    Changed the wording of the error it expected.
    
    Result:
    
    No disabled integration tests.
Commits on Apr 7, 2018
  1. Don't race waiting for a nil reference. (#289)

    Lukasa authored and normanmaurer committed Apr 7, 2018
    Motivation:
    
    Unfortunately we cannot guarantee that a weak ref is going to be nil in
    a timely fashion if a strong reference to the object exists in another
    thread. This makes us fail tests on slow machines, which is bad.
    
    Modifications:
    
    Wait for up to a second for the other thread to get its house in order
    before we give up.
    
    Result:
    
    Hopefully I'll never have to "fix" this test again.
  2. Add support for automatic HTTP error reporting. (#268)

    Lukasa authored and normanmaurer committed Apr 7, 2018
    Motivation:
    
    Currently the HTTP decoders can throw errors, but they will be ignored
    and lead to a simple EOF. That's not ideal: in most cases we should make
    a best-effort attempt to send a 4XX error code before we shut the client
    down.
    
    Modifications:
    
    Provided a new ChannelHandler that generates 400 errors when the HTTP
    decoder fails.
    Added a flag to automatically add that handler to the channel pipeline.
    Added the handler to the HTTP sample server.
    Enabled integration test 12.
    
    Result:
    
    Easier error handling for HTTP servers.