Skip to content

Commit

Permalink
inject-server: Add TwitterServer#setup lifecycle method for PUBSUB
Browse files Browse the repository at this point in the history 
Summary: Problem/Solution

For services which need to create and await on sometype of Publisher or
Subscriber is it not clear where in the lifecycle this should happen. There
are currently two places we recommend, `postInjectorStartup` and the more
common, `postWarmup`. These both come with the caveat that users need
to remember to call the `super` version in both cases and their names are
very non-obvious for the use case.

Introduce `TwitterServer#setup` which is called in the `postInjectorStartup`
lifecycle method with the specific intention of being the correct place
in the server lifecycle where users can "setup" publishers and subscribers.

JIRA Issues: CSL-5966

Differential Revision: https://phabricator.twitter.biz/D135827
  • Loading branch information
@cacoco
cacoco authored and jenkins committed Feb 24, 2018
1 parent 4baf6c9 commit e35f287
Show file tree
Hide file tree
Showing 6 changed files with 462 additions and 144 deletions.
7 changes: 7 additions & 0 deletions CHANGELOG.md
View file
Expand Up @@ -6,6 +6,13 @@ All notable changes to this project will be documented in this file. Note that `

### Added

* inject-server: Add `c.t.inject.server.TwitterServer#setup` lifecycle callback method. This is
run at the end of the `postInjectorStartup` phase as is primarily intended as a way for
servers to start pub-sub components on which the server depends. User should prefer this method
over overidding the `c.t.inject.server.TwitterServer#postWarmup` @Lifecycle-annotated method as
the callback does not require a call its super implementation for the server to correctly start
and is ideally less error-prone to use. ``PHAB_ID=D135827``

* inject-app: Add `c.t.inject.annotations.Flags#named` for getting an implementation of an `@Flag`
annotation. This is useful when trying to get or bind an instance of an `@Flag` annotated type.
``PHAB_ID=D140831``
Expand Down
202 changes: 161 additions & 41 deletions doc/src/sphinx/user-guide/getting-started/lifecycle.rst
View file
Expand Up @@ -9,10 +9,12 @@ and provides methods which can be implemented for running or starting core logic

This is done for several reasons:

- To ensure that flag parsing and module installation to build the object graph is done in the correct
order such that the injector is properly configured for use before a user attempts attempts to access flags.
- To ensure that `flag <./flags.html>`__ parsing and `module <./modules.html>`__ installation to
build the object graph is done in the correct order such that the injector is properly configured
before a user attempts attempts to access flags.

- Ensure that object promotion and garbage collection is properly handled *before* accepting traffic to a server.
- Ensure that object promotion and garbage collection is properly handled *before* accepting traffic
to a server.

- Expose any external interface *before* reporting a server is "healthy". Otherwise a server may
report it is healthy before binding to a port — which may fail. Depending on how monitoring is
Expand All @@ -21,16 +23,19 @@ This is done for several reasons:
on some frequency) it could be some interval before the server is recognized as unhealthy when in
fact it did not start properly as it could not bind to a port.

Thus you do not have access to the `app <https://github.com/twitter/util/blob/9fa550a269d2287b24e94921a352ba954f9f4bfb/util-app/src/main/scala/com/twitter/app/App.scala#L24>`__
or `server <https://twitter.github.io/twitter-server/#getting-started>`__ `main` method. Instead, any
logic should be contained in overriding an ``@Lifecycle``-annotated method or in the app or server
callbacks |c.t.inject.app.App#run|_ or |c.t.inject.server.TwitterServer#start|_ methods, respectively.
Thus you do not have access to the `App <https://github.com/twitter/util/blob/9fa550a269d2287b24e94921a352ba954f9f4bfb/util-app/src/main/scala/com/twitter/app/App.scala#L24>`__
or `TwitterServer <https://github.com/twitter/twitter-server/blob/5fea9c2a6220ab9bbdb449c99c946e2aef322e7d/server/src/main/scala/com/twitter/server/TwitterServer.scala#L93>`__
`main` method. Instead, any logic should be contained in overriding an ``@Lifecycle``-annotated
method or in the application or server callbacks.

.. caution::

If you override an ``@Lifecycle``-annotated method you **MUST** first call `super()` in the method to ensure that framework lifecycle events happen accordingly.
If you override an ``@Lifecycle``-annotated method you **MUST** first call
`super.lifecycleMethod()` in your override to ensure that framework lifecycle events happen
accordingly.

With all lifecycle methods, it is important to **not performing any blocking operations** as you will prevent the server from starting.
See the `Creating an injectable App <../app/index.html>`__ and
`Creating an injectable TwitterServer <../twitter-server/index.html>`__ sections for more information.

Startup
-------
Expand All @@ -42,60 +47,175 @@ At a high-level, the start-up lifecycle of a Finatra server looks like:
Shutdown
--------

Upon *graceful* shutdown, all registered `onExit {...}` blocks are executed. See `c.t.app.App#exits <https://github.com/twitter/util/blob/9fa550a269d2287b24e94921a352ba954f9f4bfb/util-app/src/main/scala/com/twitter/app/App.scala#L72>`__).
Upon *graceful* shutdown of an application or a server, all registered `onExit`, `closeOnExit`, and
`closeOnExitLast` blocks are executed. See
`c.t.app.App#exits <https://github.com/twitter/util/blob/9fa550a269d2287b24e94921a352ba954f9f4bfb/util-app/src/main/scala/com/twitter/app/App.scala#L72>`__
and `c.t.app.App#lastExits <https://github.com/twitter/util/blob/bf47b55ff45a31bbd541f66257f2244df5c35f5b/util-app/src/main/scala/com/twitter/app/App.scala#L86>`_.

This includes closing the `TwitterServer <https://github.com/twitter/twitter-server>`__ `HTTP Admin Interface <https://twitter.github.io/twitter-server/Features.html#admin-http-interface>`__, firing the `TwitterModuleLifecycle#singletonShutdown <https://github.com/twitter/finatra/blob/c6e4716f082c0c8790d06d9e1664aacbd0c3fede/inject/inject-core/src/main/scala/com/twitter/inject/TwitterModuleLifecycle.scala#L29>`__
on all installed modules, and for extensions of the `HttpServer <../http/server.html>`__ or `ThriftServer <../thrift/server.html>`__ traits closing any external interfaces.
For a server, this includes closing the `TwitterServer <https://github.com/twitter/twitter-server>`__
`HTTP Admin Interface <https://twitter.github.io/twitter-server/Features.html#admin-http-interface>`__
and shutting down and closing all installed modules. For extensions of the
`HttpServer <../http/server.html>`__ or `ThriftServer <../thrift/server.html>`__ traits this also
includes closing any external interfaces.

.. admonition:: Important

Note that the order of execution for all registered `onExit {...}` blocks is not guaranteed. Thus it is, up to implementors to enforce any desired ordering.
Note that the order of execution for all registered `onExit` and `closeOnExit` blocks is not
guaranteed as they are executed on graceful shutdown roughly in parallel. Thus it is up to
implementors to enforce any desired ordering.

For example, you have code which is reading from a queue (via a "listener"), transforming the data, and then publishing (via a "publisher") to another queue.
When main application is exiting you most likely want to close the "listener" first to ensure that you transform and publish all available data before closing the "publisher".
For example, you have code which is reading from a queue (via a "subscriber"), transforming the
data, and then publishing (via a "publisher") to another queue. When the main application is exiting
you most likely want to close the "subscriber" first to ensure that you transform and publish all
available data before closing the "publisher".

Assuming, that both objects are a |c.t.util.Closable|_ type, a simple way to close them would be:

Assuming, that the `close()` method of both returns `Future[Unit]`, e.g. like `c.t.util.Closable <https://github.com/twitter/util/blob/develop/util-core/src/main/scala/com/twitter/util/Closable.scala>`__,
a way of doing this could be:
.. code:: scala
.. code:: Scala
closeOnExit(subscriber)
closeOnExit(publisher)
onExit {
Await.result(listener.close())
Await.result(publisher.close())
}
However, the "subscriber" and the "publisher" would close roughly in parallel
which could lead to data inconsistencies in your server if the "subscriber" is still reading before
the "publisher" has closed.

In the example code above, we simply await on the close of the "listener" first and then the "publisher" thus ensuring that the "listener"
will close before the "publisher".
Ordering `onExit` and `closeOnExit` functions?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is instead of registering separate `onExit {...}` blocks for each:
Assuming, that the `#close()` method of both returns `Future[Unit]`, e.g. like a |c.t.util.Closable|_,
a way of doing this could be:

.. code:: Scala
.. code:: scala
onExit {
Await.result(listener.close())
}
onExit {
Await.result(subscriber.close(defaultCloseGracePeriod))
Await.result(publisher.close(defaultCloseGracePeriod))
}
onExit {
Await.result(publisher.close())
}
where the `defaultCloseGracePeriod` is the `c.t.app.App#defaultCloseGracePeriod <https://github.com/twitter/util/blob/bf47b55ff45a31bbd541f66257f2244df5c35f5b/util-app/src/main/scala/com/twitter/app/App.scala#L110>`__
function.

Which in this example, would possibly close the "publisher" before the "listener" meaning that the server would still be reading data but unable to publish it.
In the above example we simply await on the `#close()` of the "subscriber" first and then the
`#close()` of the "publisher" thus ensuring that the "subscriber" will close before the "publisher".

However, we are not providing a timeout to the `Await.result`, which we should ideally do as
well since we do not want to accidentally block our server shutdown if the `defaultCloseGracePeriod`
is set to something high or infinite (e.g., `Time.Top <https://github.com/twitter/util/blob/bf47b55ff45a31bbd541f66257f2244df5c35f5b/util-core/src/main/scala/com/twitter/util/Time.scala#L302>`__).

But if we don't know the configured value of the `defaultCloseGracePeriod` this makes things
complicated. We could just hardcode a value for the Await, or not use the `defaultCloseGracePeriod`:

.. code:: scala
onExit {
Await.result(subscriber.close(defaultCloseGracePeriod), 5.seconds)
Await.result(publisher.close(defaultCloseGracePeriod), 5.seconds)
}
...
onExit {
Await.result(subscriber.close(4.seconds), 5.seconds)
Await.result(publisher.close(4.seconds), 5.seconds)
}
However, this is obviously not ideal and there is an easier way. You can enforce the ordering of closing Closables
by using `closeOnExitLast`.

A |c.t.util.Closable|_ passed to `closeOnExitLast` will be closed *after* all `onExit` and
`closeOnExit` functions are executed. E.g.,

.. code:: scala
closeOnExit(subscriber)
closeOnExitLast(publisher)
In this code the "publisher" is guaranteed be closed **after** the "subscriber".

.. note:: All the exit functions: `onExit`, `closeOnExit`, and `closeOnExitLast` use the
`defaultCloseGracePeriod` as their close "deadline" and will raise a `TimeoutException` if
all the `exits` (collected `onExit`, `closeOnExit` functions) do not close within the deadline.
And if the `lastExits` (collected `closeOnExitLast` functions) do not close within the deadline.

If you have multiple |c.t.util.Closable|_ objects you want to close in parallel and one you want to
close after all the others, you could do:

.. code:: scala
closeOnExit(subscriberA)
closeOnExit(subscriberB)
closeOnExit(subscriberC)
closeOnExitLast(publisher)
The "publisher" is guaranteed be closed **after** the closing of "subscriberA", "subscriberB", and
"subscriberC".

What to do if you don't have a |c.t.util.Closable|_?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can simply use the `onExit` block to perform any shutdown logic, or you can wrap a function in
a |c.t.util.Closable|_ to be passed to `closeOnExit` or `closeOnExitLast`.

For example:

.. code:: scala
onExit {
DatabaseConnection.drain()
Await.result(someFutureOperation, 2.seconds)
}
closeOnExit {
Closable.make { deadline =>
prepWork.start()
anotherFutureOperation
}
}
closeOnExitLast {
Closable.make { deadline =>
queue.blockingStop(deadline)
Future.Unit
}
}
You can also wrap multiple functions in a Closable:

.. code:: scala
closeOnExit {
Closable.make { deadline =>
database.drain()
fileCleanUp.do()
pushData(deadline)
Future.Unit
}
}
Again the code in `onExit` and `closeOnExit` will be run in parallel and guaranteed to close
before the functions in `closeOnExitLast`.

.. note:: Multiple `closeOnExitLast` Closables will be closed in parallel with each other but
**after** all `onExit` and `closeOnExit` functions have closed.

Modules
-------

Modules provide hooks into the Lifecycle as well that allow instances being provided to the object graph to be plugged into the overall application or server lifecycle. See the `Module Lifecycle <../getting-started/modules.html#module-lifecycle>`__ section for more information.
Modules provide hooks into the Lifecycle as well that allow instances being provided to the object
graph to be plugged into the overall application or server lifecycle. See the
`Module Lifecycle <../getting-started/modules.html#module-lifecycle>`__ section for more information.

More Information
----------------

As noted in the diagram in the `Startup <#startup>`__ section there can be a non-trivial lifecycle especially in the case of a `TwitterServer <https://github.com/twitter/twitter-server>`__.
For more information on how to create an injectable `c.t.app.App <https://twitter.github.io/util/docs/com/twitter/app/App.html>`__ or a `c.t.server.TwitterServer <https://github.com/twitter/twitter-server/blob/develop/server/src/main/scala/com/twitter/server/TwitterServer.scala>`__
see the `Creating an injectable App <../app/index.html>`__ and `Creating an injectable TwitterServer <../twitter_server/index.html>`__ sections.
As noted in the diagram in the `Startup <#startup>`__ section the lifecycle or an application can be
non-trivial -- especially in the case of a `TwitterServer <https://github.com/twitter/twitter-server>`__.

.. |c.t.inject.app.App#run| replace:: ``c.t.inject.app.App#run``
.. _c.t.inject.app.App#run: ../app/index.html#app-run
For more information on how to create an injectable `c.t.app.App <https://twitter.github.io/util/docs/com/twitter/app/App.html>`__
or a `c.t.server.TwitterServer <https://github.com/twitter/twitter-server/blob/develop/server/src/main/scala/com/twitter/server/TwitterServer.scala>`__
see the `Creating an injectable App <../app/index.html>`__ and
`Creating an injectable TwitterServer <../twitter-server/index.html>`__ sections.

.. |c.t.inject.server.TwitterServer#start| replace:: ``c.t.inject.server.TwitterServer#start``
.. _c.t.inject.server.TwitterServer#start: ../twitter_server/index.html#twitterserver-start
.. |c.t.util.Closable| replace:: `c.t.util.Closable`
.. _c.t.util.Closable: https://github.com/twitter/util/blob/develop/util-core/src/main/scala/com/twitter/util/Closable.scala

0 comments on commit e35f287

Please sign in to comment.