Skip to content

Commit

Permalink
Documented extra pruning transitions
Browse files Browse the repository at this point in the history
  • Loading branch information
bolt12 committed Sep 28, 2021
1 parent 4c0a2e7 commit dd3c70b
Showing 1 changed file with 37 additions and 7 deletions.
44 changes: 37 additions & 7 deletions docs/network-spec/connection-manager.tex
Expand Up @@ -933,6 +933,7 @@ \subsubsection{\Prune{} transitions}
accepted any connection, it could end up serving peers and possibly go beyond
server hard limit, thus exceeding the number of file descriptors. This is
possible via the following path:

\begin{itemize}
\item[] \Connected{},
\item[] \NegotiatedDupOut{},
Expand All @@ -945,16 +946,42 @@ \subsubsection{\Prune{} transitions}
rate limits connections based on how many connections are in this state, we
could end up exceeding server hard limit.

These are all transitions that can lead to exceeding server hard limit:
\begin{itemize}
\item \DuplexState{} to \InboundStateDup{} (via \DemotedToColdDupLoc{})
\item \OutboundStateDupTau{} to \InboundStateDup{} (via \DemotedToColdDupLoc{})
\item \OutboundStateDupTau{} to \DuplexState{} (via \PromotedToWarmDupRem{})
\item \OutboundStateDup{} to \DuplexState{} (via \PromotedToWarmDupRem{})
\end{itemize}

To solve this problem, when a connection is transitioned from
\DuplexState{} to \InboundStateDup{} (via \DemotedToColdDupLoc{}) the
\connmngr{} will check if the server hard limit was exceeded. If that
\DuplexState{} to \InboundStateDup{} (via \DemotedToColdDupLoc{}), for example,
the \connmngr{} will check if the server hard limit was exceeded. If that
happened, the \connmngr{} will reset an arbitrary connection (with some preference).

We prefer to reset an inbound connection rather than close an outbound
The reason why going from \OutboundStateDupTau{} (or \OutboundStateDup{}) to
\InboundStateDup{} might exceed the server hard limit is exacty the same as the
\DuplexState{} to \InboundStateDup{} one. However, the reason why going from
\OutboundStateDupTau{} to \DuplexState{} might exceed the hard limit is more tricky.
To reach a \DuplexState{} one assumes there must have been an incoming
\textit{accepted} connection, but there's another way that two end-points
can establish a connection without a node accepting it. If two nodes try to request an
outbound connection simultaneously, it is possible, for two applications to both perform
an active open to each other at the same time. This is called a
\href{https://flylib.com/books/en/3.223.1.190/1/}{\textit{simultaneous open}}.
In a simultaneous TCP open, we can have 2 nodes establishing a connection without any of
them having explicitly accepted a connection, which can make a server serving more peers
than the hard limit.

Given this, we prefer to reset an inbound connection rather than close an outbound
connection because from a systemic point of view, outbound connections are more
valuable than inbound ones. If we keep the number of \established{} peers to
be smaller than the server hard limit, with a right policy we should never need
to reset a connection in \DuplexState{}.
to reset a connection in \DuplexState{}. However, when dealing with a connection that
transitions from \OutboundStateDupTau{} to \DuplexState{}, we actually need to
make sure this connection is closed, because we have no way to know for sure
if this connection is the result of a TCP simultaneous open and there might
not be any other connection available to prune that can make space for this one.

The \textit{inbound protocol governor} is in position to make an educated
decision about which connection to reset. Initially, we aim for a decision driven by
Expand Down Expand Up @@ -1229,7 +1256,10 @@ \subsubsection{\OutboundStateDup{} and \DuplexState{}}
This means that the \connmngr{} needs a way to inform server (which
accepts and monitors inbound connections), to start the protocols and monitor
that connection. This connection will transition to \DuplexState{} only once
we notice incoming traffic on any of \established{} protocols.
we notice incoming traffic on any of \established{} protocols. Since this connection might
have been established via TCP simultaneous open, this transition to \DuplexState{} can
also trigger \Prune{} transitions if the number of inbound connections becomes above
the limit.

\begin{detail}
The implementation is using a \texttt{TBQueue}. Server is using this channel
Expand Down Expand Up @@ -1313,7 +1343,7 @@ \subsubsection{Connection manager methods}
\UnnegotiatedStateOut{} & error \texttt{ForbiddenOperation} \\[8pt]
\UnnegotiatedStateIn{} & error \texttt{ForbiddenOperation} \\[8pt]
\OutboundStateAny{} & \DemotedToColdAnyLoc{} \\[8pt]
\OutboundStateDupTau{} & \DemotedToColdDupLoc{} \\[8pt]
\OutboundStateDupTau{} & \Prune{} or \DemotedToColdDupLoc{} \\[8pt]
\OutboundIdleStateAny{} & \texttt{no-op} \\[8pt]
\InboundIdleStateUni{} & assertion error \\[8pt]
\InboundIdleStateDup{} & \texttt{no-op} \\[8pt]
Expand Down Expand Up @@ -1423,7 +1453,7 @@ \subsubsection{Connection manager methods}
\ReservedOutboundState{} & - & \\[8pt]
\UnnegotiatedStateAny{} & - & \\[8pt]
\OutboundStateUni{} & - & \\[8pt]
\OutboundStateDup{} & \DuplexState{} & \PromotedToWarmDupRem{} \\[8pt]
\OutboundStateDup{} & \Prune{} or (\DuplexState{} & \PromotedToWarmDupRem{}) \\[8pt]
\InboundIdleStateUni{} & \InboundStateUni{} & \AwakeUniRem{} \\[8pt]
\InboundIdleStateDup{} & \InboundStateDup{} & \AwakeDupRem{} \\[8pt]
\InboundStateUni{} & - & \\[8pt]
Expand Down

0 comments on commit dd3c70b

Please sign in to comment.