WISH: Increase limit of maximum number of open connections (currently 125+3)

[HenrikBengtsson]

Background

As documents in help("connections", package="base") the maximum number of connections one can have open in R (in addition to the three always reserved) is 125;

"A maximum of 128 connections can be allocated (not necessarily open) at any one time. Three of these are pre-allocated (see stdout). The OS will impose limits on the numbers of connections of various types, but these are usually larger than 125."

Here is an example showing what happens when we try to open too many connections:

> cons <- list()
> for (ii in 1:126) { cons[[ii]] <- textConnection("foo") }
Error in textConnection("foo") : all connections are in use
> nrow(showConnections())
[1] 125
> head(showConnections())
  description class            mode text   isopen   can read can write
3 "\"foo\""   "textConnection" "r"  "text" "opened" "yes"    "no"     
4 "\"foo\""   "textConnection" "r"  "text" "opened" "yes"    "no"     
5 "\"foo\""   "textConnection" "r"  "text" "opened" "yes"    "no"     
6 "\"foo\""   "textConnection" "r"  "text" "opened" "yes"    "no"     
7 "\"foo\""   "textConnection" "r"  "text" "opened" "yes"    "no"     
8 "\"foo\""   "textConnection" "r"  "text" "opened" "yes"    "no"     
> tail(showConnections())
    description class            mode text   isopen   can read can write
122 "\"foo\""   "textConnection" "r"  "text" "opened" "yes"    "no"     
123 "\"foo\""   "textConnection" "r"  "text" "opened" "yes"    "no"     
124 "\"foo\""   "textConnection" "r"  "text" "opened" "yes"    "no"     
125 "\"foo\""   "textConnection" "r"  "text" "opened" "yes"    "no"     
126 "\"foo\""   "textConnection" "r"  "text" "opened" "yes"    "no"     
127 "\"foo\""   "textConnection" "r"  "text" "opened" "yes"    "no"     

Issue

There are several use cases where one might hit the upper limit of number of open connections possible in R. A common use case where one is may face this issue is when using SNOW compute clusters. SNOW clusters as implemented by the parallel package (a core R package) uses one connection per SNOW worker. These days more users have access to large clusters or machines with a large number of cores, making it more likely to try to use clusters with > 125 nodes.

> library("parallel")
> cl <- makeCluster(126L)
Error in socketConnection(port = port, server = TRUE, blocking = TRUE,  : 
  all connections are in use
> nrow(showConnections())
[1] 125

The problem with the low NCONNECTIONS limit in relationship to SNOW clusters has been discussed by others in the past, e.g.

• https://stat.ethz.ch/pipermail/r-devel/2004-March/029295.html
• https://stat.ethz.ch/pipermail/r-sig-hpc/2010-March/000590.html
• https://stat.ethz.ch/pipermail/r-sig-hpc/2012-May/001372.html
• http://digitheadslabnotebook.blogspot.com/2012/12/r-in-cloud.html
• https://stat.ethz.ch/pipermail/r-sig-hpc/2014-February/001818.html
• Bioconductor/BiocParallel#55

Troubleshooting

The total limit of 128 connections is hardcoded into the R source code as constant / macro NCONNECTIONS in src/main/connections.c;

#define NCONNECTIONS 128 /* snow needs one per slave node */

which is used to preallocate a set of Rconnection:s of this size;

static Rconnection Connections[NCONNECTIONS];

The NCONNECTIONS limit was increased from 50 to 128 in R 2.4.0 (released October 2006), which appears to have been done for the same reason as explained here.

Wish

• Increase the NCONNECTIONS limit, to say, 1024.
  • I've verified that it works with NCONNECTIONS=16384 on Linux (see comment below). Similar checks may have to be done on macOS and Windows as well.
  • This would only require a simple update of the above constant / macro.
  • The disadvantage of increasing the limit is that it will also increase the linear-search time of internal int ConnIndex(Rconnection con) for non-existing connections. Using a linked list would avoid this particular problem (see below).
• Make the error message informative about the actual limit, e.g. all 128 connections are in use.
• An alternative, and possibly better, approach would be to re-implement Connections as a linked list, which (including its memory usage) could grow and shrink as needed. This could even remove having a limit at all. This would require redesign of the code and increase the risk of introducing bugs. (This idea was proposed by @mtmorgan).

[HenrikBengtsson]

UPDATE: I've tested with #define NCONNECTIONS 16384 on 64-bit Linux (Ubuntu 16.04) and successfully:

• opened 16381 user-defined textConnection():s.
• opened 900 SNOW workers, i.e. cl <- makeCluster(900L). Didn't have the memory to try with a larger number (the workers run in separate R sessions each consuming ~200 MiB RAM).

[HenrikBengtsson]

A first quick mod that would allow us to modify NCONNECTIONS (via compiler flag -DNCONNECTIONS=1024) when building R from source would be:

svn diff src/main/connections.c 
Index: src/main/connections.c
===================================================================
--- src/main/connections.c	(revision 71879)
+++ src/main/connections.c	(working copy)
@@ -120,7 +120,9 @@
 # include <Startup.h>
 #endif
 
-#define NCONNECTIONS 128 /* snow needs one per slave node */
+#ifndef NCONNECTIONS
+# define NCONNECTIONS 128 /* snow needs one per slave node */
+#endif
 #define NSINKS 21
 
 static Rconnection Connections[NCONNECTIONS];

[romanzenka]

I ran into the exact same issue in a different context - we are writing software that very rapidly polls small subsets of a large amount of files and it is beneficial for us to keep the file connections open so we do not incur the file opening/closing penalty.