Question: rcmgr limit conflicts with connmgr limit

[richard-ramos]

While running this example code with GOLOG_LOG_LEVEL=WARN go run main.go , I always see this warning being printed:

2023-10-31T12:36:25.909-0400    WARN    p2p-config      config/config.go:304    rcmgr limit conflicts with connmgr limit: conn manager high watermark limit: 192, exceeds the system connection limit of: 1

What does this warning mean? I tried to follow go-libp2p code for the resource manager and connection manager to find out but had some trouble understanding it. Is there something I should change to avoid having this warning?

package main

import (
	"fmt"
	"os"
	"os/signal"
	"syscall"

	"github.com/libp2p/go-libp2p"
)

func main() {
	host, err := libp2p.New()
	if err != nil {
		panic(err.Error())
	}

	fmt.Println(host.ID())

	// Wait for a SIGINT or SIGTERM signal
	ch := make(chan os.Signal, 1)
	signal.Notify(ch, syscall.SIGINT, syscall.SIGTERM)
	<-ch
	fmt.Println("\n\n\nReceived signal, shutting down...")
}

Thank you in advance.

[burdiyan]

Facing the same issue.

[sukunrt]

This means that you have configured the connection manager in such a way that it will never be used.

Libp2p uses two modules for connection managements. Resource manager limit(rcmgr) is a hard limit. If the rcmgr limit is set to 1 you will only have 1 open connection.

The ConnMgr limit is a soft limit of sort. It allows for a burst of connections between ConnMgr.LowWaterMark and ConnMgr.HighWaterMark. This allows for a burst of connections between Low and high watermark. But when you exceed the high watermark, the system trims connections down to low watermark.

In your case HighWatermark trimming never happens because it's higher than the rcmgr limit. Ideally your high watermark limit should be a bit less than the rcmgr connections limit.

[juligasa]

I have the same issue, even if I set unlimited resources, it still complains about the system limit being 1 WARN p2p-config config/config.go:304 rcmgr limit conflicts with connmgr limit: conn manager high watermark limit: 768, exceeds the system connection limit of: 1
My configuration so far:

cm := mustConnMgr(connmgr.NewConnManager(
	512,
	768,
	connmgr.WithGracePeriod(r.cfg.ConnMgr.ConnMgrGrace),
))

opts = append(opts,
	libp2p.ConnectionManager(cm),
)

scalingLimits := rcmgr.DefaultLimits

// Add limits around included libp2p protocols
libp2p.SetDefaultServiceLimits(&scalingLimits)
// Turn the scaling limits into a concrete set of limits using `.AutoScale`. This
// scales the limits proportional to your system memory.
scaledDefaultLimits := scalingLimits.AutoScale()

// Tweak certain settings
cmcfg := rcmgr.PartialLimitConfig{
	System: rcmgr.ResourceLimits{
		Streams:  rcmgr.Unlimited,
		Conns:    rcmgr.Unlimited,
		FD:      rcmgr.Unlimited,
		Memory:  rcmgr.Unlimited64,
	},
	// Everything else is default. The exact values will come from `scaledDefaultLimits` above.
}

// Create our limits by using our cfg and replacing the default values with values from `scaledDefaultLimits`
limits := cmcfg.Build(scaledDefaultLimits)

// The resource manager expects a limiter, so we create one from our limits.
limiter := rcmgr.NewFixedLimiter(limits)

// Metrics are enabled by default. If you want to disable metrics, use the
// WithMetricsDisabled option
// Initialize the resource manager
rm, err := rcmgr.NewResourceManager(limiter)
if err != nil {
	panic(err)
}
	
opts = append(opts,
	libp2p.ResourceManager(rm),
)

r.host, err = libp2p.New(opts...)
if err != nil {
	return err
}

[burdiyan]

@juligasa I fixed that by using rcmgr.NewFixedLimiter(rcmgr.InfiniteLimits) instead of trying to manually expand the default limits to be unlimited. That fixed the issue with the connmgr conflict.

[juligasa]

Oh, I just followed the package recommendations at
https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#usage
I wonder why it did not work as expected.

[richard-ramos]

This means that you have configured the connection manager in such a way that it will never be used.

I'm confused on why I'm getting that warning even tho I have not configured any custom connection manager, but using the defaults provided by go-libp2p as seen on the example code i wrote here: #2628 (comment) . Shouldnt go-libp2p by default configure the connection manager with settings that avoid this warning being emitted?

[sukunrt]

@juligasa you need to set the Conn scope limit too.

	// Tweak certain settings
	cmcfg := rcmgr.PartialLimitConfig{
		System: rcmgr.ResourceLimits{
			Streams: rcmgr.Unlimited,
			Conns:   rcmgr.Unlimited,
			FD:      rcmgr.Unlimited,
			Memory:  rcmgr.Unlimited64,
		},
		// Everything else is default. The exact values will come from `scaledDefaultLimits` above.
		Conn: rcmgr.ResourceLimits{
			Conns: rcmgr.Unlimited,
		},
	}

Shouldnt go-libp2p by default configure the connection manager with settings that avoid this warning being emitted?

I agree. I'll check how to improve the situation here.

[juligasa]

@juligasa you need to set the Conn scope limit too.

I see, I thought the System level was the top abstraction scope, though. And by setting it to unlimited would cause the rest of scopes to be also unlimited (if no further restriction was in place). According to the documentation:

The system scope is the top level scope that accounts for global resource usage at all levels of the system. This scope nests and constrains all other scopes and institutes global hard limits

I guess the libp2p.SetDefaultServiceLimits overrides the connection limit to 1 and hence, the warning.

[sukunrt]

This constraint is the other way round, no?

The system scope is the top level scope that accounts for global resource usage at all levels of the system. This scope nests and constrains all other scopes and institutes global hard limits

If the system scope is 1 connection, peer scope cannot be 10 connections

On the other hand

If the system scope is 100 connections, peer scope can be 1 connection
If the system scope is inf connections, peer scope can be 1 connection

[juligasa]

Sure you're right. The problem is with default peer scopes though. I did not expect them to be so tight by default (actually, allowing only 1 connection is pretty tight). Will set limits accordingly, thanks!

[sukunrt]

You're right, @juligasa . This warning doesn't make sense. The system connection limit is what we should check. I've raised a PR to fix this.

Just to be clear:

You DO NOT NEED to increase Conn Scope limit as I suggested here. I'm sorry about that. #2628 (comment)
The Connection scope only ever needs 1 connection because it is well tied to the connection 😓

[juligasa]

Ok, Makes sense