Network segments #3431

kyhavlov · 2017-08-30T19:41:20Z

Adds the OSS components for network segments. This feature allows clients to be split up into named network segments, in order to separate their gossip layer from clients in different segments. The client's segment is stored in the consul-network-segment key in its node metadata.

Servers are always in all segments, and each segment can be configured with its own bind/advertise address and port (along with an option for an rpc listener if the segment's bind address is different from the default rpc addr).

slackpad

Made an initial pass and noted a few things.

slackpad · 2017-08-30T22:29:32Z

agent/agent.go

 	Leave() error
 	LANMembers() []serf.Member
+	LANSegmentMembers(name string) ([]serf.Member, error)


We should s/name/segment/ for this one so it's clearer what this does.

slackpad · 2017-08-30T22:31:22Z

agent/agent.go

+
+		segments = append(segments, consul.NetworkSegment{
+			Name:       segment.Name,
+			Bind:       segment.Bind,


The Bind and Advertise values here won't be correct if they were empty any we used the default ones. Is that OK?

Good catch, fixed

slackpad · 2017-08-30T22:33:48Z

agent/agent.go

@@ -2105,6 +2169,11 @@ func (a *Agent) loadMetadata(conf *Config) error {
 		a.state.metadata[key] = value
 	}

+	// The segment isn't reloadable so we only add it once.


Why did I add this? It seems like we call unloadMetadata anyway, so this extra check isn't necessary and we can just add it always.

slackpad · 2017-08-30T22:40:26Z

agent/agent_endpoint.go

 	var members []serf.Member
 	if wan {
 		members = s.agent.WANMembers()
-	} else {
+	} else if segment == "" {


This could be simplified to always just call LanSegmentMembers(segment) since that can handle the empty string case internally.

slackpad · 2017-08-30T22:41:10Z

agent/config.go

+	// (Enterprise-only) NetworkSegment is the network segment for this client to join
+	Segment string `mapstructure:"segment"`
+
+	// Segments


Needs a doc comment.

slackpad · 2017-08-30T22:58:30Z

agent/consul/config.go

@@ -49,6 +49,16 @@ func init() {
 	}
 }

+// (Enterprise-only)


Should have a brief doc comment.

slackpad · 2017-08-30T23:01:02Z

agent/consul/internal_endpoint.go

-	// Fire the event
-	return m.srv.serfLAN.UserEvent(eventName, args.Payload, false)
+	// Fire the event on all LAN segments
+	segments := m.srv.LANSegments()


How come you decided not to include the default segment in LANSegments(), which would eliminate the special cases? I think you had that way partway through development.

Yeah, I removed it from the segments map on the server but I should have re-added it to the map that LANSegments() returns, fixed

slackpad · 2017-08-30T23:01:53Z

agent/consul/internal_endpoint.go

 	} else {
-		mgr = m.srv.KeyManagerLAN()
+		segments := m.srv.LANSegments()


Doh this doesn't special case for the default segment, see comment above we should probably just put the default segment in here and avoid the special cases.

slackpad · 2017-08-30T23:10:23Z

command/agent.go

+	}
+
+	if !cfg.Server && len(cfg.Segments) > 0 {
+		cmd.UI.Error("Cannot define segments on clients")


Maybe "Segments can only be defined on servers" to be more symmetric with the other message.

slackpad

Made a detailed pass through everything. Nice job on the guide - I think that's super easy to follow. Noted some things and then I think we are good to merge this.

slackpad · 2017-08-31T19:07:41Z

agent/agent.go

@@ -2105,6 +2169,8 @@ func (a *Agent) loadMetadata(conf *Config) error {
 		a.state.metadata[key] = value
 	}

+	a.state.metadata[structs.MetaSegmentKey] = conf.Segment


Now I remember why I did that - since the segment isn't reloadable, this would let them change their metadata about it. I think this is fine as-is vs. adding code complexity (my thing didn't work since the map is empty). We explicitly document what can be reloaded.

slackpad · 2017-08-31T19:11:09Z

agent/config.go

+		return fmt.Errorf("Cannot exceed network segment limit of %d", SegmentLimit)
+	}
+
+	takenPorts := make(map[int]string, len(conf.Segments))


Realized we also need to make sure the names are unique, otherwise we will overlay one on the other when we create them.

slackpad · 2017-08-31T19:16:07Z

agent/config.go

 	return &result, nil
 }

+func ValidateSegments(conf *Config) error {


In OSS we should probably bail with an error if they set the segment on a client, and if they set up any segments for servers. I wonder if it's worth splitting an agent/segment_stub.go here in OSS and creating ValidateSegments() into there that does those checks, and then moving this into agent/segment.go in Enterprise to represent the two behaviors (you'd have to move the test for this over into Enterprise as well).

slackpad · 2017-08-31T19:21:35Z

agent/consul/segment_stub.go

+)
+
+const (
+	errSegmentsNotSupported = "network segments are not supported in this version of Consul"


If you pull out an agent-level segment.go and stub, you could move this error to a common place (maybe structs)?

slackpad · 2017-08-31T19:22:29Z

agent/consul/server.go

-	raftState         = "raft/"
-	snapshotsRetained = 2
+	serfLANSnapshot        = "serf/local.snapshot"
+	serfLANSegmentSnapshot = "serf/local-segment-%s.snapshot"


This could get moved to Enterprise agent/consul/segment.go.

slackpad · 2017-08-31T19:52:47Z

website/source/api/query.html.md

@@ -87,6 +87,25 @@ populate the query before it is executed. All of the string fields inside the
  doesn't match, or an invalid index is given, then `${match(N)}` will return an
  empty string.

+- `${agent.segment}` has the network segment (Enterprise only) of the agent that


Enterprise-only (hyphen for consistency with other places)

slackpad · 2017-08-31T19:54:39Z

website/source/docs/guides/segments.html.markdown.erb

+# Partial LAN Connectivity
+## Configuring Network Segments
+
+[//]: # ( ~> The network area functionality described here is available only in )


Change "network area" to "network segment" and change version to 0.9.3.

slackpad · 2017-08-31T19:58:01Z

website/source/docs/guides/segments.html.markdown.erb

+By default, all Consul agents in one datacenter are part of a shared gossip pool over the LAN;
+this means that the partial connectivity caused by segmented networks would cause health flapping
+as nodes failed to communicate. In this guide we will cover the Network Segments feature, added
+in [Consul Enterprise](https://www.hashicorp.com/products/consul/) version 0.9.3, which allows users


Need to link this network segments guide from a new landing page on https://www.consul.io/docs/enterprise/index.html..

slackpad · 2017-08-31T20:01:11Z

website/source/docs/guides/segments.html.markdown.erb

+
+Once the servers have been configured with the correct segment info, the clients only need to specify
+their own segment in the [Agent Config](/docs/agent/options.html) and join by connecting to another
+agent within the same segment.


We should add something like "If joining to a Consul server, client will need to provide the server's port for their segment along with the address of the server when performing the join (eg. consul agent -retry-join "consul.domain.internal:1234").

slackpad · 2017-08-31T20:03:01Z

website/source/docs/guides/segments.html.markdown.erb

+client1  4c29819c  192.168.0.5  dc1
+```
+
+With this metadata key, we can construct a prepared query that can be used for DNS to return


I'd link to "/api/query.html" for "prepared query".

slackpad · 2017-09-01T16:53:36Z

command/segment_stub.go

-	return nil, nil
+	members, err := client.Agent().MembersOpts(consulapi.MembersOpts{})
+	if err != nil {
+		return nil, fmt.Errorf("Error retrieving members: %s", err)


Just return err here, since you prepend the same thing above.

slackpad · 2017-09-01T16:54:29Z

website/source/docs/enterprise/index.html.md

@@ -18,6 +18,7 @@ increases both scalability and resilience. Features include:
 - [Redundancy Zones](/docs/enterprise/redundancy/index.html)
 - [Advanced Federation for Complex Network
  Topologies](/docs/enterprise/federation/index.html)
+- [Network Segments](/docs/guides/segments.html)


ping @KFishner this is fine for now but do you want to write a marketing landing thing for later?

slackpad

One tiny nit then LGTM.

slackpad · 2017-09-01T16:58:01Z

command/segment_stub.go

+func getSegmentMembers(client *consulapi.Client) ([]*consulapi.AgentMember, error) {
+	members, err := client.Agent().MembersOpts(consulapi.MembersOpts{})
+	if err != nil {
+		return nil, fmt.Errorf("Error retrieving members: %s", err)


@kyhavlov just return err here since we add the same string above

slackpad and others added 6 commits August 30, 2017 11:58

Adds open source side of network segments (feature is Enterprise-only).

b1a15e0

Add segment config validation

a30e765

Add rpc_listener option to segment config

2ada043

Add agent.segment interpolation to prepared queries

d129767

Add segment addr field to tags for LAN flood joiner

14b027a

Update coord display in ui to account for segments

21513b0

kyhavlov requested review from magiconair, preetapan and slackpad August 30, 2017 19:41

Default bind/advertise for segments to BindAddr/AdvertiseAddr

16aaf27

slackpad reviewed Aug 30, 2017

View reviewed changes

kyhavlov force-pushed the network-segments-oss branch from c2766bd to 19fd123 Compare August 31, 2017 00:00

Fix some inconsistencies with segment logic and comments

7e565d7

kyhavlov force-pushed the network-segments-oss branch from 19fd123 to 7e565d7 Compare August 31, 2017 00:43

kyhavlov added 2 commits August 31, 2017 03:35

Fill in the segment in the QuerySource for prepared query lookups

baa501e

Add doc sections for network segments

00d03f1

slackpad reviewed Aug 31, 2017

View reviewed changes

kyhavlov added 3 commits August 31, 2017 17:39

Organize segments for a cleaner split between enterprise and OSS

62102a5

Update segment docs

40f7508

Pass listeners into setupSegments

0e33e2e

slackpad reviewed Sep 1, 2017

View reviewed changes

slackpad approved these changes Sep 1, 2017

View reviewed changes

slackpad reviewed Sep 1, 2017

View reviewed changes

Remove duplicate error message

f826133

kyhavlov merged commit 220db48 into master Sep 1, 2017

kyhavlov deleted the network-segments-oss branch September 1, 2017 17:25

freddygv mentioned this pull request Nov 14, 2022

Ensure imported services are filtered in IndexedNodesWithGateways #15360

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Network segments #3431

Network segments #3431

kyhavlov commented Aug 30, 2017

slackpad left a comment

slackpad Aug 30, 2017

slackpad Aug 30, 2017

kyhavlov Aug 30, 2017

slackpad Aug 30, 2017

slackpad Aug 30, 2017

slackpad Aug 30, 2017

slackpad Aug 30, 2017

slackpad Aug 30, 2017

kyhavlov Aug 30, 2017

slackpad Aug 30, 2017

slackpad Aug 30, 2017

slackpad left a comment

slackpad Aug 31, 2017

slackpad Aug 31, 2017

slackpad Aug 31, 2017

slackpad Aug 31, 2017

slackpad Aug 31, 2017

slackpad Aug 31, 2017

slackpad Aug 31, 2017

slackpad Aug 31, 2017

slackpad Aug 31, 2017

slackpad Aug 31, 2017

slackpad Sep 1, 2017

slackpad Sep 1, 2017

KFishner Sep 4, 2017

slackpad left a comment

slackpad Sep 1, 2017

Network segments #3431

Network segments #3431

Conversation

kyhavlov commented Aug 30, 2017

slackpad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slackpad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slackpad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment