cli+server: binding to multiple specific interfaces #5816
Assignees
Labels
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
O-community
Originated from the community
Projects
Comments
|
fyi, I'm going to take a stab at this today... |
petermattis
changed the title
spencerkimball
added
the
C-enhancement
spencerkimball
added
the
help wanted
knz
changed the title
knz
changed the title
knz
added
O-community
craig bot
pushed a commit
that referenced
this issue
Aug 9, 2018
28373: cli: make --host/--{listen,advertise,http}-addr recognize port numbers r=knz a=knz
cc @jseldess @Amruta-Ranade
Fixes #23277.
Needed for #5816.
Prior to this patch, the various `cockroach` sub-commands would take
separate flags to specify an address/hostanme and to specify a port
number.
Meanwhile:
1. `--join` would recognize the syntax `host:port`.
2. the web UI, docs and other places often refer to a "server address"
as the pair hostname:portnr.
For user convenience, it is thus important to make the interface more
straightforward/regular. This patch achieves this as follows:
- the flags
`--listen-addr`/`--advertise-addr`/`--http-addr` (server-side) and
`--host` (client-side) now recognize the syntax `host/addr:port`.
- the server-side `--port` flags are still recognized for backward
compatibility but are marked as deprecated.
The client-side `--port` is still recognized and not
deprecated for now, but hidden from the contextual help.
As a side-effect of recognizing the port number inside the same flag,
the syntax with square brackets for IPv6 addresses now becomes
necessary when specifying also a port number. The syntax without
square brackets (and without port number) is temporarily still
recognized for backward compatibility, but is also marked as
deprecated.
Release note (cli change): the server-side command line flag
`--listen-addr`, which replaces the previous `--host` flag, is now
equipped to recognize both a hostname/address and port number. The
`--port` flag is deprecated as a result.
Release note (cli change): the server-side command line flag
`--http-addr`, which replaces the previous `--http-host` flag, is now
equipped to recognize both a hostname/address and port number. The
`--http-port` flag is deprecated as a result.
Release note (cli change): the server-side command line flag
`--advertise-addr`, which replaces the previous `--advertise-host`
flag, is now equipped to recognize both a hostname/address and
port number. The `--advertise-port` flag is deprecated as a result.
Release note (cli change): the client-side command line flag `--host`
is now equipped to recognize both a hostname/address and port
number. The client-side `--port` flag is still recognized,
but not documented any more; `--host` is now preferred.
Release note (cli change): the environment variable COCKROACH_PORT
that specifies the port number to use for client commands is now
deprecated. The port number can be placed in the COCKROACH_HOST
environment variable instead.
Release note (cli change): The syntax to specify IPv6 addresses with
the client-side command line flag `--host` is changed to use square
brackets, for example `--host=[::1]` instead of just `--host=::1`
previously. The previous syntax is still recognized for backward
compatibility but is deprecated.
Release note (cli change): the flag `--listen-port` which was
introduced in a recent change is now removed. (DOCS NOTE: remove both
this release note and the previous one that introduced --listen-port)
Co-authored-by: Raphael 'kena' Poss <knz@cockroachlabs.com>
knz
added a commit
to knz/cockroach
that referenced
this issue
Aug 5, 2019
This patch introduces the ability to split off the SQL server into a separate port, using the new command-line flag `--sql-addr`. **Motivation** This is a long- and oft-requested feature, aimed at facilitating deployments in professional networks that use firewalls to fence off "internal" (server-side) traffic from external (client) traffic. **Usage** How it works (simplified): - the flag `--sql-addr` indicates on which host/port to listen to for SQL connections. - it is possible to specify `--sql-addr` to be equal to `--listen-addr`, in which case both will share a single TCP connection (internally: using `cmux`). - in fact, the default for `--sql-addr` is to be equal to `--listen-addr`, for compatibility with previous versions of CockroachDB. - when `--sql-addr` and `--listen-addr` are different, then the server does not accept SQL connections any more on the `--listen-addr` address. Note (advanced): the computation of defaults is performed separately for the host and port part of the flag. The logic is the same as that used for `--http-addr`, using default port 5432 (postgres). For example, `--sql-addr=localhost` (without port number) is equivalent to `--sql-addr=localhost:5432`. **Design notes** This is one of two possible design directions that were considered: A. this implementation, where the opt-in flag splits the SQL server entirely onto a different port. *The original port becomes unable to accept SQL connections.* B. the opt-in flag *adds* a SQL server listening onto an additional address. *The original port remains able to accept SQL connections.* The option A is somewhat simpler to implement and also sufficient to answer the use cases from issues cockroachdb#5816 and cockroachdb#30828. However, for backward compatibility with pre-19.2 clients, we cannot both implement option A and make `--sql-addr=:5432` the new default configuration. This would force every client to change the port number they use to connect. That's why Option B may be more interesting for a next iteration. It offers the opportunity to let CockroachDB 19.2 *always* listen to a separate port, presumably the same as postgres, by default. This would smoothen the path to adoption by existing pg clients a little further. In this mode, Pre-19.2 clients would not be affected as the main crdb port (26257) would still accept SQL clients. Release note (cli change): CockroachDB now recognizes a flag `--sql-addr` which makes it possible to accept connections by clients on a separate TCP address and/or port number from the one used for intra-cluster (node-node) connections. This is aimed to enable firewalling client traffic from server traffic.
craig bot
pushed a commit
that referenced
this issue
Aug 8, 2019
39305: server,config,cli: offer a separate port for SQL clients r=knz a=knz (First two commits from #39452) Fixes #30828. Fixes #5816. his patch introduces the ability to split off the SQL server into a separate port, using the new command-line flag `--sql-addr`. The remainder of this commit message details: - the motivation for the change - some usage documentation for users of the new feature - how to introduce this feature in existing clusters - reading recommendations to reviewers of this change - release note **Motivation** This is a long- and oft-requested feature, aimed at facilitating deployments in professional networks that use firewalls to fence off "internal" (server-side) traffic from external (client) traffic. **Usage** How it works (simplified): - the flag `--sql-addr` indicates on which host/port to listen to for SQL connections. - it is possible to specify `--sql-addr` to be equal to `--listen-addr`, in which case both will share a single TCP connection (internally: using `cmux`). - in fact, the default for `--sql-addr` is to be equal to `--listen-addr`, for compatibility with previous versions of CockroachDB. - when `--sql-addr` and `--listen-addr` are different, then the server does not accept SQL connections any more on the `--listen-addr` address. - the flag `--advertise-sql-addr` complements `--sql-addr` in the same way as `--advertise-addr` complements `--listen-addr`. In addition, the output of `cockroach node status` (and the contents of `crdb_internal.gossip_nodes`) is extended to display the SQL address. Note (intermediate): the new flags enables both using separate ports on the same host address (e.g. listening on 127.0.0.1 with separate ports for SQL and RPC), and also using the same port on separate host addresses (e.g. listening on 127.0.0.1:26257 for SQL, and 192.168.2.123:26257 for RPC). Both use cases are legitimate and have seen demand in the wild. Note (advanced): the computation of defaults is performed separately for the host and port part of the flag. The logic is the same as that used for `--http-addr`, using default port 26257. For example, `--sql-addr=localhost` (without port number) is equivalent to `--sql-addr=localhost:26257`. Note (advanced): the computation of defaults for `--advertise-sql-addr` is a bit non-trivial as it pulls its address and port part separately from `--sql-addr`, `--listen-addr` and `--advertise-addr`. Effort was made to ensure sane defaults when some flags but not all are omitted. This is best explained by examples, see the unit tests in flags_test.go for details. **Upgrading a cluster to use separate ports** If a cluster is already online and the need arises to split the ports, the flag can be introduced as follows: - when keeping port 26257 for SQL: 1. restart all nodes in a rolling fashion, updating the `--join` flag on each node to add the new RPC address. 2. restart all nodes in a rolling fashion, adding `--sql-addr=:26257` and setting `--listen-addr=xxx` to the new RPC address. The previous address with port 26257 can also be removed from `--join` in the same step. - when keeping port 26257 for RPC, introducing a new SQL addr/port: 1. if load balancers are in use, extend the load balancer config to also attempt connections on the new SQL address. If load balancers are not in use, temporarily add one that accepts clients on the old addr/port and redirects the connection on both the new addr/port and the old. 2. restart all nodes in a rolling fashion, updating the `--sql-addr` flag. **Review notes** This change was constructed as follows: 1. introducing new fields in `base.Config` 2. implementing the address validation logic in `addr_validation.go` and corresponding unit tests. 3. picking up the new fields to listen separately in `(s *Server) startListenRPCAndSQL()` 4. adding the command line flag parsing and default logic in `flags.go` and corresponding unit tests. 5. manually testing that indeed the server can listen separately on separate ports. 6. to ensure that most unit tests also exercise the split ports, make TestServer/TestCluster set the new flags `SplitListenSQL` 7. extend the TestServer interface to make the SQL address available alongside the RPC address. 8. update all the tests using a SQL connection to use the SQL address 9. update the CLI test logic from `cli_test.go` to configure the `--host` flag to the RPC or SQL address depending on the command being invoked. At this point all tests except for `TestZip` would pass successfully. The remainder of the work is for the benefit of `cockroach zip`, which needs to discover the reachable address of every node in a cluster from the address of just 1 of them, doing so by inspecting all the node descriptors. `cockroach zip` is also peculiar in that it uses both the RPC and SQL interfaces. 1. equip the node descriptor with a field for the SQL address and ensure it gets populated. 2. make the zip logic fetch the SQL address of the primary node by a Node() status request over RPC. 3. for the loop over all nodes, use the SQL address in each node's descriptor for SQL queries separately from the RPC logic. Release note (cli change): CockroachDB now recognizes a flag `--sql-addr` which makes it possible to accept connections by clients on a separate TCP address and/or port number from the one used for intra-cluster (node-node) connections. This is aimed to enable firewalling client traffic from server traffic. Co-authored-by: Raphael 'kena' Poss <knz@cockroachlabs.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
My machines have multiple interfaces (a private network + public). I'd like to be able to bind to localhost (for my code to connect to) the private network (to talk to other nodes). i.e.
I don't want to bind to all interfaces as one is public-facing + accessible. I want to bind to just 127.0.0.1 + private internal network.
The text was updated successfully, but these errors were encountered: