feature: Add setting to block all external network communication

[WillDaSilva]

Feature scope

Other

Description

Per @dingobar's request on Slack: https://meltano.slack.com/archives/C01TCRBBJD7/p1663137839450139

[WillDaSilva]

I was originally thinking it might be useful to use unshare or some other low-level trick to block the network access, but that would just cause unwanted network access to fail, rather than preventing it in the first place. Also unshare only works on Linux. In a similar vein we could patch Python's networking stdlib functions, but that still causes failures instead of preventing the attempt in the first place.

To prevent attempted external network access before it occurs, we'll need to assemble a list of every place within Meltano that might attempt external network access, and then update them all. We should try to be granular with this. Just because some small relatively unimportant part of a feature attempts to access the external network doesn't mean that the whole feature should be skipped.

A context manager that can only be entered when external network access is permitted can be used for this. It can optionally be provided an exception to raise if it could not be entered, with the default behaviour being to log that it was skipped (including what specifically was skipped because of the setting), and then skip the enclosed block.

[cwegener]

I used to set MELTANO_DISCOVER_URL=false in order to stop meltano from talking to the discovery document in the past. Not sure if that still holds true after moving from the discovery yaml doc to the Hub API.

[aaronsteers]

@cwegener - This is a great call-out. I hadn't considered this myself. But, yes, as you say, the discover_url=false behavior is one step towards achieving the goals of this feature.

Expanding on this a bit, if a feature flag named something like meltano_network_access_disabled equals true, we likely could internally implement as equivalent to forcing hub_api_root=false, discovery_url=false, and send_anonymous_usage_stats=false. Maybe there are other things to restrict as well, but this would likely get us pretty far! We'd need to make clear that this only affects Meltano's-own network access, and that each tap or target would (somewhat obviously) still be able to make their own network-access calls.

• docs.meltano.com/reference/settings#hub_api_root
• docs.meltano.com/reference/settings#discovery_url

@edgarrmondragon - What do you think of this as a possible approach here? Of course we'd have to write tests and verify nothing else is reaching out externally, but an internal override of these component settings seems like a decent way to implement.

[cwegener]

Expanding on this a bit, if a feature flag named something like meltano_network_access_disabled equals true, we likely could internally implement as equivalent to forcing hub_api_root=false, discovery_url=false, and send_anonymous_usage_stats=false. Maybe there are other things to restrict as well, but this would likely get us pretty far!

I couldn't think of anything else myself.

Btw. a good way to validate this manually is via the repro steps I provided in this bug report: #7298 by visually inspecting the http traffic via mitmproxy (I use mitmproxy as part of my local debugging and development efforts quite often)