bug: api/v1/version?include_latest=true fails behind firewall

[martimors]

Meltano Version

2.6.0

Python Version

3.9

Bug scope

API

Operating System

Linux - Ubuntu 22.04

Description

Not sure if everyone would agree but I file this as a bug. I find this reasonable because meltano should not depend on external metadata to do its job.

When running behind a restrictive firewall, meltano ui start succeeds, but fails on request due to calling api/v1/version?include_latest=true, which calls out to pypi.org to get the latest version of meltano from there. For environments with strict firewall policies (typically environments with a large amount of critical infrastructure like in our case the telco industry), we cannot open for egress to external domains that are not our own. In these cases, I would like meltano to fail gracefully. In general, it could be worth considering how we will deal with these types of external metadata dependencies going forwards - crash? succeed but fail external calls silently? succeed but log warnings of failed calls? global switch to disable egress requests?

Here is the controller code for the endpoint.

Here is where meltano ui calls the endpoint with the latest_version=true parameter.

Suggested fix

I suggest we simply add a try-except on that call to the endpoint, and return latest_version: null if we cannot retrieve the latest version.

I don't mind implementing that. I have some time on my hands the coming days too.

Code

The error stacktrace for meltano UI.

2022-09-13T07:13:46.096021Z [debug    ] Authenticated as <meltano.api.security.identity.FreeUser object at 0x7fc3012a7250>
2022-09-13T07:16:41.764133Z [error    ] Exception on /api/v1/version [GET] 
Traceback (most recent call last):
  File "/app/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/app/.venv/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/app/.venv/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/app/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/app/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/app/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/app/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "/app/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fc30217e200>: Failed to establish a new connection: [Errno 101] Network is unreachable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/app/.venv/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/app/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/app/.venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/meltano/json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fc30217e200>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/app/.venv/lib/python3.10/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/app/.venv/lib/python3.10/site-packages/flask/app.py", line 1519, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/app/.venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 271, in error_router
    return original_handler(e)
  File "/app/.venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 271, in error_router
    return original_handler(e)
  File "/app/.venv/lib/python3.10/site-packages/flask/app.py", line 1517, in full_dispatch_request
    rv = self.dispatch_request()
  File "/app/.venv/lib/python3.10/site-packages/flask/app.py", line 1503, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/app/.venv/lib/python3.10/site-packages/meltano/api/controllers/root.py", line 79, in version
    res = requests.get("https://pypi.org/pypi/meltano/json")
  File "/app/.venv/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/app/.venv/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/app/.venv/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/app/.venv/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/app/.venv/lib/python3.10/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/meltano/json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fc30217e200>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
2022-09-13T07:16:41.765133Z [info     ] Error: 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
2022-09-13T07:16:41.765499Z [debug    ] Using JSON Scheme: camel
2022-09-13T07:16:41.767334Z [debug    ] Using JSON Scheme: camel
2022-09-13T07:16:41.990129Z [debug    ] Authentication not required because it's disabled
2022-09-13T07:16:41.990751Z [debug    ] Using JSON Scheme: camel
[2022-09-13 07:16:41,992] [11|MainThread|gunicorn.access] [INFO] 127.0.0.1 - - [13/Sep/2022:07:16:41 +0000] "GET /api/v1/repos/models HTTP/1.1" 200 1385 "http://localhost:5000/api/docs" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"
2022-09-13T07:16:41.993842Z [debug    ] Using JSON Scheme: camel
2022-09-13T07:16:42.277294Z [debug    ] Authentication not required because it's disabled
2022-09-13T07:16:42.277536Z [debug    ] Authenticated as <meltano.api.security.identity.FreeUser object at 0x7fc30217fd60>
2022-09-13T07:16:48.800253Z [debug    ] Using JSON Scheme: camel
[2022-09-13 07:16:48,875] [11|MainThread|gunicorn.access] [INFO] 127.0.0.1 - - [13/Sep/2022:07:16:48 +0000] "GET /api/v1/plugins/all HTTP/1.1" 200 17444 "http://localhost:5000/api/docs" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"
2022-09-13T07:16:48.877340Z [debug    ] Using JSON Scheme: camel
2022-09-13T07:16:49.098389Z [debug    ] Authentication not required because it's disabled
2022-09-13T07:16:49.098696Z [debug    ] Authenticated as <meltano.api.security.identity.FreeUser object at 0x7fc30217e830>
2022-09-13T07:16:56.476360Z [debug    ] Found plugin parent            parent=tap-csv plugin=tap-csv source=discovery

[tayloramurphy]

Related to

• feature: Add setting to block all external network communication #6754

@aaronsteers this seems like an uncontroversial change. Any objections?

[aaronsteers]

@dingobar, @tayloramurphy re:

Suggested fix

I suggest we simply add a try-except on that call to the endpoint, and return latest_version: null if we cannot retrieve the latest version.

I don't mind implementing that. I have some time on my hands the coming days too.

Yes, I agree with this approach. Totally makes sense.

Also, and not mutually exclusive with the above, we could add a UI setting called ui.update_check (here as sibling to ui.read_only) which defaults to true but can be set to false to completely disable the check.

What do you think?

[tayloramurphy]

@aaronsteers I like it 👍

@dingobar we'd love a PR on this if you're willing 😄

[martimors]

I'll give it a shot

[martimors]

@aaronsteers I like it +1

@dingobar we'd love a PR on this if you're willing smile

I took the liberty of creating a separate issue for it here #6760 as I see it as a separate issue (in fact a FR not a BUG). I submitted a PR here #6761.

For the issue at hand, I have a PR hanging here #6751 which I would love to get a review on as well

[WillDaSilva]

Thank you for the issue @dingobar. As you've hopefully heard by now, the Meltano UI has been deprecated, and is scheduled for removal in Meltano 3.0. Since no additional work/improvements for the UI are planned leading up to its removal, I'll be closing this issue.