bug: Meltano should not need to reach out to Hub if plugins are all locked

[aaronsteers]

Meltano Version

(current)

Description

When plugins in a project all have lock files created, Meltano is still reaching out to the Hub - which causes hard failure in scenarios where network connectivity is tightly governed.

Code

No response

Related

• Slack report: https://meltano.slack.com/archives/C01TCRBBJD7/p1675952792259879
• Feature flag to disallow Meltano calling Hub outside of add and discover #7095
• feature: Add setting to block all external network communication #6754

[edgarrmondragon]

@aaronsteers I'd have to confirm, but I don't think Meltano reaches out to Hub if a lockfile is found for a plugin. The lookup order is:

1. Parent (for inherited plugins)
2. Lockfile
3. Legacy discovery.yml
4. Hub

[aaronsteers]

@edgarrmondragon - That's my expectation as well. If Meltano is reaching out to the Hub for a plugin that's already locked, and especially when all plugins in the project are locked, that would be a bug, I think.

[edgarrmondragon]

@aaronsteers tested without an internet connection:

$ meltano@stable invoke tap-getpocket --about
2023-02-09T19:02:49.268226Z [info     ] Environment 'prod' is active
Name: tap-getpocket
Description: Pocket tap class.
Version: 0.0.1
Sdk_Version: 0.19.0
Capabilities: [catalog, state, discover, about, stream-maps, schema-flattening]
Settings: <omitted for brevity>

$ mv plugins/extractors/tap-getpocket--edgarrmondragon.lock plugins/extractors/tap-getpocket--edgarrmondragon.lock2

$ meltano@stable invoke tap-getpocket --about
2023-02-09T19:03:10.666046Z [info     ] Environment 'prod' is active
2023-02-09T19:03:12.342724Z [warning  ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x109cc1e20>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /meltano/api/v1/plugins/extractors/index
2023-02-09T19:03:12.343848Z [warning  ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x109ce0430>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /meltano/api/v1/plugins/extractors/index
2023-02-09T19:03:12.344510Z [warning  ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x109ce0a00>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /meltano/api/v1/plugins/extractors/index
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Could not connect to Meltano Hub. Could not reach Meltano Hub.

[aaronsteers]

Thanks, @edgarrmondragon. So, based on the above, it looks like we're not able to repro the issue on our side. Perhaps a lock file is missing from the user's project, or stored in an unexpected location, or else something else is going on that would cause this.

The only thing I can think to recommend on the user side would be to rerun meltano lock --all just to be sure, and then perform a similar test to the above.

[WillDaSilva]

@edgarrmondragon @aaronsteers This may be caused by the variant not being specified in meltano.yml: https://meltano.slack.com/archives/C01TCRBBJD7/p1676044092702169?thread_ts=1675952792.259879&cid=C01TCRBBJD7

[aaronsteers]

Closing and replacing with new issue - this appears to be better described by the issue here:

• Meltano should treat plugins as 'custom' if no variant is provided #7305