airbyte-lib: Add pip_url option and enforce source versions in a more consistent manner

[flash1293]

This PR does two things:

• Add source install via pip_url from https://github.com/airbytehq/airbyte/pull/33878/files
• Enforce versions only if the version was explicitly stated when initializing the source:
  • If the user does not specify the version, the latest version is installed if there is no installation already, otherwise whatever installed version is available will be used
  • If the user specifies "latest" as their target version, the latest version of the connector is looked up via the registry and enforced (initalization fails if the latest version can't be installed)
  • If the user specifies a specific version, this version is enforced (initalization fails if the latest version can't be installed)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)		Jan 5, 2024 11:47am

[aaronsteers]

One comment/suggestion below, non-blocking.

Apart from that though, what do you think of adding an "uninstall()" action on the venv executor which would simply remove the venv directory? This would be as an inverse of install() or ensure_installation(). In my testing on the jupyter notebook, there were cases where the install action failed, but the venv was already (partially) created. In that case, I was left in an unusable state - not able to run the connector, but auto-detection inferred I already had installed the connector because the venv existed.

[aaronsteers]

Only a suggestion, but I seem to run into a lot of python cases where I get '0.0.0' back - such as when installing from a local path. This might not be necessary, but I think it's worth considering. We can always implement as you've written and just keep an eye out for edge cases. It may already be solved in the latest handling (ignoring when a specific version is not requested), but I want to mention anyway while I'm here.

Suggested change

	if installed_version != self.target_version:
	# If the version doesn't match, reinstall
	self.install()
	if installed_version == '0.0.0':
	# pass or warn; version is either not declared or not detectable
	pass
	elif installed_version != self.target_version:
	# If the version doesn't match, reinstall
	self.install()

[flash1293]

I considered this in the interim, but I'm hoping that we don't need to hardcode this with the current scheme - if you are working with local / pip-url installed connectors you would just not worry about the version (and thus not specify it) and airbyte-lib won't worry either, so it won't hit this.

I'll go ahead and merge and we can add the 0.0.0 exception if it keeps coming up in some scenarios.

[aaronsteers]

Makes sense! Thanks!