Octavia import connection fails when using the new Columns Selection feature

[robertoarnetoli]

Octavia CLI Version

0.50.4

Revelant information

I have a connection (Greenhouse -> S3) in which I use the recently introduced Columns Selection feature. It works correctly and extracts the data as configured.
However, when running the octavia import connection my_connection, I get the following error:
airbyte_api_client.exceptions.ApiTypeError: Invalid type for variable '0'. Required value type is SelectedFieldInfo and passed type was dict at ['selected_fields'][0]
Investigating the issue, I was able to replicate it consistently. The error happens when using Columns Selection to filter out some columns. I have tried selecting and deselecting different column types and it still errors out. Instead, when selecting all columns (i.e without filtering out any columns) the import works correctly.
I'm using Airbyte on AWS EC2 Linux with Docker Compose and both Airbyte and Octavia-CLI version is 0.50.4

Relevant log output

Traceback (most recent call last):
  File "/usr/local/bin/octavia", line 8, in <module>
    sys.exit(octavia())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/base_commands.py", line 54, in invoke
    raise e
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/base_commands.py", line 51, in invoke
    result = super().invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/check_context.py", line 91, in wrapper
    f(ctx, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/_import/commands.py", line 169, in all
    import_connection(api_client, workspace_id, resource_id)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/_import/commands.py", line 121, in import_connection
    managed_resource, state = resources.factory(api_client, workspace_id, new_configuration_path).manage(connection_id)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/apply/resources.py", line 868, in factory
    return Connection(api_client, workspace_id, raw_configuration, configuration_path)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/apply/resources.py", line 279, in __init__
    self.configuration = self._deserialize_raw_configuration()
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/apply/resources.py", line 631, in _deserialize_raw_configuration
    configuration["sync_catalog"] = self._create_configured_catalog(configuration["sync_catalog"])
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/apply/resources.py", line 748, in _create_configured_catalog
    stream=AirbyteStream(**stream["stream"]), config=AirbyteStreamConfiguration(**stream["config"])
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model/airbyte_stream_configuration.py", line 296, in __init__
    setattr(self, var_name, var_value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 185, in __setattr__
    self[attr] = value
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 510, in __setitem__
    self.set_attribute(name, value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 157, in set_attribute
    value = validate_and_convert_types(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1620, in validate_and_convert_types
    input_value[index] = validate_and_convert_types(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1582, in validate_and_convert_types
    raise get_type_error(input_value, path_to_item, valid_classes,
airbyte_api_client.exceptions.ApiTypeError: Invalid type for variable '0'. Required value type is SelectedFieldInfo and passed type was dict at ['selected_fields'][0]

[ravenscroftj]

I can confirm that I am also affected by this issue.

We're using a Source connector for MySQL and Destination is BigQuery. I can manually configure a connection where I turn certain columns and fields on and off and import it into octavia giving the following error (the same as the chap above)

Traceback (most recent call last):
  File "/usr/local/bin/octavia", line 8, in <module>
    sys.exit(octavia())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/base_commands.py", line 54, in invoke
    raise e
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/base_commands.py", line 51, in invoke
    result = super().invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/check_context.py", line 91, in wrapper
    f(ctx, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/_import/commands.py", line 169, in all
    import_connection(api_client, workspace_id, resource_id)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/_import/commands.py", line 121, in import_connection
    managed_resource, state = resources.factory(api_client, workspace_id, new_configuration_path).manage(connection_id)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/apply/resources.py", line 868, in factory
    return Connection(api_client, workspace_id, raw_configuration, configuration_path)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/apply/resources.py", line 279, in __init__
    self.configuration = self._deserialize_raw_configuration()
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/apply/resources.py", line 631, in _deserialize_raw_configuration
    configuration["sync_catalog"] = self._create_configured_catalog(configuration["sync_catalog"])
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/apply/resources.py", line 748, in _create_configured_catalog
    stream=AirbyteStream(**stream["stream"]), config=AirbyteStreamConfiguration(**stream["config"])
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model/airbyte_stream_configuration.py", line 296, in __init__
    setattr(self, var_name, var_value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 185, in __setattr__
    self[attr] = value
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 510, in __setitem__
    self.set_attribute(name, value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 157, in set_attribute
    value = validate_and_convert_types(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1620, in validate_and_convert_types
    input_value[index] = validate_and_convert_types(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1582, in validate_and_convert_types
    raise get_type_error(input_value, path_to_item, valid_classes,
airbyte_api_client.exceptions.ApiTypeError: Invalid type for variable '0'. Required value type is SelectedFieldInfo and passed type was dict at ['selected_fields'][0]

[ravenscroftj]

Ok I think I got to the root of the problem here.

The octavia CLI install script is hard coded to install v0.44.4 of the octavia docker image. I can see that there are much more recent versions of the image in the repo. if I manually change my .zshrc file to 0.50.7 this problem goes away.

Maybe it would make sense to bump the version in the install script (unless there's some reason that it's pinned to 0.44.4 that I don't have context for?) and maybe document how to upgrade octavia in the manual somewhere (I'd be happy to make a PR)?

[marcosmarxm]

Octavia was removed from the main repo and won't receive updates anymore.