fix for non-string/array anyOf types getting lost to the void

[rabidaudio]

This form of property definition for SCHEMA events causes the columns to be skipped silently, unless the non-null type is string or array:

{
    "type": "object",
    "properties": {
        "duration": {
            "anyOf": [
                {
                    "type": "number",
                    "format": "float"
                },
                {
                    "type": "null"
                }
            ]
        }
    }
}

This is actually preferred to {"type": ["null", "number"], "format": "float"} because under a strict reading of the specification, a null value would be expected to match the format which is impossible.

This form is in use in tap-gitlab:

{"type": "SCHEMA", "stream": "pipelines_extended", "schema": {"type": "object", "properties": {"project_id": {"type": "integer"}, "id": {"type": "integer"}, "status": {"type": "string"}, "ref": {"type": ["null", "string"]}, "sha": {"type": "string"}, "before_sha": {"type": "string"}, "tag": {"type": "boolean"}, "yaml_errors": {"type": ["null", "string"]}, "user": {"type": "object", "properties": {"name": {"type": "string"}, "username": {"type": "string"}, "id": {"type": "integer"}, "state": {"type": "string"}}}, "created_at": {"type": "string", "format": "date-time"}, "updated_at": {"type": "string", "format": "date-time"}, "started_at": {"anyOf": [{"type": "string", "format": "date-time"}, {"type": "null"}]}, "finished_at": {"anyOf": [{"type": "string", "format": "date-time"}, {"type": "null"}]}, "committed_at": {"anyOf": [{"type": "string", "format": "date-time"}, {"type": "null"}]}, "duration": {"anyOf": [{"type": "integer"}, {"type": "null"}]}, "coverage": {"anyOf": [{"type": "number"}, {"type": "null"}]}, "web_url": {"type": "string"}}}, "key_properties": ["id"]}

You can see the columns missing from CREATE TABLE:

INFO Table 'projects' does not exist. Creating...
CREATE TABLE tap_gitlab.projects ("approvals_before_merge" bigint, "archived" boolean, "avatar_url" character varying, "builds_enabled" boolean, "container_registry_enabled" boolean, "created_at" timestamp without time zone, "creator_id" bigint, "default_branch" character varying, "description" character varying, "forks_count" bigint, "http_url_to_repo" character varying, "id" bigint, "issues_enabled" boolean, "last_activity_at" timestamp without time zone, "lfs_enabled" boolean, "merge_method" character varying, "merge_requests_enabled" boolean, "name" character varying, "name_with_namespace" character varying, "namespace__full_path" character varying, "namespace__id" bigint, "namespace__kind" character varying, "namespace__name" character varying, "namespace__parent_id" bigint, "namespace__path" character varying, "only_allow_merge_if_all_discussions_are_resolved" boolean, "only_allow_merge_if_build_succeeds" boolean, "open_issues_count" bigint, "owner_id" bigint, "path" character varying, "path_with_namespace" character varying, "public" boolean, "public_builds" boolean, "request_access_enabled" boolean, "shared_runners_enabled" boolean, "shared_with_groups" jsonb, "snippets_enabled" boolean, "ssh_url_to_repo" character varying, "star_count" bigint, "statistics__commit_count" bigint, "statistics__job_artifacts_size" bigint, "statistics__lfs_objects_size" bigint, "statistics__repository_size" bigint, "statistics__storage_size" bigint, "tag_list" jsonb, "visibility" character varying, "visibility_level" bigint, "web_url" character varying, "wiki_enabled" boolean, PRIMARY KEY ("id"))

In this fix, I tried to make it more explicit about what JSON Schema definitions are actually valid in Postgres, namely only supporting the anyOf meta type if there are exactly 2 type definitions and one of them is {"type":"null"}.

[DouweM]

@rabidaudio Looks good, thanks!

[DouweM]

Version bumped to meltano.1.1.6 :)