Add pubsub as an option for server

[adaminsta]

We are consuming data from a pub-sub topic and putting it into snowflake, but there is no pubsub option for servers. Need something like this:

source:
  type: pubsub
  project: gcp_project_name
  name: my_topic
 endpoint:
    type: snowflake
    account: xxxxxxxx
    database: my_db
    schema: my_schema

[jochenchrist]

@adaminsta thanks for suggesting, I think this makes sense, equivalent to kafka.

[jochenchrist]

@adaminsta have a look here:
https://github.com/datacontract/datacontract-specification/pull/33/files

[adaminsta]

Looks great!

[jochenchrist]

Closed in datacontract/datacontract-specification#33

[jochenchrist]

Do you have the need for testing the messages in pub/sub?
Do you have any suggestions what engines could help here?

[adaminsta]

We are planning to consume messages from pubsub with dataflow and then merge the data into snowflake tables. My initial idea was to validate format of the input in the dataflow job by comparing with the contract yaml.

An even more powerful solution would be if we can genereate a pubsub schema (only avro/protocol buffer) from the contract yaml, which I can attach to the pubsub topic (https://cloud.google.com/pubsub/docs/schemas)

[simonharrer]

We're planning on such export functionality. See #56 and #57 Feel free to provide us some examples (data contract -> avro schema, data contract -> protobuf) there to help us drive the implementation.