Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DBZ-60 Added MySQL server ID and timestamp to event's source info #47

Merged
merged 1 commit into from May 20, 2016

Conversation

rhauch
Copy link
Member

@rhauch rhauch commented May 20, 2016

Added to the Debezium event message's source information the MySQL server ID for the cluster process that recorded the event and the MySQL timestamp at which the event was recorded.

The following is a sample of the new source struct in a message's envelope:

      "source" : {
        "name" : "kafka-connect",
        "server-id" : 112233,
        "ts" : 1463753313000,
        "file" : "mysql-bin.000003",
        "pos" : 1081,
        "row" : 0
      },

Note that the server-id and ts fields are both new, and the name field was previously called server and contains the connector's logical name of the server; the rest of the fields in source are unchanged, as are any other part of the event structure. The schema for the source struct is now:

   {
        "type" : "struct",
        "fields" : [ {
          "type" : "string",
          "optional" : false,
          "field" : "name"
        }, {
          "type" : "int64",
          "optional" : false,
          "field" : "server-id"
        }, {
          "type" : "int64",
          "optional" : false,
          "field" : "ts"
        }, {
          "type" : "string",
          "optional" : false,
          "field" : "file"
        }, {
          "type" : "int64",
          "optional" : false,
          "field" : "pos"
        }, {
          "type" : "int32",
          "optional" : false,
          "field" : "row"
        } ],
        "optional" : false,
        "name" : "io.debezium.connector.mysql.Source",
        "field" : "source"
      }

And finally, the following shows the source struct and schema in the context of a DELETE event's message value:

{
    "schema" : {
      "type" : "struct",
      "fields" : [ {
        "type" : "struct",
        "fields" : [ {
          "type" : "int32",
          "optional" : false,
          "field" : "id"
        }, {
          "type" : "string",
          "optional" : false,
          "field" : "name"
        }, {
          "type" : "string",
          "optional" : true,
          "field" : "description"
        }, {
          "type" : "double",
          "optional" : true,
          "field" : "weight"
        } ],
        "optional" : true,
        "name" : "connector_test.products",
        "field" : "before"
      }, {
        "type" : "struct",
        "fields" : [ {
          "type" : "int32",
          "optional" : false,
          "field" : "id"
        }, {
          "type" : "string",
          "optional" : false,
          "field" : "name"
        }, {
          "type" : "string",
          "optional" : true,
          "field" : "description"
        }, {
          "type" : "double",
          "optional" : true,
          "field" : "weight"
        } ],
        "optional" : true,
        "name" : "connector_test.products",
        "field" : "after"
      }, {
        "type" : "struct",
        "fields" : [ {
          "type" : "string",
          "optional" : false,
          "field" : "name"
        }, {
          "type" : "int64",
          "optional" : false,
          "field" : "server-id"
        }, {
          "type" : "int64",
          "optional" : false,
          "field" : "ts"
        }, {
          "type" : "string",
          "optional" : false,
          "field" : "file"
        }, {
          "type" : "int64",
          "optional" : false,
          "field" : "pos"
        }, {
          "type" : "int32",
          "optional" : false,
          "field" : "row"
        } ],
        "optional" : false,
        "name" : "io.debezium.connector.mysql.Source",
        "field" : "source"
      }, {
        "type" : "string",
        "optional" : false,
        "field" : "op"
      }, {
        "type" : "int64",
        "optional" : true,
        "field" : "ts"
      } ],
      "optional" : false,
      "name" : "kafka-connect.connector_test.products",
      "version" : 1
    },
    "payload" : {
      "before" : {
        "id" : 1001,
        "name" : "roy",
        "description" : "old robot",
        "weight" : 1234.56005859375
      },
      "after" : null,
      "source" : {
        "name" : "kafka-connect",
        "server-id" : 112233,
        "ts" : 1463753313000,
        "file" : "mysql-bin.000003",
        "pos" : 1081,
        "row" : 0
      },
      "op" : "d",
      "ts" : 1463753314594
    }
}

Added to the Debezium event message's `source` information the MySQL server ID for the cluster process that recorded the event and the MySQL timestamp at which the event was recorded.
@@ -165,6 +165,9 @@ public void shouldConsumeAllEventsFromDatabase() throws SQLException, Interrupte
assertInsert(updates.get(0), "id", 2001);
assertDelete(updates.get(1), "id", 1001);
assertTombstone(updates.get(2), "id", 1001);

//Testing.Print.enable();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO clean this up? :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've intentionally left these as I often want to enable printing things when running the tests, and it's far easier to uncomment these than repeatedly figure out the best place to put them.

@criccomini
Copy link
Contributor

LGTM one nit on some commented out test code

@rhauch rhauch merged commit 0a4068a into debezium:master May 20, 2016
@rhauch rhauch deleted the dbz-60 branch May 24, 2016 00:46
@xinbinhuang xinbinhuang mentioned this pull request Jun 27, 2023
xinbinhuang pushed a commit to xinbinhuang/debezium that referenced this pull request Jun 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants