Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improve][Sort] Split one record to multiple records when the physical data has more records for KafkaLoadNode #6296

Closed
2 tasks done
yunqingmoswu opened this issue Oct 26, 2022 · 0 comments · Fixed by #6298

Comments

@yunqingmoswu
Copy link
Contributor

Description

Split one record to multiple records when the physical data has more records for KafkaLoadNode.
It is only used for multiple sink scenario, for example, the raw data is:

    "data":[
        {
            "id":352,
            "price":1.14,
            "currency":"US",
            "order_time":"2022-08-16 18:59:09"
        },
        {
            "id":353,
            "price":1.22,
            "currency":"US",
            "order_time":"2022-08-16 18:59:09"
        }
    ],
    "type":"INSERT",
    "pkNames":[
        "id"
    ],
    "database":"inlong",
    "ts":1666772931248,
    "table":"orders"
}

and then it will be split into two records when writing as follows:

{
    "data":[
        {
            "id":352,
            "price":1.14,
            "currency":"US",
            "order_time":"2022-08-16 18:59:09"
        }
    ],
    "type":"INSERT",
    "pkNames":[
        "id"
    ],
    "database":"inlong",
    "ts":1666772931248,
    "table":"orders"
}
{
    "data":[
        {
            "id":353,
            "price":1.22,
            "currency":"US",
            "order_time":"2022-08-16 18:59:09"
        }
    ],
    "type":"INSERT",
    "pkNames":[
        "id"
    ],
    "database":"inlong",
    "ts":1666772931248,
    "table":"orders"
}

InLong Component

InLong Sort

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Code of Conduct

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment