Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-6841][table]Using TableSourceTable for both Stream and Batch #4061

Closed
wants to merge 2 commits into from

Conversation

sunjincheng121
Copy link
Member

Remove StreamTableSourceTable, And using TableSourceTable for both Stream and Batch.

  • General

    • The pull request references the related JIRA issue ("[FLINK-6841][table]Using TableSourceTable for both Stream and Batch")
    • The pull request addresses only one issue
    • Each commit in the PR has a meaningful commit message (including the JIRA id)
  • Documentation

    • Documentation has been added for new functionality
    • Old documentation affected by the pull request has been updated
    • JavaDoc for public methods has been added
  • Tests & Build

    • Functionality added by the pull request is covered by tests
    • mvn clean verify has been executed successfully locally or a Travis build has passed

@twalthr
Copy link
Contributor

twalthr commented Jun 6, 2017

I'm not sure about this change. A batch table source would be able to create a time attribute if it implements DefinedProctimeAttribute, right?

@sunjincheng121
Copy link
Member Author

sunjincheng121 commented Jun 7, 2017

Hi @twalthr I think DefinedProctimeAttribute can be implemented by a batch table source. If we do not want a batch table source can implements DefinedProctimeAttribute , we should move DefinedProctimeAttribute to StreamTableSourceTable. Otherwise, I feel that StreamTableSourceTable can be deleted.Just like the changes of this PR. What do you think?

@@ -42,7 +42,7 @@ class ExternalTableSourceUtilTest {
val schema = new TableSchema(Array("foo"), Array(BasicTypeInfo.INT_TYPE_INFO))
val table = ExternalCatalogTable("mock", schema)
val tableSource = ExternalTableSourceUtil.fromExternalCatalogTable(table)
assertTrue(tableSource.isInstanceOf[StreamTableSourceTable[_]])
assertTrue(tableSource.isInstanceOf[TableSourceTable[_]])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think checking the result of fromExternalCatalogTable is a TableSourceTable is meaningless now (it definitely returns a TableSourceTable). I suggest to check whether the TableSource in the TableSourceTable is a StreamTableSource.

assertTrue(tableSource.tableSource.isInstanceOf[StreamTableSource[_]])

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense to me. :)

@wuchong
Copy link
Member

wuchong commented Jun 11, 2017

@twalthr @sunjincheng121 DefinedProctimeAttribute and DefinedRowtimeAttribute doesn't work in batch table source, currently. BTW, I don't think batch table source needs DefinedRowtimeAttribute. Users can use an existing column as the row time attribute without declare the column as a rowtime, isn't that right ?

For the following example, the ts is a physical column in table T.

SELECT overAgg(b, a) FROM T GROUP BY TUMBLE(ts, INTERVAL '2' HOUR)`

@sunjincheng121
Copy link
Member Author

sunjincheng121 commented Jun 11, 2017

Hi, @wuchong Thanks for your reviewing. I meant from the perspective of grammar is allow batch table source to implement the DefinedProctimeAttribute and DefinedRowtimeAttribute. We are on the same page about batch table source do not needs DefinedRowtimeAttribute.

I have updated the PR. according your comment. @wuchong

Thanks,
SunJincheng

@fhueske
Copy link
Contributor

fhueske commented Jun 21, 2017

Hi @sunjincheng121! I left a comment on the JIRA issue. Short summary: I don't think this PR is a significant improvement. I'd rather keep it as it is unless you have a good argument to convince me.

Thank you, Fabian

@sunjincheng121
Copy link
Member Author

Thanks @fhueske, I noticed your description on the JIRA. issue. I think it's make sense to me. Because at the current time between StreamTableSourceTable and (Batch)TableSourceTable really some different, such as watermarks, time attributes etc. which you have mentioned. So I agree with you, and close both the PR. and JIRA.

Thanks,
SunJincheng

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants