Skip to content

[Bug](materialized-view) add limit for drop column on mv#24493

Merged
BiteTheDDDDt merged 7 commits intoapache:masterfrom
BiteTheDDDDt:fix_0916
Sep 19, 2023
Merged

[Bug](materialized-view) add limit for drop column on mv#24493
BiteTheDDDDt merged 7 commits intoapache:masterfrom
BiteTheDDDDt:fix_0916

Conversation

@BiteTheDDDDt
Copy link
Contributor

Proposed changes

add limit for drop column on mv

2023-09-15 20:08:01,085 WARN (leaderCheckpointer|102) [MaterializedIndexMeta.parseStmt():310] CreateMaterializedViewStmt analyze failed, mv=create materialized view mv1 as select user_id, date, city, age, sum(cost) from schema_change_agg_mv_regression_test group by u
ser_id, date, city, age;, reason=
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = Unknown column 'cost' in 'table list'
        at org.apache.doris.common.ErrorReport.reportAnalysisException(ErrorReport.java:53) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.ErrorReport.reportAnalysisException(ErrorReport.java:48) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.analysis.Analyzer.registerColumnRef(Analyzer.java:980) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.analysis.SlotRef.analyzeImpl(SlotRef.java:204) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.analysis.Expr.analyze(Expr.java:443) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.analysis.Expr.analyze(Expr.java:435) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.analysis.SelectStmt.analyze(SelectStmt.java:537) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.analysis.CreateMaterializedViewStmt.analyze(CreateMaterializedViewStmt.java:168) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.MaterializedIndexMeta.parseStmt(MaterializedIndexMeta.java:308) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.OlapTable.analyze(OlapTable.java:2328) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Database.analyze(Database.java:616) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.InternalCatalog.loadDb(InternalCatalog.java:3136) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Env.loadDb(Env.java:1848) ~[doris-fe.jar:1.2-SNAPSHOT]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_131]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_131]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131]
        at org.apache.doris.persist.meta.MetaReader.read(MetaReader.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Env.loadImage(Env.java:1779) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.master.Checkpoint.doCheckpoint(Checkpoint.java:151) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.master.Checkpoint.runAfterCatalogReady(Checkpoint.java:79) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
2023-09-15 20:08:01,087 WARN (leaderCheckpointer|102) [MaterializedIndexMeta.parseStmt():310] CreateMaterializedViewStmt analyze failed, mv=create materialized view mv2 as select user_id, date, city, age, cost from schema_change_uniq_mv_regression_test;, reason=
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = Unknown column 'cost' in 'table list'

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@BiteTheDDDDt
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.9 seconds
stream load tsv: 604 seconds loaded 74807831229 Bytes, about 118 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162364072 Bytes

@BiteTheDDDDt
Copy link
Contributor Author

run buildall

1 similar comment
@BiteTheDDDDt
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.66 seconds
stream load tsv: 606 seconds loaded 74807831229 Bytes, about 117 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162416833 Bytes

@BiteTheDDDDt
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.78 seconds
stream load tsv: 612 seconds loaded 74807831229 Bytes, about 116 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162371052 Bytes

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 19, 2023
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@BiteTheDDDDt BiteTheDDDDt merged commit 5e4ab7c into apache:master Sep 19, 2023
JingDas pushed a commit to JingDas/doris that referenced this pull request Sep 20, 2023
xiaokang pushed a commit to xiaokang/doris that referenced this pull request Sep 27, 2023
xiaokang pushed a commit that referenced this pull request Oct 3, 2023
@BiteTheDDDDt BiteTheDDDDt deleted the fix_0916 branch January 20, 2025 06:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

Comments