Skip to content

[branch-3.0](pick) Pick #41789 #42482 #41210 #42460 #42914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Gabriel39 merged 5 commits into from
Oct 31, 2024
Merged

[branch-3.0](pick) Pick #41789 #42482 #41210 #42460 #42914

Gabriel39 merged 5 commits into from
Oct 31, 2024

Conversation

@Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Oct 30, 2024 •
edited by morningman
Loading

@doris-robot
Copy link

doris-robot commented Oct 30, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Contributor Author

Gabriel39 commented Oct 30, 2024

run buildall

1 similar comment
@Gabriel39
Copy link
Contributor Author

Gabriel39 commented Oct 31, 2024

run buildall

@Gabriel39
[Improvement](local shuffle) Improve local shuffle strategy (apache#4…
05736f1 
…1789)

Add local shuffle to unpartitioned fragment to add parallel for
perfomance
```sql
SELECT h1.UserID, h2.URL, COUNT(*) AS visit_count
FROM (
  SELECT *
  FROM hits_10m
  LIMIT 5000
) AS h1
CROSS JOIN (
  SELECT *
  FROM hits_10m
  LIMIT 5000
) AS h2
GROUP BY h1.UserID, h2.URL
ORDER BY visit_count DESC
LIMIT 1000
```

Add a rule to apply local exchanger:

```
┌───────────────────────┐               ┌───────────────────────┐
│                       │               │                       │
│Exchange(UNPARTITIONED)│               │Exchange(UNPARTITIONED)│
│                       │               │                       │
└───────────────────────┴──────┬────────┴───────────────────────┘
                               │
                               │
                               │
                               │
                               │
                               │
                        ┌──────▼──────┐
                        │             │
                        │ CROSS JOIN  │
                        │             │
                        └──────┬──────┘
                               │
                               │
                               │
            ┌──────────────────▼─────────────────────┐
            │                                        │
            │ LOCAL EXCHANGE (HASH PARTITION) 1 -> n │
            │                                        │
            └──────────────────┬─────────────────────┘
                               │
                               │
                               │
                               │
                   ▼        ┌──▼────┐
                            │       │
                            │  AGG  │
                            │       │
                            └───────┘
```

before: 1 min 17.79 sec
after: 16.73 sec
@Gabriel39
[refactor](pipeline) Refactor local exchange planning (apache#42482)
249bcde
@Gabriel39
[minor](annotation) Do refactoring and complete annotations (apache#4…
206d213 
…1210)
@Gabriel39 Gabriel39 changed the title [branch-3.0](pick) Pick #41789 #42482 [branch-3.0](pick) Pick #41789 #42482 #41210 Oct 31, 2024
@Gabriel39
Copy link
Contributor Author

Gabriel39 commented Oct 31, 2024

run buildall

@Gabriel39
Copy link
Contributor Author

Gabriel39 commented Oct 31, 2024

run buildall

@Gabriel39
[refactor](coordinator) Refactor local shuffle logics (apache#42460)
1639662 
Unify `useSerialSource` of fragments contain no scan nodes and `ignoreDataDistribution` of fragments contain scan nodes
@Gabriel39 Gabriel39 changed the title [branch-3.0](pick) Pick #41789 #42482 #41210 [branch-3.0](pick) Pick #41789 #42482 #41210 #42460 Oct 31, 2024
@Gabriel39
Copy link
Contributor Author

Gabriel39 commented Oct 31, 2024

run buildall

@Gabriel39 Gabriel39 merged commit 7769c68 into apache:branch-3.0 Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Gabriel39 @doris-robot