cmd/compile: boost inlining into FORs #48209

nimelehin · 2021-09-06T07:52:26Z

As already Than McIntosh mentioned it's a common practise to boost
inlining to FORs, since the callsite could be "hotter". This patch
implements this functionality.

The implementation uses a stack of FORs to recognise calls which are
in a loop. The stack is maintained alongside inlnode function works
and contains information about ancenstor FORs relative to a current
node in inlnode.

There is "big" FOR which cost is >= inlineBigForCost(105). In such FORs
no boost is applied.

Updates #17566

The following results on GO1, while binary size not increased significantly
10454800 -> 10475120, which is less than 0.3%.

goos: linux
goarch: amd64
pkg: test/bench/go1
cpu: Intel(R) Xeon(R) Gold 6230N CPU @ 2.30GHz
name old time/op new time/op delta
BinaryTree17-8 2.15s ± 1% 2.17s ± 1% ~ (p=0.065 n=6+6)
Fannkuch11-8 2.70s ± 0% 2.69s ± 0% -0.25% (p=0.010 n=6+4)
FmtFprintfEmpty-8 31.9ns ± 0% 31.4ns ± 0% -1.61% (p=0.008 n=5+5)
FmtFprintfString-8 57.0ns ± 0% 57.1ns ± 0% +0.26% (p=0.013 n=6+5)
FmtFprintfInt-8 65.2ns ± 0% 63.9ns ± 0% -1.95% (p=0.008 n=5+5)
FmtFprintfIntInt-8 103ns ± 0% 102ns ± 0% -1.01% (p=0.000 n=5+4)
FmtFprintfPrefixedInt-8 119ns ± 0% 118ns ± 0% -0.50% (p=0.008 n=5+5)
FmtFprintfFloat-8 169ns ± 0% 174ns ± 0% +2.75% (p=0.008 n=5+5)
FmtManyArgs-8 445ns ± 0% 447ns ± 0% +0.46% (p=0.002 n=6+6)
GobDecode-8 4.37ms ± 1% 4.40ms ± 0% +0.62% (p=0.009 n=6+6)
GobEncode-8 3.07ms ± 0% 3.04ms ± 0% -0.78% (p=0.004 n=5+6)
Gzip-8 195ms ± 0% 195ms ± 0% ~ (p=0.429 n=5+6)
Gunzip-8 28.2ms ± 0% 28.2ms ± 0% ~ (p=0.662 n=5+6)
HTTPClientServer-8 45.0µs ± 1% 45.4µs ± 1% ~ (p=0.093 n=6+6)
JSONEncode-8 8.01ms ± 0% 8.03ms ± 0% +0.31% (p=0.008 n=5+5)
JSONDecode-8 35.3ms ± 1% 35.1ms ± 0% -0.72% (p=0.008 n=5+5)
Mandelbrot200-8 4.50ms ± 0% 4.49ms ± 1% ~ (p=0.937 n=6+6)
GoParse-8 3.03ms ± 1% 3.00ms ± 1% ~ (p=0.180 n=6+6)
RegexpMatchEasy0_32-8 55.4ns ± 0% 53.2ns ± 3% -3.92% (p=0.004 n=5+6)
RegexpMatchEasy0_1K-8 178ns ± 0% 175ns ± 1% -1.57% (p=0.004 n=5+6)
RegexpMatchEasy1_32-8 50.1ns ± 0% 48.3ns ± 5% ~ (p=0.082 n=5+6)
RegexpMatchEasy1_1K-8 271ns ± 1% 262ns ± 1% -3.26% (p=0.004 n=6+5)
RegexpMatchMedium_32-8 949ns ± 0% 886ns ± 7% ~ (p=0.329 n=5+6)
RegexpMatchMedium_1K-8 27.1µs ± 7% 28.1µs ± 6% ~ (p=0.394 n=6+6)
RegexpMatchHard_32-8 1.28µs ± 2% 1.29µs ± 0% ~ (p=0.056 n=6+6)
RegexpMatchHard_1K-8 38.5µs ± 0% 38.4µs ± 0% -0.25% (p=0.009 n=6+5)
Revcomp-8 397ms ± 0% 396ms ± 0% ~ (p=0.429 n=6+5)
Template-8 48.1ms ± 1% 48.1ms ± 0% ~ (p=0.222 n=5+5)
TimeParse-8 213ns ± 0% 213ns ± 0% ~ (p=0.210 n=4+6)
TimeFormat-8 295ns ± 1% 259ns ± 0% -12.22% (p=0.002 n=6+6)
[Geo mean] 40.5µs 40.1µs -1.00%

name old speed new speed delta
GobDecode-8 176MB/s ± 1% 174MB/s ± 0% -0.61% (p=0.009 n=6+6)
GobEncode-8 250MB/s ± 0% 252MB/s ± 0% +0.79% (p=0.004 n=5+6)
Gzip-8 100MB/s ± 0% 100MB/s ± 0% ~ (p=0.351 n=5+6)
Gunzip-8 687MB/s ± 0% 687MB/s ± 0% ~ (p=0.662 n=5+6)
JSONEncode-8 242MB/s ± 0% 242MB/s ± 0% -0.31% (p=0.008 n=5+5)
JSONDecode-8 54.9MB/s ± 1% 55.3MB/s ± 0% +0.71% (p=0.008 n=5+5)
GoParse-8 19.1MB/s ± 1% 19.3MB/s ± 1% ~ (p=0.143 n=6+6)
RegexpMatchEasy0_32-8 578MB/s ± 0% 601MB/s ± 3% +4.10% (p=0.004 n=5+6)
RegexpMatchEasy0_1K-8 5.74GB/s ± 1% 5.85GB/s ± 1% +1.90% (p=0.002 n=6+6)
RegexpMatchEasy1_32-8 639MB/s ± 0% 663MB/s ± 4% ~ (p=0.082 n=5+6)
RegexpMatchEasy1_1K-8 3.78GB/s ± 1% 3.91GB/s ± 1% +3.38% (p=0.004 n=6+5)
RegexpMatchMedium_32-8 33.7MB/s ± 0% 36.2MB/s ± 7% ~ (p=0.268 n=5+6)
RegexpMatchMedium_1K-8 37.9MB/s ± 6% 36.5MB/s ± 6% ~ (p=0.411 n=6+6)
RegexpMatchHard_32-8 24.9MB/s ± 2% 24.8MB/s ± 0% ~ (p=0.063 n=6+6)
RegexpMatchHard_1K-8 26.6MB/s ± 0% 26.7MB/s ± 0% +0.25% (p=0.009 n=6+5)
Revcomp-8 640MB/s ± 0% 641MB/s ± 0% ~ (p=0.429 n=6+5)
Template-8 40.4MB/s ± 1% 40.3MB/s ± 0% ~ (p=0.222 n=5+5)
[Geo mean] 175MB/s 177MB/s +1.05%

gopherbot · 2021-09-06T07:56:53Z

This PR (HEAD: 2fec790) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/347732 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2021-09-09T16:05:17Z

This PR (HEAD: 7de33c3) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/347732 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2021-09-09T18:07:53Z

Message from Keith Randall:

Patch Set 3:

(17 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/347732.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2021-09-10T15:41:52Z

This PR (HEAD: d7f3135) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/347732 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2021-09-10T16:05:37Z

Message from Nikita Melekhin:

Patch Set 4:

(18 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/347732.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2021-09-10T16:05:45Z

This PR (HEAD: c662bfa) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/347732 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2021-09-10T16:08:33Z

Message from Nikita Melekhin:

Patch Set 5:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/347732.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2021-09-13T22:37:51Z

Message from Keith Randall:

Patch Set 5:

(6 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/347732.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2021-09-16T16:50:23Z

Message from Nikita Melekhin:

Patch Set 5:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/347732.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2021-09-16T16:55:50Z

Message from Keith Randall:

Patch Set 5:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/347732.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2021-09-23T14:42:49Z

This PR (HEAD: 7eb224f) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/347732 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2021-09-23T15:14:06Z

This PR (HEAD: f398028) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/347732 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2021-09-23T15:53:23Z

This PR (HEAD: ba9fc48) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/347732 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2021-09-23T16:13:50Z

Message from Nikita Melekhin:

Patch Set 9:

(8 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/347732.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2021-10-06T15:19:16Z

This PR (HEAD: d2928d0) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/347732 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2021-10-06T15:25:06Z

Message from Nikita Melekhin:

Patch Set 10:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/347732.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2021-10-11T19:50:45Z

Message from Nikita Melekhin:

Patch Set 10:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/347732.
After addressing review feedback, remember to publish your drafts!

As already Than McIntosh mentioned it's a common practise to boost inlining to FORs, since the callsite could be "hotter". This patch implements this functionality. The implementation uses a stack of FORs to recognise calls which are in a loop. The stack is maintained alongside inlnode function works and contains information about ancenstor FORs relative to a current node in inlnode. There is "big" FOR which cost is >= inlineBigForCost(105). In such FORs no boost is applied. Updates golang#17566 The following results on GO1, while binary size not increased significantly 10454800 -> 10475120, which is less than 0.3%. goos: linux goarch: amd64 pkg: test/bench/go1 cpu: Intel(R) Xeon(R) Gold 6230N CPU @ 2.30GHz name old time/op new time/op delta BinaryTree17-8 2.15s ± 1% 2.17s ± 1% ~ (p=0.065 n=6+6) Fannkuch11-8 2.70s ± 0% 2.69s ± 0% -0.25% (p=0.010 n=6+4) FmtFprintfEmpty-8 31.9ns ± 0% 31.4ns ± 0% -1.61% (p=0.008 n=5+5) FmtFprintfString-8 57.0ns ± 0% 57.1ns ± 0% +0.26% (p=0.013 n=6+5) FmtFprintfInt-8 65.2ns ± 0% 63.9ns ± 0% -1.95% (p=0.008 n=5+5) FmtFprintfIntInt-8 103ns ± 0% 102ns ± 0% -1.01% (p=0.000 n=5+4) FmtFprintfPrefixedInt-8 119ns ± 0% 118ns ± 0% -0.50% (p=0.008 n=5+5) FmtFprintfFloat-8 169ns ± 0% 174ns ± 0% +2.75% (p=0.008 n=5+5) FmtManyArgs-8 445ns ± 0% 447ns ± 0% +0.46% (p=0.002 n=6+6) GobDecode-8 4.37ms ± 1% 4.40ms ± 0% +0.62% (p=0.009 n=6+6) GobEncode-8 3.07ms ± 0% 3.04ms ± 0% -0.78% (p=0.004 n=5+6) Gzip-8 195ms ± 0% 195ms ± 0% ~ (p=0.429 n=5+6) Gunzip-8 28.2ms ± 0% 28.2ms ± 0% ~ (p=0.662 n=5+6) HTTPClientServer-8 45.0µs ± 1% 45.4µs ± 1% ~ (p=0.093 n=6+6) JSONEncode-8 8.01ms ± 0% 8.03ms ± 0% +0.31% (p=0.008 n=5+5) JSONDecode-8 35.3ms ± 1% 35.1ms ± 0% -0.72% (p=0.008 n=5+5) Mandelbrot200-8 4.50ms ± 0% 4.49ms ± 1% ~ (p=0.937 n=6+6) GoParse-8 3.03ms ± 1% 3.00ms ± 1% ~ (p=0.180 n=6+6) RegexpMatchEasy0_32-8 55.4ns ± 0% 53.2ns ± 3% -3.92% (p=0.004 n=5+6) RegexpMatchEasy0_1K-8 178ns ± 0% 175ns ± 1% -1.57% (p=0.004 n=5+6) RegexpMatchEasy1_32-8 50.1ns ± 0% 48.3ns ± 5% ~ (p=0.082 n=5+6) RegexpMatchEasy1_1K-8 271ns ± 1% 262ns ± 1% -3.26% (p=0.004 n=6+5) RegexpMatchMedium_32-8 949ns ± 0% 886ns ± 7% ~ (p=0.329 n=5+6) RegexpMatchMedium_1K-8 27.1µs ± 7% 28.1µs ± 6% ~ (p=0.394 n=6+6) RegexpMatchHard_32-8 1.28µs ± 2% 1.29µs ± 0% ~ (p=0.056 n=6+6) RegexpMatchHard_1K-8 38.5µs ± 0% 38.4µs ± 0% -0.25% (p=0.009 n=6+5) Revcomp-8 397ms ± 0% 396ms ± 0% ~ (p=0.429 n=6+5) Template-8 48.1ms ± 1% 48.1ms ± 0% ~ (p=0.222 n=5+5) TimeParse-8 213ns ± 0% 213ns ± 0% ~ (p=0.210 n=4+6) TimeFormat-8 295ns ± 1% 259ns ± 0% -12.22% (p=0.002 n=6+6) [Geo mean] 40.5µs 40.1µs -1.00% name old speed new speed delta GobDecode-8 176MB/s ± 1% 174MB/s ± 0% -0.61% (p=0.009 n=6+6) GobEncode-8 250MB/s ± 0% 252MB/s ± 0% +0.79% (p=0.004 n=5+6) Gzip-8 100MB/s ± 0% 100MB/s ± 0% ~ (p=0.351 n=5+6) Gunzip-8 687MB/s ± 0% 687MB/s ± 0% ~ (p=0.662 n=5+6) JSONEncode-8 242MB/s ± 0% 242MB/s ± 0% -0.31% (p=0.008 n=5+5) JSONDecode-8 54.9MB/s ± 1% 55.3MB/s ± 0% +0.71% (p=0.008 n=5+5) GoParse-8 19.1MB/s ± 1% 19.3MB/s ± 1% ~ (p=0.143 n=6+6) RegexpMatchEasy0_32-8 578MB/s ± 0% 601MB/s ± 3% +4.10% (p=0.004 n=5+6) RegexpMatchEasy0_1K-8 5.74GB/s ± 1% 5.85GB/s ± 1% +1.90% (p=0.002 n=6+6) RegexpMatchEasy1_32-8 639MB/s ± 0% 663MB/s ± 4% ~ (p=0.082 n=5+6) RegexpMatchEasy1_1K-8 3.78GB/s ± 1% 3.91GB/s ± 1% +3.38% (p=0.004 n=6+5) RegexpMatchMedium_32-8 33.7MB/s ± 0% 36.2MB/s ± 7% ~ (p=0.268 n=5+6) RegexpMatchMedium_1K-8 37.9MB/s ± 6% 36.5MB/s ± 6% ~ (p=0.411 n=6+6) RegexpMatchHard_32-8 24.9MB/s ± 2% 24.8MB/s ± 0% ~ (p=0.063 n=6+6) RegexpMatchHard_1K-8 26.6MB/s ± 0% 26.7MB/s ± 0% +0.25% (p=0.009 n=6+5) Revcomp-8 640MB/s ± 0% 641MB/s ± 0% ~ (p=0.429 n=6+5) Template-8 40.4MB/s ± 1% 40.3MB/s ± 0% ~ (p=0.222 n=5+5) [Geo mean] 175MB/s 177MB/s +1.05%

gopherbot · 2021-10-19T15:21:40Z

This PR (HEAD: d68cada) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/347732 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2021-10-25T19:20:48Z

Message from Nikita Melekhin:

Patch Set 11:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/347732.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2021-10-26T17:29:13Z

Message from Keith Randall:

Patch Set 11:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/347732.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2021-10-26T18:31:43Z

Message from Nikita Melekhin:

Patch Set 11:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/347732.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2021-10-26T20:01:30Z

Message from Keith Randall:

Patch Set 11:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/347732.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2021-10-28T14:08:17Z

Message from Nikita Melekhin:

Patch Set 11:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/347732.
After addressing review feedback, remember to publish your drafts!

Mifrill · 2023-11-22T23:24:33Z

src/cmd/compile/internal/inline/inl.go

+	finalBudget := ctx.initialInlineBudget
+	if ctx.canBoostInliningIntoFor() && ctx.initialInlineBudget == inlineMaxBudget {
+		// Boosts only regular functions
+		finalBudget += inlineIntoForExtraBudget
+	}
+
+	return finalBudget


When the calculated value can be directly returned without being stored in an intermediary variable, it results in a more concise and straightforward function:

Suggested change

finalBudget := ctx.initialInlineBudget

if ctx.canBoostInliningIntoFor() && ctx.initialInlineBudget == inlineMaxBudget {

// Boosts only regular functions

finalBudget += inlineIntoForExtraBudget

}

return finalBudget

if ctx.canBoostInliningIntoFor() && ctx.initialInlineBudget == inlineMaxBudget {

// Only boost regular functions

return ctx.initialInlineBudget + inlineIntoForExtraBudget

}

return ctx.initialInlineBudget

google-cla bot added the cla: yes Used by googlebot to label PRs as having a valid CLA. The text of this label should not change. label Sep 6, 2021

nimelehin force-pushed the boost_inline_into_fors branch 3 times, most recently from 415ea1b to 7de33c3 Compare September 9, 2021 15:51

nimelehin force-pushed the boost_inline_into_fors branch from 7de33c3 to d7f3135 Compare September 10, 2021 15:39

nimelehin force-pushed the boost_inline_into_fors branch from d7f3135 to c662bfa Compare September 10, 2021 15:56

nimelehin force-pushed the boost_inline_into_fors branch from c662bfa to 7eb224f Compare September 23, 2021 14:40

nimelehin force-pushed the boost_inline_into_fors branch from 7eb224f to f398028 Compare September 23, 2021 15:11

nimelehin force-pushed the boost_inline_into_fors branch from f398028 to ba9fc48 Compare September 23, 2021 15:52

nimelehin force-pushed the boost_inline_into_fors branch from ba9fc48 to d2928d0 Compare October 6, 2021 15:17

nimelehin force-pushed the boost_inline_into_fors branch from d2928d0 to d68cada Compare October 19, 2021 15:20

Mifrill reviewed Nov 22, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/compile: boost inlining into FORs #48209

cmd/compile: boost inlining into FORs #48209

nimelehin commented Sep 6, 2021 •

edited

Loading

gopherbot commented Sep 6, 2021

gopherbot commented Sep 9, 2021

gopherbot commented Sep 9, 2021

gopherbot commented Sep 10, 2021

gopherbot commented Sep 10, 2021

gopherbot commented Sep 10, 2021

gopherbot commented Sep 10, 2021

gopherbot commented Sep 13, 2021

gopherbot commented Sep 16, 2021

gopherbot commented Sep 16, 2021

gopherbot commented Sep 23, 2021

gopherbot commented Sep 23, 2021

gopherbot commented Sep 23, 2021

gopherbot commented Sep 23, 2021

gopherbot commented Oct 6, 2021

gopherbot commented Oct 6, 2021

gopherbot commented Oct 11, 2021

gopherbot commented Oct 19, 2021

gopherbot commented Oct 25, 2021

gopherbot commented Oct 26, 2021

gopherbot commented Oct 26, 2021

gopherbot commented Oct 26, 2021

gopherbot commented Oct 28, 2021

Mifrill Nov 22, 2023

cmd/compile: boost inlining into FORs #48209

Are you sure you want to change the base?

cmd/compile: boost inlining into FORs #48209

Conversation

nimelehin commented Sep 6, 2021 • edited Loading

gopherbot commented Sep 6, 2021

gopherbot commented Sep 9, 2021

gopherbot commented Sep 9, 2021

gopherbot commented Sep 10, 2021

gopherbot commented Sep 10, 2021

gopherbot commented Sep 10, 2021

gopherbot commented Sep 10, 2021

gopherbot commented Sep 13, 2021

gopherbot commented Sep 16, 2021

gopherbot commented Sep 16, 2021

gopherbot commented Sep 23, 2021

gopherbot commented Sep 23, 2021

gopherbot commented Sep 23, 2021

gopherbot commented Sep 23, 2021

gopherbot commented Oct 6, 2021

gopherbot commented Oct 6, 2021

gopherbot commented Oct 11, 2021

gopherbot commented Oct 19, 2021

gopherbot commented Oct 25, 2021

gopherbot commented Oct 26, 2021

gopherbot commented Oct 26, 2021

gopherbot commented Oct 26, 2021

gopherbot commented Oct 28, 2021

Mifrill Nov 22, 2023

Choose a reason for hiding this comment

nimelehin commented Sep 6, 2021 •

edited

Loading