Remove potential Go module versions from shortened names #571

zikaeroh · 2020-10-09T02:16:05Z

Fixes #515.

Remove potential module path versions (v2, v3, v4, etc) from the input string before extracting a shortened name. This makes it easier to tell which packages are which if the versions happen to match.

An example similar to my report:

codecov-io · 2020-10-09T02:45:14Z

Codecov Report

Merging #571 into master will decrease coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #571      +/-   ##
==========================================
- Coverage   67.14%   67.13%   -0.01%     
==========================================
  Files          78       78              
  Lines       14072    14074       +2     
==========================================
  Hits         9449     9449              
- Misses       3788     3789       +1     
- Partials      835      836       +1

Impacted Files	Coverage Δ
internal/graph/graph.go	`28.03% <100.00%> (+0.15%)`	⬆️
.../github.com/google/pprof/internal/report/source.go	`80.76% <0.00%> (-0.65%)`	⬇️
...rc/github.com/google/pprof/internal/graph/graph.go	`28.03% <0.00%> (+0.15%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 67992a1...b05cf5f. Read the comment docs.

nolanmar511

Thank you for this PR!

nolanmar511 · 2020-10-09T03:05:49Z

internal/graph/graph.go

@@ -34,6 +34,8 @@ var (
 	// Removes package name and method arugments for Go function names.
 	// See tests for examples.
 	goRegExp = regexp.MustCompile(`^(?:[\w\-\.]+\/)+(.+)`)
+	// Checks for a package name that could be a module version.
+	goVerRegExp = regexp.MustCompile(`^v[2-9]+\.`)


I think that a regex like (?:^(?:[\w\-\.]+\/)+((?:[\w\-\.]+\/v[0-9]+)(?:\.[^.\n]+){2})$)|(?:^(?:[\w\-\.]+\/)+(.+)) could be used for goRegExp rather than adding goVerRegExp. Though, I think @aalexand should make the call as to whether we'd prefer to add code or add a more complex regexp.

Otherwise, I think goVerRegExp would miss "v14" and "v10". Perhaps ^v[0-9]+\. or ^v[2-9][0-9]+\. would match better?

Oh, you're right, I brainfarted on that one. Should have been ^v([2-9]|[1-9][0-9]+)\..

I switched it to a corrected regex, but if there's some more complicated one that works better it can be used. I know my brain shuts down trying to read that long one... 🙂

The key thing is to allow v2, v3, v10, v1234, etc, but not v0 or v1.

(?:^(?:[\w\-\.]+\/)+((?:[\w\-\.]+\/v(?:[2-9]|[1-9][0-9]+)+)(?:\.[^.\n]+){2})$)|(?:^(?:[\w\-\.]+\/)+(.+))

Is the long one with the restricted version select; that'd be a one line change and it appears to pass the tests. I'm happy to use it instead.

nolanmar511 · 2020-10-09T03:11:57Z

internal/graph/graph_test.go

@@ -451,6 +451,18 @@ func TestShortenFunctionName(t *testing.T) {
 			"github.com/blah/blah/vendor/gopkg.in/redis.v3.(*baseClient).(github.com/blah/blah/vendor/gopkg.in/redis.v3.process)-fm",
 			"redis.v3.(*baseClient).(github.com/blah/blah/vendor/gopkg.in/redis.v3.process)-fm",
 		},
+		{
+			"github.com/jackc/pgx/v4.(*Conn).Query",


Would you consider using more abstract test case names? (To be in keeping with the style of existing tests)

Sure thing.

nolanmar511 · 2020-10-09T04:41:40Z

internal/graph/graph.go

+			end = idx
+		}
+	}
+


Perhaps remove this line break.

Sure, if you want. I view the above block as calculating end, so that was my thinking.

…pty line

nolanmar511 · 2020-10-15T18:05:04Z

@aalexand -- Did some initial review; wanted to know your thoughts on the approach; specifically if it makes sense to try to use a single regexp or if adding an additional function (as is done here) is the right approach.

aalexand · 2020-10-16T03:49:24Z

internal/graph/graph_test.go

@@ -451,6 +451,30 @@ func TestShortenFunctionName(t *testing.T) {
 			"github.com/blah/blah/vendor/gopkg.in/redis.v3.(*baseClient).(github.com/blah/blah/vendor/gopkg.in/redis.v3.process)-fm",
 			"redis.v3.(*baseClient).(github.com/blah/blah/vendor/gopkg.in/redis.v3.process)-fm",
 		},
+		{
+			"github.com/foo/bar/v4.(*Foo).Bar",


Curious - why is version string sometimes a separate subdirectory and sometimes a prefix of the package name? Is this something that the package owners choose? Are these options restricted at these two, or are there more?

Oh, I guess it's a function of how deep below the versioning level the actual symbol is?

I think you're referring to the tests; some of the tests I've added are where the "version" isn't a version at all. The only valid version in paths are "v2", "v3", ... "v1234", etc. So you'd have github.com/foo/bar, github.com/foo/bar/v2, github.com/foo/bar/v3, and so on, then subpackages like github.com/foo/bar/v3/baz. Custom domains mean you can have things like gotest.tools/assert, gotest.tools/v3/assert. But a package can be at the level where the version appears, so when github.com/jackc/pgx was bumped to github.com/jackc/pgx/v4, it's still referred to as pgx in the code.

But if it isn't a valid version part, then I don't want to treat it as one naively (i.e. "something.com/hello/v123xyz" isn't versioned, "something.com/hello/v123/xyz" is because the version is its own element).

aalexand · 2020-10-16T03:50:12Z

internal/graph/graph.go

+		return name
+	}
+
+	// The shortened name could start with a module version (like "v2"). Go back one slash.


Keep comments in 80 columns please.

aalexand · 2020-10-16T03:55:59Z

internal/graph/graph.go

-			return strings.Join(matches[1:], "")
+			name := strings.Join(matches[1:], "")
+			if re == goRegExp {
+				return shortenGoFunc(f, name)


I feel it might be simpler to first remove the version substring from the name, and then handle it just like before.

Not sure if you saw the previous review comments, but if preferred this all can be removed and replaced with a single regex change.

I feel it might be simpler to first remove the version substring from the name, and then handle it just like before.

I'm not sure how this is possible; the name here is extracted from the regex directly. If we remove the version suffix, you get the empty string.

Oh, I see, you mean that if the name matches a version, remove the suffix from the whole path and then try again. It wouldn't distinguish two versions of the same module, but I guess it's no worse than any other name aliasing within the same graph. Would be short; I can do that if preferred.

I think I misinterpreted again (sorry!), so I'll wait for clarification.

I think you meant just starting with something like github.com/jackc/pgx/v4/foo.bar, then replacing the first instance of /v4/ with /, then running the regex again. Not quite a suffix, but functional enough. This is all heuristics after all.

Sorry, /v[1-9][0-9]*[./].

And not all occurrences but at most one occurrence (assuming there can't be two version substrings in the name).

All instances wouldn't work, because it's legal for me to write github.com/foo/bar/v4/something/v8 or similar. First instance I believe would work as intended, though.

I'll give it a try. Would be v([2-9]|[1-9][0-9]+)\., though, as v0 and v1 don't exist.

Done, and retitled to match the new fix. regexp doesn't have a nice "just replace once", so I used the replace all with two capture groups method.

zikaeroh · 2020-10-16T04:59:34Z

I just realized I didn't add a test for the multi-version case; I can add one quick if that's alright.

zikaeroh · 2020-10-16T05:02:03Z

Ah, yeah, my replacement was overzealous. github.com/foo/bar/v4/foo/bar/v4.(*Foo).Bar is converted to bar.(*Foo).Bar when it shouldn't be. How about a method similar to the one @nolanmar511 suggested with a modified Go regex?

I'm not sure off the top of my head how to construct my regex to capture as little text on the left side as possible.

zikaeroh · 2020-10-16T05:03:59Z

Doh; one character. (.*?)

Sorry to dismiss the review.

Bump from 20191205061153 => 20201109224723 My personal interest is to pull in google/pprof#564, which adds support for displaying names with `"` in them, which julia functions sometimes have (e.g. `var"#foo#23"`) Includes: - google/pprof#564 - google/pprof#575 - google/pprof#574 - google/pprof#571 - google/pprof#572 - google/pprof#570 - google/pprof#562 - google/pprof#561 - google/pprof#565 - google/pprof#560 - google/pprof#563 - google/pprof#557 - google/pprof#554 - google/pprof#552 - google/pprof#545 - google/pprof#549 - google/pprof#547 - google/pprof#541 - google/pprof#534 - google/pprof#542 - google/pprof#535 - google/pprof#531 - google/pprof#530 - google/pprof#528 - google/pprof#522 - google/pprof#525 - google/pprof#527 - google/pprof#519 - google/pprof#520 - google/pprof#517 - google/pprof#518 - google/pprof#514 - google/pprof#513 - google/pprof#510 - google/pprof#508 - google/pprof#506 - google/pprof#509 - google/pprof#504

* Update pprof to latest revision Bump from 20191205061153 => 20201109224723 My personal interest is to pull in google/pprof#564, which adds support for displaying names with `"` in them, which julia functions sometimes have (e.g. `var"#foo#23"`) Includes: - google/pprof#564 - google/pprof#575 - google/pprof#574 - google/pprof#571 - google/pprof#572 - google/pprof#570 - google/pprof#562 - google/pprof#561 - google/pprof#565 - google/pprof#560 - google/pprof#563 - google/pprof#557 - google/pprof#554 - google/pprof#552 - google/pprof#545 - google/pprof#549 - google/pprof#547 - google/pprof#541 - google/pprof#534 - google/pprof#542 - google/pprof#535 - google/pprof#531 - google/pprof#530 - google/pprof#528 - google/pprof#522 - google/pprof#525 - google/pprof#527 - google/pprof#519 - google/pprof#520 - google/pprof#517 - google/pprof#518 - google/pprof#514 - google/pprof#513 - google/pprof#510 - google/pprof#508 - google/pprof#506 - google/pprof#509 - google/pprof#504 * Update P/pprof/build_tarballs.jl - use a real version number Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com> * Remove now unused `timestamp` * [pprof] Use `GitSource` Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

* Expand shortened name if package name appears to be a module version * Correct version regexp, use more generic names in tests, remove an empty line * 80 character columns * Remove first matching Go module version from path * Test and fix multi-version case Co-authored-by: Maggie Nolan <nolanmar@google.com>

Expand shortened name if package name appears to be a module version

88c96e4

google-cla bot added the cla: yes label Oct 9, 2020

nolanmar511 reviewed Oct 9, 2020

View reviewed changes

zikaeroh and others added 2 commits October 8, 2020 21:51

Correct version regexp, use more generic names in tests, remove an em…

011171f

…pty line

Merge branch 'master' into module-version

f25590d

nolanmar511 requested a review from aalexand October 15, 2020 18:02

aalexand reviewed Oct 16, 2020

View reviewed changes

zikaeroh added 2 commits October 15, 2020 21:12

80 character columns

4b5efb1

Remove first matching Go module version from path

5ca898f

zikaeroh changed the title ~~Expand shortened name if package name appears to be a module version~~ Remove potential Go module versions from shortened names Oct 16, 2020

aalexand previously approved these changes Oct 16, 2020

View reviewed changes

Test and fix multi-version case

b05cf5f

zikaeroh dismissed aalexand’s stale review via b05cf5f October 16, 2020 05:04

aalexand approved these changes Oct 16, 2020

View reviewed changes

aalexand merged commit 8ef5528 into google:master Oct 16, 2020

NHDaly mentioned this pull request Nov 13, 2020

Update pprof to latest revision JuliaPackaging/Yggdrasil#2096

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove potential Go module versions from shortened names #571

Remove potential Go module versions from shortened names #571

zikaeroh commented Oct 9, 2020 •

edited

Loading

codecov-io commented Oct 9, 2020 •

edited

Loading

nolanmar511 left a comment

nolanmar511 Oct 9, 2020 •

edited

Loading

zikaeroh Oct 9, 2020 •

edited

Loading

zikaeroh Oct 9, 2020

zikaeroh Oct 9, 2020

zikaeroh Oct 9, 2020

nolanmar511 Oct 9, 2020

zikaeroh Oct 9, 2020

nolanmar511 Oct 9, 2020

zikaeroh Oct 9, 2020

nolanmar511 commented Oct 15, 2020

aalexand Oct 16, 2020

zikaeroh Oct 16, 2020

aalexand Oct 16, 2020

zikaeroh Oct 16, 2020

aalexand Oct 16, 2020

zikaeroh Oct 16, 2020

zikaeroh Oct 16, 2020

zikaeroh Oct 16, 2020 •

edited

Loading

zikaeroh Oct 16, 2020 •

edited

Loading

aalexand Oct 16, 2020

aalexand Oct 16, 2020

zikaeroh Oct 16, 2020

zikaeroh Oct 16, 2020

zikaeroh Oct 16, 2020

zikaeroh commented Oct 16, 2020

zikaeroh commented Oct 16, 2020 •

edited

Loading

zikaeroh commented Oct 16, 2020 •

edited

Loading

Remove potential Go module versions from shortened names #571

Remove potential Go module versions from shortened names #571

Conversation

zikaeroh commented Oct 9, 2020 • edited Loading

codecov-io commented Oct 9, 2020 • edited Loading

Codecov Report

nolanmar511 left a comment

Choose a reason for hiding this comment

nolanmar511 Oct 9, 2020 • edited Loading

Choose a reason for hiding this comment

zikaeroh Oct 9, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nolanmar511 commented Oct 15, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zikaeroh Oct 16, 2020 • edited Loading

Choose a reason for hiding this comment

zikaeroh Oct 16, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zikaeroh commented Oct 16, 2020

zikaeroh commented Oct 16, 2020 • edited Loading

zikaeroh commented Oct 16, 2020 • edited Loading

zikaeroh commented Oct 9, 2020 •

edited

Loading

codecov-io commented Oct 9, 2020 •

edited

Loading

nolanmar511 Oct 9, 2020 •

edited

Loading

zikaeroh Oct 9, 2020 •

edited

Loading

zikaeroh Oct 16, 2020 •

edited

Loading

zikaeroh Oct 16, 2020 •

edited

Loading

zikaeroh commented Oct 16, 2020 •

edited

Loading

zikaeroh commented Oct 16, 2020 •

edited

Loading