New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug 1799579 - Handle OverloadExpr in MozSearchIndexer #613
Conversation
Re: being a decl and not a def, assuming the logic is analogous to a pure virtual decl being basically a def, we'd want to add an additional case to https://searchfox.org/mozilla-central/rev/e66cff951667dacd0faa95dfde830564a58a8a3f/build/clang-plugin/mozsearch-plugin/MozsearchIndexer.cpp#1621-1622 // We treat pure virtual declarations as definitions.
Kind = (D2->isThisDeclarationADefinition() || D2->isPure()) ? "def" : "decl"; |
Ahh, ok, I'm just being silly then and failed to notice that showing the menu item requires an actual definition and not just a declaration. (I'm used to clangd which interprets "go to definition" broadly and will take you to a declaration if that's all it can find.) I don't think there's a need to special-case anything here. |
Ah, so your branch probably would benefit from being rebased to the current tip. I added "Go to declaration" as a new functionality of the context menu in https://bugzilla.mozilla.org/show_bug.cgi?id=1776522 but it's not present in your branch. In your branch, there's only support for "go to def" and "search". Actually, just as a general awareness thing, as part of that change, I added the concept of binding slots. This is generally really only relevant for IDLs and language bindings where we're dealing with the same conceptual symbol expressed in different languages (interface definition, C++, JS, usually)... I expect for templates we'd generally be using superclass/overrides relationships, but there may be cases where it could make sense. The docs in the code there provide some context, but also, concretely: If we run [
{
"structured": 1,
"pretty": "nsIXPCTestParams::testOctet",
"sym": "XPIDL_nsIXPCTestParams_testOctet",
"kind": "method",
"implKind": "idl",
"sizeBytes": null,
"bindingSlots": [
{
"slotKind": "method",
"slotLang": "cpp",
"sym": "_ZN16nsIXPCTestParams9TestOctetEhPhS0_"
},
{
"slotKind": "method",
"slotLang": "js",
"sym": "#testOctet"
}
],
"supers": [],
"methods": [],
"fields": [],
"overrides": [],
"props": []
},
{
"structured": 1,
"pretty": "nsIXPCTestParams::TestOctet",
"sym": "_ZN16nsIXPCTestParams9TestOctetEhPhS0_",
"kind": "method",
"parentsym": "T_nsIXPCTestParams",
"slotOwner": {
"slotKind": "method",
"slotLang": "cpp",
"sym": "XPIDL_nsIXPCTestParams_testOctet"
},
"implKind": "",
"sizeBytes": null,
"bindingSlots": [],
"supers": [],
"methods": [],
"fields": [],
"overrides": [],
"props": [
"instance",
"virtual",
"user"
],
"overriddenBy": [
"_ZN15nsXPCTestParams9TestOctetEhPhS0_"
]
}
] |
Apologies for the long delay here; I just recently finished digging out from under a bunch of APZ work I had fallen behind on while being on PTO earlier in the year. I rebased the patch and updated it to illustrate the behaviour in the presence of multiple overloads, which is currently pretty rudimentary: we just offer all of them as candidate targets, even if e.g. the number of parameters is off. There is room to improve on this, but I'd like to leave such improvements to a future patch as they're not completely trivial (the options are either (1) using the One other small note: the reason the test file is named The patch should be ready for review now. |
(FWIW clangd also currently offers all candidates in a situation like this, so the patch gets us to parity with clangd.) |
I'm reviewing now, here's
|
Sorry for the delay, I wanted to try and dig into what might be happening for the structured information for Brief Structured Info InvestigationNot too surprisingly, we currently don't emit anything for the ClassTemplateDecl for Foo or the CXXRecordDecl it wraps as indicated by adding a check of [
{
"loc": "00004:7-10",
"source": 1,
"nestingRange": "4:11-15:0",
"syntax": "def,type",
"pretty": "type Foo",
"sym": "T_Foo"
},
{
"loc": "00004:7-10",
"target": 1,
"kind": "def",
"pretty": "Foo",
"sym": "T_Foo",
"peekRange": "4-4"
}
] We do have explicit handling of TraverseClassTemplateDecl for our existing template machinery which pushes a template context. And here is currently where we emitStructuredInfo for RecordDecl subclasses which explicitly bails out because there is a template context because:
|
One thing I find that can be handy for new functionality is to use the snapshot mechanism to capture the state of the snapshot before and after the fix. This is also not too hard to do after the fact, so just for my reference, here is pre-fix for the new test case: [
{
"loc": "00006:7-14",
"source": 1,
"nestingRange": "6:25-6:26",
"syntax": "def,function",
"type": "void (Point<F>)",
"pretty": "function Foo::Project",
"sym": "_ZN3Foo7ProjectE5PointITL0__E"
},
{
"loc": "00006:7-14",
"target": 1,
"kind": "def",
"pretty": "Foo::Project",
"sym": "_ZN3Foo7ProjectE5PointITL0__E",
"context": "Foo",
"contextsym": "T_Foo",
"peekRange": "6-6"
},
{
"loc": "00009:7-14",
"source": 1,
"nestingRange": "9:35-9:36",
"syntax": "def,function",
"type": "void (Point<F>, Point<F>)",
"pretty": "function Foo::Project",
"sym": "_ZN3Foo7ProjectE5PointITL0__ES2_"
},
{
"loc": "00009:7-14",
"target": 1,
"kind": "def",
"pretty": "Foo::Project",
"sym": "_ZN3Foo7ProjectE5PointITL0__ES2_",
"context": "Foo",
"contextsym": "T_Foo",
"peekRange": "9-9"
}
] and the new bits that show up with the fix are: {
"loc": "00013:4-11",
"source": 1,
"syntax": "use,function",
"type": "void (Point<F>)",
"pretty": "function Foo::Project",
"sym": "_ZN3Foo7ProjectE5PointITL0__E"
},
{
"loc": "00013:4-11",
"source": 1,
"syntax": "use,function",
"type": "void (Point<F>, Point<F>)",
"pretty": "function Foo::Project",
"sym": "_ZN3Foo7ProjectE5PointITL0__ES2_"
},
{
"loc": "00013:4-11",
"target": 1,
"kind": "use",
"pretty": "Foo::Project",
"sym": "_ZN3Foo7ProjectE5PointITL0__E",
"context": "Foo::Bar",
"contextsym": "_ZN3Foo3BarEv"
},
{
"loc": "00013:4-11",
"target": 1,
"kind": "use",
"pretty": "Foo::Project",
"sym": "_ZN3Foo7ProjectE5PointITL0__ES2_",
"context": "Foo::Bar",
"contextsym": "_ZN3Foo3BarEv"
} |
This is the relevant JSON dump for the call line excerpted from {
"id": "0x7def08",
"kind": "CallExpr",
"range": {
"begin": {
"offset": 226,
"line": 13,
"col": 5,
"tokLen": 7
},
"end": {
"offset": 235,
"col": 14,
"tokLen": 1
}
},
"type": {
"qualType": "<dependent type>"
},
"valueCategory": "prvalue",
"inner": [
{
"id": "0x7dee88",
"kind": "UnresolvedMemberExpr",
"range": {
"begin": {
"offset": 226,
"col": 5,
"tokLen": 7
},
"end": {
"offset": 226,
"col": 5,
"tokLen": 7
}
},
"type": {
"qualType": "<bound member function type>"
},
"valueCategory": "lvalue"
},
{
"id": "0x7deee8",
"kind": "DeclRefExpr",
"range": {
"begin": {
"offset": 234,
"col": 13,
"tokLen": 1
},
"end": {
"offset": 234,
"col": 13,
"tokLen": 1
}
},
"type": {
"desugaredQualType": "Point<float>",
"qualType": "Point<float>"
},
"valueCategory": "lvalue",
"referencedDecl": {
"id": "0x7de6a0",
"kind": "VarDecl",
"name": "p",
"type": {
"desugaredQualType": "Point<float>",
"qualType": "Point<float>"
}
}
}
]
} |
Yeah, this definitely seems like the right incremental step to take given the complexity here. And this is largely consistent with searchfox's previous treatment of C++ method overrides where searchfox would essentially conflate all the overrides in a set. (Note that this was due to both the C++ indexer's behavior plus identifier lookup policies.) That said, before those changes, visitIdentifier used to take a vector of symbols so that it could emit a single source record with all the symbols combined. That would avoid the weirdness with the context menu which currently results in the VM having 2 sets of "Go to definition of", "Search for", etc. However, this shouldn't actually be a problem in practice for mozilla-central trees because the merge-analyses step will coalesce the 2 "source" records into a single "source" record with both symbols, so I don't think it makes sense to go back to taking a vector unless we expect that we'll never be able to address this specific situation. For example, running {
"loc": "00013:4-11",
"target": 1,
"kind": "use",
"pretty": "Foo::Project",
"sym": "_ZN3Foo7ProjectE5PointITL0__E",
"context": "Foo::Bar",
"contextsym": "_ZN3Foo3BarEv"
}
{
"loc": "00013:4-11",
"target": 1,
"kind": "use",
"pretty": "Foo::Project",
"sym": "_ZN3Foo7ProjectE5PointITL0__ES2_",
"context": "Foo::Bar",
"contextsym": "_ZN3Foo3BarEv"
}
{
"loc": "00013:4-11",
"source": 1,
"syntax": "function,use",
"pretty": "function Foo::Project",
"sym": "_ZN3Foo7ProjectE5PointITL0__ES2_,_ZN3Foo7ProjectE5PointITL0__E",
"type": "void (Point<F>)"
} Note that there's only 1 source record because we explicitly coalesce source records that have the same "pretty" value and exist at the same location. (This is necessary for differences in symbols between platforms.) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consistent with the above, this looks good! Thank you!
I wonder if you should rebase this stack to no longer include the already landed #617 ? I'm not sure.
I invited you to be part of the mozsearch org so you can potentially just hit the merge button yourself.
Note that of course there's no automation to get this into m-c, so you'll need to copy the file over to there, sanity check you're not clobbering anything, and then push a review to phabricator, etc. I'm happy to r+ there too, although I don't know if that's strictly necessary given that this is canonical upstream, but it can't hurt.
57343e3
to
9505e99
Compare
Thanks for the review!
Yep, rebased.
IIUC, Searchfox's m-c index itself will pick up the change from this repo, right? Perhaps, if there's no particularly pressing reason to get the changes into the m-c copy, I can do a batch cherry-pick of all the bug 1781178 fixes once they're all done? |
No? The in-tree copy of https://searchfox.org/mozilla-central/source/build/clang-plugin/mozsearch-plugin/MozsearchIndexer.cpp is what is used to build the taskcluster artifacts that get downloaded from taskcluster by fetch-tc-artifacts.sh. We only use the copy of MozsearchIndexer.cpp that's checked into this repository for configurations where we run the build process on the indexer, something we prefer to avoid. Currently that means only the following trees which do builds in their Coincidentally, it also means that mozilla-beta and mozilla-release and mozilla-esr102 would only see changes if we uplift any landed changes to MozsearchIndexer.cpp. Usually this isn't a major problem and we also don't uplift, but it does mean that we always need to be very careful about any potentially breaking schema changes. Note that ESRs older than 102 don't have semantic indexing because the taskcluster jobs get turned off for them and we need them, so we change the scripts for the ESRs once they become unsupported.
I have no strong opinion on this and would leave that call to @emilio. My weak opinion is that if you think you'll be able to address the overload ambiguity in the next ~month, it probably makes sense to intentionally delay propagating to m-c to avoid creating extra confusion from the changes in the behavior. |
Ah, my bad, I misunderstood this aspect of the setup.
Given my updated understanding that rolling a change out to users does require copying the change over into m-c, my updated preference is to copy the fixes over as soon as they're merged into this repo, for a couple of reasons:
My thinking is to look at the rest of the improvements described in bug 1781178 (which I'm fairly confident in as "porting the behaviour over from clangd" type of work) before exploring improvements related to the overload ambiguity issue (which is more of an unknown as something that clangd currently doesn't get right either), and thus I probably wouldn't bank on having the latter addressed within the next month.
My sense is that going from (1) a symbol usage site having no associated semantic actions, only textual search (the current state), to (2) having an associated semantic search action which finds all the candidates, and then later to (3) also having a go-to-definition action that goes to a particular result, are both strictly improvements; I'm not particularly concerned about confusion from users having (2) for a while. |
Works for me! |
I filed bug 1833555 for this, or rather a refinement of this that takes into account the fact that (based on our discussion in #628) we wouldn't want to discard candidates that aren't used by known instantiations, but we would want to annotate them differently. |
Posting a WIP patch for bug 1799579.
The patch partially fixes the bug, in that the call to
Project
is now clickable and offers the menu item "Search for functionFoo::Project
", but it does not yet offer the menu item "Go to definition ofFoo::Project
", which we'd like as well.I will continue investigating why it doesn't offer that second menu item, but in the meantime I wanted to post what I have.
Also, a note on why I didn't use the
AutoTemplateContext
machinery:While
OverloadExpr
is (usually, and in this testcase in particular) dependent, it's a bit different fromCXXDependentScopeMemberExpr
in that candidate declarations being referenced are available, and it's only the selection among the candidates that's deferred until instantiation (whereas, withCXXDependentScopeMemberExpr
, candidate declarations are not available prior to instantiation because the scope in which name lookup would have to be performed to find the candidates is itself not available).As a result, we can provide heuristic results for
OverloadExpr
without looking at instantiations (namely, just provide all the candidates), and I think this is useful enough for a first pass.I think there's an opportunity to improve the results by using the
AutoTemplateContext
machinery to discover which candidates are actually used by instantiations and potentially narrow the results that way, but I'm proposing to defer that to a future improvement.(I should definitely expand the testcase to include a scenario with multiple overloads to illustrate the behaviour, I will do that in an upcoming revision of this patch.)