[SourceKit] Allow module refs to be indexed #19243

rockbruno · 2018-09-11T01:30:41Z

Pitch

This allows Swift to index explicit reference to modules, such as Swift.String.

Resolves SR-8677.

rockbruno · 2018-09-11T01:32:31Z

@akyrtzi Here it is
I got the mangler to generate an USR to modules, but I'm not sure if that's the correct way

akyrtzi

Here are some directions:

For the module USR, update the clang checkout and use generateFullUSRForTopLevelModuleName(StringRef ModName, raw_ostream &OS) which comes from clang/Index/USRGeneration.h. This will allow us to use same USR and track module references across both Swift and ObjC.
For testing, primarily use swift-ide-test, and follow the examples from test/Index directory. Testing core functionality (e.g. how are symbols indexed) should occur via swift-ide-test, and SourceKit testing should be used mainly for verifying that the information goes through the SourceKit responses as expected. From your PR I think the modifications to existing SourceKit tests are sufficient to verify that the information goes through, you don't need to add additional SourceKit test cases.
For testing, I recommend to extend your test case with also:
- Test that module references in import func type of imports also get handled
- Test that module references in clang submodules (like import SomeClangModule.SubModule) are handled properly. You may find generateFullUSRForModule(const clang::Module *Mod, raw_ostream &OS) useful for this

rockbruno · 2018-09-26T01:04:39Z

I had to add a few more methods to treat the clang modules - unit-pcm-dependency.swift is failing, but I wanted to iterate from here:

rockbruno · 2018-09-26T01:05:54Z

lib/Index/IndexSymbol.cpp

@@ -134,10 +134,18 @@ static SymbolKind getVarSymbolKind(const VarDecl *VD) {
  return SymbolKind::Variable;
 }

+SymbolInfo index::getSymbolInfoForModule(ModuleEntity Mod) {
+  SymbolInfo info{ SymbolKind::Unknown, SymbolSubKind::None,


Should we give different parameters to clang modules?

We can use the language kind to differentiate, SymbolLanguage::Swift for Swift modules and generic SymbolLanguage::C for clang modules.

test/Index/Store/unit-pcm-dependency.swift

akyrtzi · 2018-09-27T00:18:56Z

include/swift/AST/USRGeneration.h

@@ -13,6 +13,7 @@
 #ifndef SWIFT_AST_USRGENERATION_H
 #define SWIFT_AST_USRGENERATION_H

+#include "swift/AST/Module.h"


Generally try to keep headers lean, meaning don't include other headers if you can just forward declare the symbols you need instead.
Here you included swift/AST/Module.h so you can reference ModuleEntity, but you can just forward declare it as class ModuleEntity;.
You'd need to include the header itself in the cases where the full definition of ModuleEntity is needed, for example if you use it inside an inline function.

Ah forgive me, I didn't know you could do that in C++. Changes applied

akyrtzi · 2018-09-27T00:20:05Z

include/swift/Index/IndexSymbol.h

@@ -13,6 +13,7 @@
 #ifndef SWIFT_INDEX_INDEXSYMBOL_H
 #define SWIFT_INDEX_INDEXSYMBOL_H

+#include "swift/AST/Module.h"


Forward declare here as well.

akyrtzi · 2018-09-27T00:22:50Z

lib/AST/Module.cpp

@@ -365,6 +365,13 @@ ModuleDecl::ModuleDecl(Identifier name, ASTContext &ctx)
  setAccess(AccessLevel::Public);
 }

+bool ModuleDecl::isClangModule() const {
+  if (!getFiles().empty())


Should this be using ModuleDecl::findUnderlyingClangModule() instead ?

Was the isClangModule logic at https://github.com/apple/swift/blob/c94bc1d7dbde329242cf68f856284095e07d9f1f/tools/swift-ide-test/swift-ide-test.cpp#L2436 duplicated? I took it from there, but changing it to use findUnderlyingClangModule works as well.

Was the isClangModule logic duplicated?

Quite possible, utility code in testing tools generally gets less scrutiny than additions in libAST.

lib/AST/Module.cpp

akyrtzi · 2018-09-27T00:34:18Z

lib/AST/USRGeneration.cpp

@@ -205,6 +203,17 @@ swift::USRGenerationRequest::evaluate(Evaluator &evaluator, const ValueDecl* D)
  llvm::SmallString<128> Buffer;
  llvm::raw_svector_ostream OS(Buffer);

+  if (auto ModuleD = dyn_cast<ModuleDecl>(D)) {
+    StringRef moduleName = ModuleD->getName().str();


A bit better to use ModuleD->getNameStr()

lib/AST/USRGeneration.cpp

lib/Index/Index.cpp

test/Index/index_module_refs.swift

akyrtzi · 2018-09-27T21:04:54Z

include/swift/AST/Module.h

@@ -264,7 +264,6 @@ class ModuleDecl : public DeclContext, public TypeDecl {
    return { Files.begin(), Files.size() };
  }

-  bool isClangModule() const;


Sorry for being unclear, I actually liked that you added isClangModule() method, I was mainly suggesting to look into implementing this in terms of calling findUnderlyingModule()

No worries! I've misread it

rockbruno · 2018-09-27T21:31:55Z

lib/AST/Module.cpp

+bool ModuleDecl::isClangModule() const {
+  if (findUnderlyingClangModule())
+    return true;
+  return false;


(I mainly did this instead of return findUnderlyingClangModule() because I thought the latter might be confusing in terms of the actual return type)

In such situations I usually write it as:

return findUnderlyingClangModule() != nullptr;

akyrtzi · 2018-09-27T21:50:34Z

include/swift/AST/Module.h

@@ -1290,6 +1289,7 @@ class ModuleEntity {
  bool isBuiltinModule() const;
  const ModuleDecl *getAsSwiftModule() const;
  const clang::Module *getAsClangModule() const;
+  void *getOpaqueValue() const;


I'd recommend to make this an inline method (move the definition inline). The definition is simple and inlining it would be good for performance.

akyrtzi · 2018-09-27T22:10:30Z

test/Index/index_module_refs.swift

+import ClangModuleB
+// CHECK: [[@LINE-1]]:8 | module/C | ClangModuleB | c:@M@ClangModuleB | Ref | rel: 0
+import ClangModuleC.Sub1
+// CHECK: [[@LINE-1]]:8 | module/C | ClangModuleC | c:@M@ClangModuleC | Ref | rel: 0


Here's a nice improvement you can make for how test checking works in this file:

Match the USR and add it as a FileCheck variable. So instead of having c:@M@ClangModuleC make it [[ClangModuleC_USR:c:@M@ClangModuleC]]

Later in the file, instead of repeating the same USR to match (c:@M@ClangModuleC) you match with the variable you defined earlier ([[ClangModuleC_USR]]).

Do the same for the other USRs that get repeated, like for SwiftModuleC.

The benefit of this is that it reduces the duplication of the USR string. Later on, if for some reason we decide to change how the USRs for modules are formed, there will be much fewer places in the test cases that would be necessary to correct.

Amazing feature. Thanks for the tip!

akyrtzi · 2018-09-27T22:13:35Z

test/Index/index_module_refs.swift

+  func myMethod() {
+    _ = SwiftModuleC.MyGeneric<SwiftModuleC.MyType, MyType>()
+    // CHECK: [[@LINE-1]]:9 | module/Swift | SwiftModuleC | c:@M@SwiftModuleC | Ref | rel: 0
+    // CHECK: [[@LINE-2]]:22 | struct/Swift | MyGeneric | s:12SwiftModuleC9MyGenericV | Ref,RelCont | rel: 1


I think it will be a bit better to keep the test case focused on testing that module references are indexed. Meaning we don't need to check for MyGeneric and MyType here, other test cases should be responsible to make sure that we index these properly.
Essentially I'd recommend to remove the check lines for those.

Since some of the references rely on getParentDecl(), I was concerned that a change to this feature could potentially break the reference that comes right after it - perhaps I should make a separate test file for this interaction?

That's fair, it'd be fine to have the additional checks but at least reduce how strict the CHECK string needs to be, for example if we only want to check that the struct reference shows up we could have it like this:

// CHECK: [[@LINE-2]]:22 | struct/Swift | MyGeneric | {{.*}} | Ref

So essentially we care that MyGeneric reference shows up but we don't care to match its USR.

akyrtzi · 2018-09-28T01:33:35Z

include/swift/AST/Module.h

@@ -1288,6 +1289,12 @@ class ModuleEntity {
  bool isSystemModule() const;
  bool isBuiltinModule() const;
  const ModuleDecl *getAsSwiftModule() const;
+  const clang::Module *getAsClangModule() const;
+
+  inline void *getOpaqueValue() const {


inline is redundant, it's the fact that you have the definition present that makes it inlinable.

akyrtzi · 2018-09-28T01:46:32Z

test/Index/index_module_refs.swift

+  // CHECK: [[@LINE-1]]:11 | module/Swift | SwiftModuleC | [[SwiftModuleC_USR]] | Ref | rel: 0
+  func myMethod() {
+    _ = SwiftModuleC.MyGeneric<SwiftModuleC.MyType, MyType>()
+    // CHECK: [[@LINE-1]]:9 | module/Swift | SwiftModuleC | [[SwiftModuleC_USR]] | Ref | rel: 0


One I think I noticed is that while the other type references have | Ref,RelCont | rel: 1 (a 'RelCont' relation with the containing method), the module references do not have that.
I wouldn't say it is super important to have the containing relation, but could you take a look to see if it is simple to add, so that the module references do not diverge ?
If you see other test cases, we check for relations by using CHECK-NEXT to match the relations that are printed immediately after the symbol occurrence.

Does this rule applies only to AbstractFunctionDecl? It seems that merely mimicking the generic Decl behaviour does the trick:

auto Parent = getParentDecl(); if (Parent && isa<AbstractFunctionDecl>(Parent)) addRelation(Info, (unsigned)SymbolRole::RelationContainedBy, Parent);

module/Swift | SwiftModuleC | [[SwiftModuleC_USR]] | Ref,RelCont | rel: 1
RelCont | instance-method/Swift | myMethod() | [[myMethod_USR:.*]]

akyrtzi · 2018-09-28T16:50:59Z

test/Index/index_module_refs.swift

@@ -0,0 +1,29 @@
+// RUN: %target-swift-ide-test -print-indexed-symbols -enable-source-import -source-filename %s -I %S/Store/Inputs | %FileCheck %s
+


It's shaping up great! One final (I think) suggestion about the test case; add a line like this:

import NonExistingModuleName // make sure there's no problem with invalid imports

as per comment this would be to guard that there's no issue occurring with invalid imports, like some crash/assertion hit while trying to index its module reference.

Makes sense! I added // CHECK-NOT: {{.*}} | NonExistingModuleName to it to guarantee it doesn't even show up, but if it has performance implications we can leave it out.

akyrtzi · 2018-09-28T17:24:42Z

@swift-ci smoke test

akyrtzi · 2018-09-28T17:27:06Z

Thanks for your patience and work on this! It will be critically useful for providing global rename functionality for frameworks at some point.

rockbruno · 2018-09-28T17:42:32Z

Thank -you- for the tips and assistance! I promise the next PRs will be far less painful...

akyrtzi · 2018-09-28T18:16:14Z

I think test fails on linux because ClangModuleB.h uses @import ClangModuleA; and ObjC interop does not exist on linux.
Try changing that line to #import "ClangModuleA.h". Also I think this change would eliminate the need to have // REQUIRES: objc_interop in unit-pcm-dependency.swift

akyrtzi · 2018-09-28T18:28:29Z

Also add // REQUIRES: objc_interop to SourceKit/Indexing/index_constructors.swift.

rockbruno · 2018-09-28T18:37:49Z

Done

akyrtzi · 2018-09-28T18:40:31Z

@swift-ci smoke test

bruno-rocha-movile and others added 7 commits March 10, 2018 09:38

Merge remote-tracking branch 'apple/master'

e04b034

Merge remote-tracking branch 'apple/master'

e1ac19d

Merge remote-tracking branch 'apple/master'

cc48d84

Enable SK Module Indexing

cde8d3d

Update tests

a19bff4

Generate USR for ModuleDecls

699bfbb

Lint code

4d7a956

akyrtzi self-requested a review September 24, 2018 17:48

akyrtzi requested changes Sep 24, 2018

View reviewed changes

rockbruno added 5 commits September 24, 2018 16:41

Using Clang USR generator

259d03a

Merge branch 'apple/master' into sourcekit-index-module

bbd079a

Indexing Clang modules

fc149dd

Merge branch 'apple/master' into sourcekit-index-module

d0dfe5d

Clang module indexing tests

e2f70b7

rockbruno commented Sep 26, 2018

View reviewed changes

Updating clang module test

05c4d01

rockbruno commented Sep 26, 2018

View reviewed changes

test/Index/Store/unit-pcm-dependency.swift Outdated Show resolved Hide resolved

rockbruno added 2 commits September 26, 2018 16:16

Identify Clang Modules as C modules

eda9734

[Gardening] Remove unnecessary info

0dda357

akyrtzi requested changes Sep 27, 2018

View reviewed changes

Optimizing Module Indexing logic

7f3fa4b

akyrtzi reviewed Sep 27, 2018

View reviewed changes

Add isClangModule

ed5417f

rockbruno commented Sep 27, 2018

View reviewed changes

akyrtzi reviewed Sep 27, 2018

View reviewed changes

Optimizing module indexing logic

d7d2e59

akyrtzi reviewed Sep 28, 2018

View reviewed changes

Add Contained relation to module indexes

287fd61

akyrtzi reviewed Sep 28, 2018

View reviewed changes

Test against invalid modules

46246e7

akyrtzi approved these changes Sep 28, 2018

View reviewed changes

Update tests for linux

c1e30bc

akyrtzi merged commit bf84b29 into swiftlang:master Sep 28, 2018

swift-ci mentioned this pull request Sep 11, 2018

[SR-8677] SourceKit: Add module references as indexSource entities #51191

Closed

		@@ -0,0 +1,29 @@
		// RUN: %target-swift-ide-test -print-indexed-symbols -enable-source-import -source-filename %s -I %S/Store/Inputs \| %FileCheck %s

[SourceKit] Allow module refs to be indexed #19243

[SourceKit] Allow module refs to be indexed #19243

Conversation

rockbruno commented Sep 11, 2018 • edited Loading

rockbruno commented Sep 11, 2018

akyrtzi left a comment

Choose a reason for hiding this comment

rockbruno commented Sep 26, 2018

Choose a reason for hiding this comment

akyrtzi Sep 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akyrtzi commented Sep 28, 2018

akyrtzi commented Sep 28, 2018

rockbruno commented Sep 28, 2018 • edited Loading

akyrtzi commented Sep 28, 2018

akyrtzi commented Sep 28, 2018

rockbruno commented Sep 28, 2018

akyrtzi commented Sep 28, 2018

rockbruno commented Sep 11, 2018 •

edited

Loading

akyrtzi Sep 26, 2018 •

edited

Loading

rockbruno commented Sep 28, 2018 •

edited

Loading