Using index.scip locally?

[for-just-we]

I want to write a tool that can find all references of given variable based on index.scip locally, how could I parse index.scip file?

[i-ky]

I think you should refer to https://github.com/sourcegraph/scip

[varungandhi-src]

I want to write a tool that can find all references of given variable based on index.scip locally

Could you elaborate a bit more on the use case? Are you trying to write an automated refactoring tool? Are you trying to write a language server? Something else?

The basic idea would be as follows:

1. In a language of your choice, generate protobuf bindings (or use the ones vendored in this repository) which can be used to parse the index. You can also convert the index to JSON using scip print --json and consume that, but that may have a lot of overhead depending on the codebase size. This would avoid having to deal with protobuf, but you don't get the various types etc. provided by the bindings.
2. Construct hash tables with reference and definition information keyed on the symbol name, and an ordered container (e.g. a B-Tree or a sorted vector) for occurrences for searching, keyed by the source range. Or you can put the information into a database.
3. Based on the source range of the variable, find the matching occurrence, if any. Based on the occurrence's symbol name, look up the references table to see if there are references to that local variable or not.

[for-just-we]

3. variable, find the matching occurrence, if any. Based on the occurrence's symbol name, look up the references table to see if there are references to that local variable o

My goal is kind like

• given a variable: funcptr, I want to find where it is defined and find all other locations that refer the variable. The location can be given by relative/absolute path with line and col ranges.

• Also, I want to find all locations that refer a function, including function calls and assignment to func pointers.

So for now I want to build a database with this information, I tried universal ctags (only provide define information for function, customized types and global variables. No reference for this and no info for global variables) and scip-clang. I use scip to dump the index in json format. But the content in the json is kind confused to me? It seems provide a documents object for each c files and each document contains language, relativePath, occurance, symbols field. But I do not know how to automatically find any reference for a given variable or location with this json file. So could scip help me with this?

[varungandhi-src]

But the content in the json is kind confused to me? It seems provide a documents object for each c files and each document contains language, relativePath, occurance, symbols field. But I do not know how to automatically find any reference for a given variable or location with this json file. So could scip help me with this?

The scip CLI doesn't have any functionality for finding reference information. It's really meant for wholesale processing of indexes, not one-off operations like find references on a single variable.

That's why I described the approach earlier:

2. Construct hash tables with reference and definition information keyed on the symbol name, and an ordered container (e.g. a B-Tree or a sorted vector) for occurrences for searching, keyed by the source range. Or you can put the information into a database.

So you would iterate over all the documents in the JSON output, and construct these tables first. Once that's done, you can run the queries (like Find references) that you're interested in.

For example, for

given a variable: funcptr, I want to find where it is defined and find all other locations that refer the variable. The location can be given by relative/absolute path with line and col ranges

and

Also, I want to find all locations that refer a function, including function calls and assignment to func pointers.

The general approach would be that you first look in the occurrences table based on the source range. If you find a matching occurrence, use the symbol name for that (notice the an "occurrence" object will have a "symbol" field inside it), and look in the definitions/references tables based on what you're interested in.

If you haven't already done so, it might be useful to spend some time reading through the documentation in the scip.proto file to better understand the core concepts first. If something in the documentation is unclear, please let us know, and we can fix that.

[for-just-we]

But the content in the json is kind confused to me? It seems provide a documents object for each c files and each document contains language, relativePath, occurance, symbols field. But I do not know how to automatically find any reference for a given variable or location with this json file. So could scip help me with this?

The scip CLI doesn't have any functionality for finding reference information. It's really meant for wholesale processing of indexes, not one-off operations like find references on a single variable.

That's why I described the approach earlier:

2. Construct hash tables with reference and definition information keyed on the symbol name, and an ordered container (e.g. a B-Tree or a sorted vector) for occurrences for searching, keyed by the source range. Or you can put the information into a database.

So you would iterate over all the documents in the JSON output, and construct these tables first. Once that's done, you can run the queries (like Find references) that you're interested in.

For example, for

given a variable: funcptr, I want to find where it is defined and find all other locations that refer the variable. The location can be given by relative/absolute path with line and col ranges

and

Also, I want to find all locations that refer a function, including function calls and assignment to func pointers.

The general approach would be that you first look in the occurrences table based on the source range. If you find a matching occurrence, use the symbol name for that (notice the an "occurrence" object will have a "symbol" field inside it), and look in the definitions/references tables based on what you're interested in.

If you haven't already done so, it might be useful to spend some time reading through the documentation in the scip.proto file to better understand the core concepts first. If something in the documentation is unclear, please let us know, and we can fix that.

Than you, I am going to look at the protoco file. By the way, I have another question, is SourceGraph fully depend on scip-clang for C/C++ repos now? Like when you view source codes in github with SourceGraph, the infos all generated by scip-clang? Because scip-clang seem to require compile database. But I prefer more to index the repos without the compile database, and ctags seems could not generate reference info, is there other options that do not rely on compile database?

[varungandhi-src]

is SourceGraph fully depend on scip-clang for C/C++ repos now? Like when you view source codes in github with SourceGraph, the infos all generated by scip-clang?

At the moment, for C++, only repos which have specifically been indexed manually/in CI and have uploaded indexes support precise code navigation (like in the examples in the scip-clang README). Search-based code navigation still falls back to ctags.

But I prefer more to index the repos without the compile database

There needs to be a way of knowing the compiler invocations so that scip-clang can process the source code in the same way as the compiler did. Any other technique will necessarily have to rely on approximations.

and ctags seems could not generate reference info, is there other options that do not rely on compile database?

Another alternative is to use tree-sitter-cpp to parse the source code yourself and generate the needed information. But similar to ctags, this can't be fully precise.

[varungandhi-src]

I'm closing this issue as there is no clear action item for us here. If you have more questions, of course please free to ask them.

[yikengboy]

@varungandhi-src I want through symbol name semanticdb maven maven/com.qunhe.devops/moon.api 0.0.1-SNAPSHOT com/DateUtils#(). find symbol’s path and some information like Range, can you help me？

[varungandhi-src]

@yikengboy please file a new issue describing what you've tried and what you'd like to do.

[yikengboy]

@varungandhi-src

#244