Dependency Graphs

[rosafilgueira]

I have been reading the different types of dependencies graphs, and some tools that we can use for each of them.
So far, I've identified two types of dependency graphs:

1. Modules (and packages) dependency graph: Shows the dependencies among modules. Example of tools:
   1.1. Pydeps tool -- It seems that is the current standard. It gives us all the modules and packages.
   1.2 Modulegraph2: Modulegraph2 is a library for creating and introspecting the dependency graph between Python modules. The graph is created using static analisys from source and byte code.

2. Call Graphs: dependency graph that just shows which functions within a program are calling other functions (although it can also show other relationships such as definitions and file groupings). Example of tools:
   2.1 Pyan: static analysis, and constructs a directed graph of the objects in the combined source, and how they define or use each other. The graph can be output for rendering by GraphViz or yEd.

[dgarijo]

Thanks. I think this is an important distinction, and both could affect different types of similarity in code.

We will have to assess which of these graphs is indeed the one that is most useful for different tasks.

Maybe the conclusion for now is that both should be supported in code_inspector

[rosafilgueira]

For the call_graph (represents calling relationships between subroutines in a computer program) I've tested pyan3. A specific version is needed:

pip install pyan3==1.1.1

To test it with code_inspector: pyan3 code_inspector.py --uses --no-defines --colored --grouped --annotated --dot > code_inspector.dot

To test it with somef: pyan3 /Users/rosafilgueira/HW-Work/Research/somef/src/somef/*.py --uses --no-defines --colored --grouped --annotated --dot > somef_pyan3.dot

[rosafilgueira]

For the Modules (and packages) dependency graph, I've tried pydeps:

pip install pydeps

To test it with code_inspector: pydeps code_inspector.py -o code_inspector_modules_dependency.png -T png --show-deps > code_inspector_modules_dep.json

To test it with somef: pydeps /Users/rosafilgueira/HW-Work/Research/somef/src/somef/ -o somef_modules_dependency.png -T png --show-deps > somef_modules_dep.json

[dgarijo]

Awesome, I think it will probably make sense to have a distinction between module dependencies (external) and internal classes/files, right? Maybe through a command option. I will think on how to best integrate this together.

[rosafilgueira]

Definitely, but remember this:

• Module dependency graph: It tells us about the different modules - basically it finds the relationship among the imports for a given repository or file --> how interconnected their imports are. The graph is the import graph. Tool used: pydeps.
• Call graph (CG): It tells us about the different functions and methods (as the call graph) - basically finds the relationships among the functions for a given repository or file . The graph is the call graph--> which functions within a program are calling other functions. Tool used: pyan3
• Control flow graph (CFG): It tells us the flow or computation during the execution of programs or applications -- It provides finer "details" into the structure of the program as a whole, and of the subroutines in particular. Tool used: staticfg. Note: It has been already integrated.
• Directory structure tree --> It tells us the directory structure (subdirectories and files ) of a specific path. Developed by us : structure_tree.py. Note: It has been already integrated.

[dgarijo]

Agreed, these are all important from their own perspective. My remark probably applied to the firs bulletpoint (pydeps)

[rosafilgueira]

I have made a local experiment, in which I've created a src directory, and move there:

• init.py (I created this file manually for this test)
• structure_tree.py
• code_inspector.py
• staticfg

And later run the call_graph (pyan3) like this: pyan3 src/*.py --uses --no-defines --colored --grouped --annotated --dot > test.dot

Notice, how this program is able to detect which file has the MAIN :) !!

[dgarijo]

Excellent, then maybe we can apply this to the other types of apps in main_command. Exciting!

[rosafilgueira]

For the call_graph - I've found another tool: pycg --> this one generates directly the json.
They have a paper published ICSE 2021 paper.. It works ... but I am still experimenting with it.

They compare (in the paper) this tool (pycg) with the other one I've been using the last days (pyan3)

Install: pip install pycg

• Test 1: pycg code_inspector.py -o code_inspector_call_graph_pycg.json
• Test 2: pycg --package /Users/rosafilgueira/HW-Work/Research/somef/src/somef $(find /Users/rosafilgueira/HW-Work/Research/somef/src/somef -type f -name "*.py") -o somef_pycg.json

[rosafilgueira]

The last tool reviewed: code2graph - gitLab
Paper: https://ieeexplore.ieee.org/abstract/document/9000043

• pip install code2graph

• Test 1 : code2graph.py test/code_inspector/src/ --png --scope all (It generates a png with all the information aggregated, and one per *.py)

• 1.1. Aggregated PNG:

• 1.2. code_inspector.py PNG:

• Test 2: code2graph.py /Users/rosafilgueira/HW-Work/Research/somef/src/somef --png --scope all

• 2.1. Aggregated PNG:

• 2.2. main__.py PNG:

The above examples usage will generate one dot file and one png file for each python file in the PATH (e.g. test/code_inspector/src/ ) directory and in addition to that it will also generate one dot file and one png file for all the python files in that directory

Note: This tool has been previously compared with ENRE: A Tool Framework for Extensible eNtity
Relation Extraction - ICSE 2019 - GitHub

[dgarijo]

This is done now with the option --cl