Not detecting imports in python file and questions

[mariene]

Hi,
I run pydeps on my file with this following command line (with different parameters) :
pydeps JobScriptTemplate.py --max-bacon 6 --noise-level 2 --pylib

There is some imports in my file but it detects nothing, the graph is empty.
This file comes from a package and it imports a class from another directory in this package.
It imports :

import os
import sys
import time
import shutil
from commons.core.checker.RepetException import RepetException
from commons.core.sql.TableJobAdaptator import TableJobAdaptator
from commons.core.sql.DbFactory import DbFactory
from commons.core.sql.Job import Job

The directory 'commons' is like :

.
├── core
│   ├── checker
│   ├── launcher
│   └── sql
├── launcher
└── tools

and my file is on '/core/launcher' (each directory has a init.py file).

Morever I don't really understand the difference between --max-bacon and --noise-level. What is the effect of noise-level on max-bacon ?
I'm wondering if I can attribute a color for a folder level. How can I do this ? Because I want to run pydeps on different folders and I want to keep the same color for each folder.
And can I import and use pydeps in a python script ? I don't find any constructor in pydeps sources.

I thank you in advance for your help.
Mariene

[thebjorn]

Which version (pydeps --version) are you using?

I don't really understand the difference between --max-bacon and --noise-level.

--max-bacon determines how many "hops" from the root that should be included in the graph (max bacon of 6 will likely include the entire universe.. -- the name comes from the game "6 degrees of Kevin Bacon"), while --noise-level=n excludes nodes that have more than n incoming/outgoing arrows. --max-bacon increases the size of the graph, --noise-level decreases the size of the graph.

If I remember correctly, then graphviz/dot has a limitation of 1000 nodes (or maybe it was 10 000), and if you hit that limit there will be no output.

I'm wondering if I can attribute a color for a folder level.

The color assignment is done here https://github.com/thebjorn/pydeps/blob/master/pydeps/colors.py#L47 (it uses the part before the first dot in the module name and adjusts saturation depending on how many arrows go in/out of the node). Early versions of pydeps tried to use different colors for different modules, but my implementation(s) didn't work that well. Maybe you'll have better luck..?

can I import and use pydeps in a python script ?

Yes, check e.g. https://github.com/thebjorn/pydeps/blob/master/tests/test_cli.py#L25

I run pydeps on my file with this following command line (with different parameters) :
pydeps JobScriptTemplate.py --max-bacon 6 --noise-level 2 --pylib

I would suggest starting without any flags, i.e. pydeps JobScriptTemplate.py, although the preferred method is to call it on the root of the module: pydeps commons. Then you can increase --max-bacon if you don't reach all the modules you want to find. Changing --noise-level is almost never needed (setting it to 2 will remove most nodes - the default is 200).

PS: Python is not designed to support running scripts from sub-folders (mostly because Guido doesn't like it), so you're fighting against the system when you try to run a script that is two levels deep. In particular, you need to use absolute imports (like you're doing: from commons.core.sql.TableJobAdaptator import TableJobAdaptator) instead of relative imports (from ..sql.TableAdapter import ..). The implications for pydeps is that from a.b.c import d loads a.__init__, b.__init__, c.__init__, and d. If you try to import from sub-packages you'll easily get circular imports.

The solution is to create a myproj/setup.py file with entry points to the file:function you want to use as a script (like pydeps does: https://github.com/thebjorn/pydeps/blob/master/setup.py#L47). After you run pip install -e myproj the command is available directly from the shell, and you're free to use relative imports again.

[mariene]

Thank you for your quick reply.
I'm using pydeps v1.7.3. I will try your suggestions and I come back to you if I encounter any issue.

[mariene]

I tried without any flag and pydeps finds nothing for JobScriptTemplate.py. Then I tried with --max-bacon (0 to 30) and it's the same, I obtain an empty graph.

[thebjorn]

Does pydeps --show-dot JobScriptTemplate.py give any output? How about pydeps --show-raw-deps JobScriptTemplate.py?

[mariene]

With pydeps --show-dot. I obtain an empty graph and this :

digraph G {
    concentrate = true;
    rankdir = TB;
    node [style=filled,fillcolor="#ffffff",fontcolor="#000000",fontname=Helvetica,fontsize=10];
}

With pydeps --show-raw-deps. I obtain an empty graph too and this :

{
    "__main__": {
        "bacon": 0, 
        "imports": [
            "commons", 
            "commons.core", 
            "commons.core.launcher"
        ], 
        "name": "__main__", 
        "path": null
    }, 
    "commons": {
        "bacon": 1, 
        "imported_by": [
            "__main__"
        ], 
        "name": "commons", 
        "path": "/usr/local/REPET_linux-x64-2.5/commons/__init__.py"
    }, 
    "commons.core": {
        "bacon": 1, 
        "imported_by": [
            "__main__"
        ], 
        "name": "commons.core", 
        "path": "/usr/local/REPET_linux-x64-2.5/commons/core/__init__.py"
    }, 
    "commons.core.launcher": {
        "bacon": 1, 
        "imported_by": [
            "__main__"
        ], 
        "name": "commons.core.launcher", 
        "path": "/usr/local/REPET_linux-x64-2.5/commons/core/launcher/__init__.py"
    }
}

[thebjorn]

Hmm.. I'm afraid I can't reproduce what you're seeing. I've created the following directory tree:

(dev) go|c:\srv\tmp\pydeps28> tree commons
commons
|-- __init__.py
|-- core
|   |-- __init__.py
|   |-- checker
|   |   |-- RepetException.py
|   |   `-- __init__.py
|   |-- launcher
|   |   |-- JobScriptTemplate.py
|   |   `-- __init__.py
|   `-- sql
|       |-- DbFactory.py
|       |-- Job.py
|       |-- TableJobAdaptator.py
|       `-- __init__.py
|-- launcher
|   `-- __init__.py
`-- tools
    `-- __init__.py

6 directories, 12 files

with the following contents:

(dev) go|c:\srv\tmp\pydeps28> cat pydeps28.yaml
commons:
  __init__.py: ''
  core:
    __init__.py: ''
    checker:
      __init__.py: ''
      RepetException.py: |
        class RepetException: pass
    launcher:
      __init__.py: ''
      JobScriptTemplate.py: |
        import os
        import sys
        import time
        import shutil
        from commons.core.checker.RepetException import RepetException
        from commons.core.sql.TableJobAdaptator import TableJobAdaptator
        from commons.core.sql.DbFactory import DbFactory
        from commons.core.sql.Job import Job
    sql:
      __init__.py: ''
      DbFactory.py: |
        class DbFactory: pass
      Job.py: |
        class Job: pass
      TableJobAdaptator.py: |
        class TableJobAdaptator: pass
  launcher:
    __init__.py: ''
  tools:
    __init__.py: ''

If you grab yamldirs==1.1.7 (published a few seconds ago), you can save the yaml in a file (e.g. myfile.yaml) and reconstitute the directory with yamldirs myfile.yaml.

With the above, and the following command:

(dev) go|c:\srv\tmp\pydeps28> pydeps commons

or commands (I set the PYTHONPATH to the directory containing commons so Python will find the imports):

(dev) go|c:\srv\tmp\pydeps28> set PYTHONPATH=c:\srv\tmp\pydeps28
(dev) go|c:\srv\tmp\pydeps28> cd commons\core\launcher
(dev) go|c:\srv\tmp\pydeps28\commons\core\launcher> pydeps JobScriptTemplate.py

I get the same graph:

I can look at it further if you can change the code above to demonstrate your problem, and then attach the output of yamldirs commons.

[mariene]

I tried your example and I obtained the same result.

For my case, I created a minimal example that reproduce the issue REPET_linux-x64-2.5.zip.

You'll notice that the script is working for commons/core/launcher/Launcher.py but not for commons/core/launcher/JobScriptTemplate.py.

[thebjorn]

I'm guessing that is because you have syntax errors in that file:

(dev) go|c:\srv\tmp\REPET_linux-x64-2.5\REPET_linux-x64-2.5> python -c "import commons.core.launcher.JobScriptTemplate"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "commons\core\launcher\JobScriptTemplate.py", line 43
    @@cmdStart@@
     ^
SyntaxError: invalid syntax

[mariene]

You're absolutely right, it's @@cmdStart@@ and @@cmdFinish@@ that causes the error, I assume they exist because this script is used to generate python script that is submitted as a job (for a scheduler).Pydeps worked after removing them.

Thank you for your time.