New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: maximum recursion depth exceeded in cmp #122

Closed
gvanrossum opened this Issue Mar 13, 2015 · 7 comments

Comments

Projects
None yet
2 participants
@gvanrossum

gvanrossum commented Mar 13, 2015

I have this build.py:

from pybuilder.core import task, depends

@task
@depends('readline')
def sqlite():
    print('sqlite')

@task
@depends(sqlite)
def readline():
    print('readline')

@task
@depends(sqlite, readline)
def python():
    print('python')

Now when I run pyb sqlite it will correctly tell me I have a circular dependency. But when I run 'pyb python` I get "maximum recursion depth exceeded in cmp" which seems to be a bug in pyb.

(Also, is there an API to ask a task for its -- direct and indirect -- dependencies? I am trying to use pybuilder as a kind of Make for project artefacts and each task needs to use the artefacts of its dependent tasks.)

@mriehl

This comment has been minimized.

Show comment
Hide comment
@mriehl

mriehl Mar 13, 2015

Member

Thanks for bringing this to my attention. The circular dependency detection is very naive right now.
I have replaced it with a tarjan SCC on the dependency graph which passes any test case I could come up with, but it still needs some more polish, I'll try to get it out tomorrow.

There's no API to obtain a task's dependencies currently. Can you clarify what you'd expect / how you'd use it, maybe with some sample code - are you looking for the name of the dependencies (["sqlite", "readline"]) or for the actual objects?

Member

mriehl commented Mar 13, 2015

Thanks for bringing this to my attention. The circular dependency detection is very naive right now.
I have replaced it with a tarjan SCC on the dependency graph which passes any test case I could come up with, but it still needs some more polish, I'll try to get it out tomorrow.

There's no API to obtain a task's dependencies currently. Can you clarify what you'd expect / how you'd use it, maybe with some sample code - are you looking for the name of the dependencies (["sqlite", "readline"]) or for the actual objects?

@gvanrossum

This comment has been minimized.

Show comment
Hide comment
@gvanrossum

gvanrossum Mar 13, 2015

Thanks for the quick response!

Here's what I'm trying to accomplish -- I'm still not sure whether
pybuilder is useful for this or not. I'm not building Python packages --
I'm building a large app written in C, which uses many other libraries
written in C. Each dependency (e.g. sqlite) has its own checkout and we're
building from source. Once built, we collect the artifacts in a directory
named after the task name. Then the dependent task needs to reference these
artifacts, and ideally we don't want to encode the names of the
dependencies in the task body again. For example:

@task
@Depends(sqlite, readline, ...)
def python():
copy('sqlite')
copy('readline')
copy(...)

I would prefer to be able to write the body of the task as

for d in get_dependencies(python):
    copy(d.name)

(I'm okay with having to reference the task itself by name, though it would
be even nicer if there was a shortcut for that too.)

Am I making sense?

On Fri, Mar 13, 2015 at 12:24 PM, Maximilien Riehl <notifications@github.com

wrote:

Thanks for bringing this to my attention. The circular dependency
detection is very naive right now.
I have replaced it with a tarjan SCC on the dependency graph which passes
any test case I could come up with, but it still needs some more polish,
I'll try to get it out tomorrow.

There's no API to obtain a task's dependencies currently. Can you clarify
what you'd expect / how you'd use it, maybe with some sample code - are you
looking for the name of the dependencies (["sqlite", "readline"]) or for
the actual objects?


Reply to this email directly or view it on GitHub
#122 (comment).

--Guido van Rossum (python.org/~guido)

gvanrossum commented Mar 13, 2015

Thanks for the quick response!

Here's what I'm trying to accomplish -- I'm still not sure whether
pybuilder is useful for this or not. I'm not building Python packages --
I'm building a large app written in C, which uses many other libraries
written in C. Each dependency (e.g. sqlite) has its own checkout and we're
building from source. Once built, we collect the artifacts in a directory
named after the task name. Then the dependent task needs to reference these
artifacts, and ideally we don't want to encode the names of the
dependencies in the task body again. For example:

@task
@Depends(sqlite, readline, ...)
def python():
copy('sqlite')
copy('readline')
copy(...)

I would prefer to be able to write the body of the task as

for d in get_dependencies(python):
    copy(d.name)

(I'm okay with having to reference the task itself by name, though it would
be even nicer if there was a shortcut for that too.)

Am I making sense?

On Fri, Mar 13, 2015 at 12:24 PM, Maximilien Riehl <notifications@github.com

wrote:

Thanks for bringing this to my attention. The circular dependency
detection is very naive right now.
I have replaced it with a tarjan SCC on the dependency graph which passes
any test case I could come up with, but it still needs some more polish,
I'll try to get it out tomorrow.

There's no API to obtain a task's dependencies currently. Can you clarify
what you'd expect / how you'd use it, maybe with some sample code - are you
looking for the name of the dependencies (["sqlite", "readline"]) or for
the actual objects?


Reply to this email directly or view it on GitHub
#122 (comment).

--Guido van Rossum (python.org/~guido)

@mriehl

This comment has been minimized.

Show comment
Hide comment
@mriehl

mriehl Mar 13, 2015

Member

I understand.
The following is possible with the changes I have made to fix the circular dependency detection issue (I have yet to push them to github):

from pybuilder.reactor import Reactor
from pybuilder.core import task, depends

@task
@depends("readline")
def sqlite():
    pass

@task
def readline():
    pass

@task
@depends("sqlite")
def python():
    reactor = Reactor.current_instance()
    for dependency in reactor.execution_manager.collect_all_transitive_tasks([python.__name__]):
        copy(dependency.name)

This looks like it would solve your problem, doesn't it? I'd add some kind of wrapper, of course, since the code above makes a lot of assumptions about the pybuilder internals.

Member

mriehl commented Mar 13, 2015

I understand.
The following is possible with the changes I have made to fix the circular dependency detection issue (I have yet to push them to github):

from pybuilder.reactor import Reactor
from pybuilder.core import task, depends

@task
@depends("readline")
def sqlite():
    pass

@task
def readline():
    pass

@task
@depends("sqlite")
def python():
    reactor = Reactor.current_instance()
    for dependency in reactor.execution_manager.collect_all_transitive_tasks([python.__name__]):
        copy(dependency.name)

This looks like it would solve your problem, doesn't it? I'd add some kind of wrapper, of course, since the code above makes a lot of assumptions about the pybuilder internals.

@gvanrossum

This comment has been minimized.

Show comment
Hide comment
@gvanrossum

gvanrossum Mar 13, 2015

Yes, that would solve my problem. Thanks! (Another time I'd like to tell
you about my experience trying to understand pybuilder's docs.)

On Fri, Mar 13, 2015 at 1:44 PM, Maximilien Riehl notifications@github.com
wrote:

I understand.
The following is possible with the changes I have made to fix the circular
dependency detection issue (I have yet to push them to github):

from pybuilder.reactor import Reactorfrom pybuilder.core import task, depends
@task@depends("readline")def sqlite():
pass
@taskdef readline():
pass
@task@depends("sqlite")def python():
reactor = Reactor.current_instance()
for dependency in reactor.execution_manager.collect_all_transitive_tasks([python.name]):
copy(dependency.name)

This looks like it would solve your problem, doesn't it? I'd add some kind
of wrapper, of course, since the code above makes a lot of assumptions
about the pybuilder internals.


Reply to this email directly or view it on GitHub
#122 (comment).

--Guido van Rossum (python.org/~guido)

gvanrossum commented Mar 13, 2015

Yes, that would solve my problem. Thanks! (Another time I'd like to tell
you about my experience trying to understand pybuilder's docs.)

On Fri, Mar 13, 2015 at 1:44 PM, Maximilien Riehl notifications@github.com
wrote:

I understand.
The following is possible with the changes I have made to fix the circular
dependency detection issue (I have yet to push them to github):

from pybuilder.reactor import Reactorfrom pybuilder.core import task, depends
@task@depends("readline")def sqlite():
pass
@taskdef readline():
pass
@task@depends("sqlite")def python():
reactor = Reactor.current_instance()
for dependency in reactor.execution_manager.collect_all_transitive_tasks([python.name]):
copy(dependency.name)

This looks like it would solve your problem, doesn't it? I'd add some kind
of wrapper, of course, since the code above makes a lot of assumptions
about the pybuilder internals.


Reply to this email directly or view it on GitHub
#122 (comment).

--Guido van Rossum (python.org/~guido)

@mriehl

This comment has been minimized.

Show comment
Hide comment
@mriehl

mriehl Mar 13, 2015

Member

Great, I'll try to get it out asap then. Feedback on the docs is always welcome, though I fear we need an overhaul there :-/

Member

mriehl commented Mar 13, 2015

Great, I'll try to get it out asap then. Feedback on the docs is always welcome, though I fear we need an overhaul there :-/

mriehl added a commit that referenced this issue Mar 14, 2015

improve cyclic dependency detection
We now run a full tarjan algorithm to find out about possible dependency
cycles.
This resolves the bug reported in #122

mriehl added a commit that referenced this issue Mar 14, 2015

Add possibility to introspect task dependencies
During pybuilder execution (i.E. in a task or action) this can be used
to obtain the resolved dependencies of the task.
This solves the feature request brought forward in #122
@mriehl

This comment has been minimized.

Show comment
Hide comment
@mriehl

mriehl Mar 14, 2015

Member

Using pybuilder==0.10.54 should detect the cyclic dependency you reported (and any other you can come up with).
What's more, this now works:

from pybuilder.core import task, depends
from pybuilder.utils import get_all_dependencies_for_task


@task
@depends("sqlite", "readline")
def python(logger):
    logger.info("I depend on {dependencies}".format(
        dependencies=[dependency.name for dependency in get_all_dependencies_for_task(python)]))


@task
@depends("something_else")
def sqlite():
    pass


@task
def readline():
    pass


@task
def something_else():
    pass

and it outputs:

$ pyb python  
PyBuilder version 0.10.54
Build started at 2015-03-14 11:26:11
------------------------------------------------------------
[INFO]  Building test version 1.0.dev0
[INFO]  Executing build in /private/tmp/test
[INFO]  Going to execute task python
[INFO]  I depend on ['something_else', 'readline', 'sqlite']
------------------------------------------------------------
BUILD SUCCESSFUL
------------------------------------------------------------
Build Summary
             Project: test
             Version: 1.0.dev0
      Base directory: /private/tmp/test
        Environments: 
               Tasks: something_else [0 ms] sqlite [0 ms] readline [0 ms] python [0 ms]
Build finished at 2015-03-14 11:26:11
Build took 0 seconds (0 ms)

Thanks again for reporting this bug.

Member

mriehl commented Mar 14, 2015

Using pybuilder==0.10.54 should detect the cyclic dependency you reported (and any other you can come up with).
What's more, this now works:

from pybuilder.core import task, depends
from pybuilder.utils import get_all_dependencies_for_task


@task
@depends("sqlite", "readline")
def python(logger):
    logger.info("I depend on {dependencies}".format(
        dependencies=[dependency.name for dependency in get_all_dependencies_for_task(python)]))


@task
@depends("something_else")
def sqlite():
    pass


@task
def readline():
    pass


@task
def something_else():
    pass

and it outputs:

$ pyb python  
PyBuilder version 0.10.54
Build started at 2015-03-14 11:26:11
------------------------------------------------------------
[INFO]  Building test version 1.0.dev0
[INFO]  Executing build in /private/tmp/test
[INFO]  Going to execute task python
[INFO]  I depend on ['something_else', 'readline', 'sqlite']
------------------------------------------------------------
BUILD SUCCESSFUL
------------------------------------------------------------
Build Summary
             Project: test
             Version: 1.0.dev0
      Base directory: /private/tmp/test
        Environments: 
               Tasks: something_else [0 ms] sqlite [0 ms] readline [0 ms] python [0 ms]
Build finished at 2015-03-14 11:26:11
Build took 0 seconds (0 ms)

Thanks again for reporting this bug.

@mriehl mriehl closed this Mar 14, 2015

@gvanrossum

This comment has been minimized.

Show comment
Hide comment
@gvanrossum

gvanrossum Mar 16, 2015

Thanks for your quick response to both issues!!

On Sat, Mar 14, 2015 at 3:28 AM, Maximilien Riehl notifications@github.com
wrote:

Using pybuilder==0.10.54 should detect the cyclic dependency you reported
(and any other you can come up with).
What's more, this now works:

from pybuilder.core import task, dependsfrom pybuilder.utils import get_all_dependencies_for_task

@task@depends("sqlite", "readline")def python(logger):
logger.info("I depend on {dependencies}".format(
dependencies=[dependency.name for dependency in get_all_dependencies_for_task(python)]))

@task@depends("something_else")def sqlite():
pass

@taskdef readline():
pass

@taskdef something_else():
pass

and it outputs:

$ pyb python
PyBuilder version 0.10.54

Build started at 2015-03-14 11:26:11

[INFO] Building test version 1.0.dev0
[INFO] Executing build in /private/tmp/test
[INFO] Going to execute task python

[INFO] I depend on ['something_else', 'readline', 'sqlite']

BUILD SUCCESSFUL

Build Summary
Project: test
Version: 1.0.dev0
Base directory: /private/tmp/test
Environments:
Tasks: something_else [0 ms] sqlite [0 ms] readline [0 ms] python [0 ms]
Build finished at 2015-03-14 11:26:11
Build took 0 seconds (0 ms)

Thanks again for reporting this bug.


Reply to this email directly or view it on GitHub
#122 (comment).

--Guido van Rossum (python.org/~guido)

gvanrossum commented Mar 16, 2015

Thanks for your quick response to both issues!!

On Sat, Mar 14, 2015 at 3:28 AM, Maximilien Riehl notifications@github.com
wrote:

Using pybuilder==0.10.54 should detect the cyclic dependency you reported
(and any other you can come up with).
What's more, this now works:

from pybuilder.core import task, dependsfrom pybuilder.utils import get_all_dependencies_for_task

@task@depends("sqlite", "readline")def python(logger):
logger.info("I depend on {dependencies}".format(
dependencies=[dependency.name for dependency in get_all_dependencies_for_task(python)]))

@task@depends("something_else")def sqlite():
pass

@taskdef readline():
pass

@taskdef something_else():
pass

and it outputs:

$ pyb python
PyBuilder version 0.10.54

Build started at 2015-03-14 11:26:11

[INFO] Building test version 1.0.dev0
[INFO] Executing build in /private/tmp/test
[INFO] Going to execute task python

[INFO] I depend on ['something_else', 'readline', 'sqlite']

BUILD SUCCESSFUL

Build Summary
Project: test
Version: 1.0.dev0
Base directory: /private/tmp/test
Environments:
Tasks: something_else [0 ms] sqlite [0 ms] readline [0 ms] python [0 ms]
Build finished at 2015-03-14 11:26:11
Build took 0 seconds (0 ms)

Thanks again for reporting this bug.


Reply to this email directly or view it on GitHub
#122 (comment).

--Guido van Rossum (python.org/~guido)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment