Layout does not work when edges exist in the model that are along the heirarchy tree.

[finger563]

I.e. edges from a parent node to one of its children, grand-children, etc. or the reverse.
Example:

When at least one such edge exists in the model, the layout does not do anything (i.e. no nodes are moved).

[ugurdogrusoz]

Correct, we consider such edges illegal and do not do anything. You could remove such edges prior to layout and restore them back when layout is finished.

[finger563]

Hmm, wouldn't it be easier (since they don't affect layout much) for that to happen internally within the layout routine? :)

[ugurdogrusoz]

CoSE is a compound graph layout algorithm and such edges are illegal for compound graphs by definition. Feel free to make the necessary changes and make a PR though.