Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

leoBridge commander open Leo-Editor file always fails #534

Closed
SegundoBob opened this issue Jul 22, 2017 · 44 comments
Closed

leoBridge commander open Leo-Editor file always fails #534

SegundoBob opened this issue Jul 22, 2017 · 44 comments
Assignees
Milestone

Comments

@SegundoBob
Copy link
Contributor

Creating a leoBridge commander works, but using the commander to open a file always returns None. This happens with both Python 2 and Python 3.

This fits the well established pattern of completely breaking leoBridge every few months.

Test Version

Leo 5.5, build 20170718062026, Tue Jul 18 06:20:26 CDT 2017
Git repo info: branch = master, commit = 911001d

Test Program leoBridgeOpenFails.py

#!/usr/bin/python
#coding=utf-8
import sys
import leo.core.leoBridge as leoBridge

def main():
    """ Module entry point
    """

    bridge = leoBridge.controller(gui='nullGui', silent=True,
        verbose=False, loadPlugins=False, readSettings=False)
    cmdrX = bridge.openLeoFile(sys.argv[1])
    print('bridge.openLeoFile({0}) returned {1}'.format(sys.argv[1], cmdrX))

if __name__ == "__main__":
    main()

Command Line

python3 leoBridgeOpenFails.py <Leo-Editor file pathname>

@SegundoBob
Copy link
Contributor Author

Fails with latest version.

Test Version

Leo 5.5, build 20170721094758, Fri Jul 21 09:47:58 CDT 2017
Git repo info: branch = master, commit = 44a7d3a

@vitalije
Copy link
Contributor

I'll look into immediately. Bisecting gave 83575d4 that chaned bridge.getLeoID()

@vitalije
Copy link
Contributor

Fixed at 821c750. Tested by script of @SegundoBob

@SegundoBob
Copy link
Contributor Author

Thanks. Revision 821c750 works for me.

@edreamleo
Copy link
Member

edreamleo commented Jul 22, 2017 via email

@xgid
Copy link

xgid commented Jul 25, 2017

Hi to all,

I saw yesterday the need for adding some nodes to a Leo document by running an script from outside Leo and today I've seen that this issue has something to do with it and has just been fixed, so I've tried to run the @SegundoBob script as a first test, but I got this error:

D:\proj\code\leo\scripts>python.exe leoBridgeOpen.py hello.leo
Traceback (most recent call last):
  File "leoBridgeOpen.py", line 4, in <module>
    import leo.core.leoBridge as leoBridge
ModuleNotFoundError: No module named 'leo'

It seems logical because I have not installed leo as a module in my Python installation, so I've made a search on the question at Leo site and found the page about leoBridge. But, as incredible as it may seem, the first chapter in that page gives a basic example which is nearly identicall to @SegundoBob example, but no clue as how one should make "leo" module reachable in the first place from outside Leo. And I've seen no setup.py file in the root folder of Leo installation either, so I don't know how should I do it.

Is there any documentation about this in Leo web site that I'm missing?

@tbnorth
Copy link
Contributor

tbnorth commented Jul 25, 2017 via email

@xgid
Copy link

xgid commented Jul 27, 2017

Thanks Terry, it worked.

Anyway, shouldn't it be documented somewhere in the docs? It is not evident for any end user that installing Leo does not install leo modules in your current python installation.

By the other side: should it be better to have a setup.py to the root folder of leo installation? I could help by contributing a minimal one if you wish. AFAIK, this would allow even having "Editable" Installs of the "package".

Xavier

@xgid
Copy link

xgid commented Jul 27, 2017

Oh, and another question: would leoBridge allow me to dinamically add nodes to an already opened Leo document?
I mean to add nodes to a Leo document from outside Leo while I have the document open in Leo editor.

@edreamleo
Copy link
Member

edreamleo commented Jul 27, 2017 via email

@tbnorth
Copy link
Contributor

tbnorth commented Jul 28, 2017 via email

@tbnorth
Copy link
Contributor

tbnorth commented Jul 28, 2017 via email

@xgid
Copy link

xgid commented Jul 28, 2017

Thanks to all for your answers.

Xavier clearly wants to change an outline open in a Leo GUI instance from code triggered outside that instance. The leoserver plugin used to do that, but got retired, mod_http supports that function now.

Exactly, that's the idea! So I've tried the mod_http trick but got this error shown in the Log pane:

127.0.0.1 - - [28/Jul/2017 13:30:55] "GET /?cmd=g.es%28%27Test%20message%21%27%29 HTTP/1.1" 404 -

Traceback (most recent call last):
  File "C:\dev\Leo-5.4\leo\plugins\mod_http.py", line 421, in send_head
    f = self.leo_actions.get_response()
  File "C:\dev\Leo-5.4\leo\plugins\mod_http.py", line 903, in get_response
    return self.exec_handler.get_response()
  File "C:\dev\Leo-5.4\leo\plugins\mod_http.py", line 935, in get_response
    ans = self.proc_cmds()
  File "C:\dev\Leo-5.4\leo\plugins\mod_http.py", line 969, in proc_cmds
    ans = valuespace.eval_text(g.app.commanders()[c_idx], cmd)
  File "C:\dev\Leo-5.4\leo\plugins\valuespace.py", line 367, in eval_text
    vsc = get_vs(c)
  File "C:\dev\Leo-5.4\leo\plugins\valuespace.py", line 299, in get_vs
    return controllers[c.hash()]
KeyError: 'd:\\proj\\code\\devenv\\devenv.leo'
error: uncaptured python exception, closing channel <leo.plugins.mod_http.RequestHandler connected 127.0.0.1:53148 at 0xeb77210> (<class 'KeyError'>:'d:\\proj\\code\\devenv\\devenv.leo' [C:\dev\Leo-5.4\leo\plugins\mod_http.py|a_read|1185] [C:\dev\Leo-5.4\leo\plugins\mod_http.py|handle_read_event|1112] [C:\dev\Anaconda3.6\lib\asyncore.py|handle_read_event|423] [C:\dev\Anaconda3.6\lib\asynchat.py|handle_read|171] [C:\dev\Leo-5.4\leo\plugins\mod_http.py|found_terminator|1142] [C:\dev\Leo-5.4\leo\plugins\mod_http.py|handle_request_line|1131] [C:\dev\Leo-5.4\leo\plugins\mod_http.py|do_GET|1068] [C:\dev\Leo-5.4\leo\plugins\mod_http.py|handle_data|1106] [C:\dev\Leo-5.4\leo\plugins\mod_http.py|send_head|421] [C:\dev\Leo-5.4\leo\plugins\mod_http.py|get_response|903] [C:\dev\Leo-5.4\leo\plugins\mod_http.py|get_response|935] [C:\dev\Leo-5.4\leo\plugins\mod_http.py|proc_cmds|969] [C:\dev\Leo-5.4\leo\plugins\valuespace.py|eval_text|367] [C:\dev\Leo-5.4\leo\plugins\valuespace.py|get_vs|299])

As you can see, the cmd I was trying to run was just cmd=g.es('Test message!'), so the full URL was:

http://localhost:8130/_/exec/?cmd=g.es%28%27Test%20message%21%27%29

Should I open a bug report about that or am I doing anything wrong?

(For the record, accessing the plain URL http://localhost:8130 does properly show the list of open files in the Leo GUI instance).

By the other side, I have also tried to run a very basic script using leoBridge (as I have another use case where it may be helpful) and I must say that it is not working as I initially thought.

This basic script:

import sys
import leo.core.leoBridge as leoBridge

def main():
    """ Module entry point
    """

    bridge = leoBridge.controller(gui='nullGui', silent=False,
        verbose=False, loadPlugins=True, readSettings=True)
    g = bridge.globals()
    print('g: ' + repr(g))
    print('bridge.isOpen: ' + str(bridge.isOpen()))

    c = bridge.openLeoFile(sys.argv[1])
    print('bridge.openLeoFile({0}) returned {1}'.format(sys.argv[1], c))

    print(c.p)
    
if __name__ == "__main__":
    main()

Gives this output:

g: False
bridge.isOpen: False
bridge.openLeoFile(D:\proj\code\leo\scripts\hello.leo) returned None
Traceback (most recent call last):
  File "leoBridgeOpen.py", line 22, in <module>
    main()
  File "leoBridgeOpen.py", line 19, in main
    print(c.p)
AttributeError: 'NoneType' object has no attribute 'p'

@tbnorth
Copy link
Contributor

tbnorth commented Jul 28, 2017 via email

@xgid
Copy link

xgid commented Jul 28, 2017

Thanks Terry, now it works!

I already had the @bool http_allow_remote_exec = True, but did not have the valuespace plugin enabled in my @enabled-plugins. It seems to be necessary. Should it be documented somewhere?

Any idea about the leoBridge module errors?

@tbnorth
Copy link
Contributor

tbnorth commented Jul 29, 2017 via email

@xgid
Copy link

xgid commented Jul 29, 2017

Thanks Terry.
Regarding the leoBridge module errors: I was refering to the output of the script I commented above.

Specifically:

  • bridge.globals() is returning False,
  • bridge.isOpen() is returning False also,
  • bridge.openLeoFile(sys.argv[1]) with a valid filename is returning None.

So leoBridge is not working at all for me. I've also tried with the examples at leoeditor.com with similar results.

Xavier

@vitalije
Copy link
Contributor

@xgid which revision do you use?

@xgid
Copy link

xgid commented Jul 29, 2017

Oh, of course!

Leo 5.5, build 20170721094758, Fri Jul 21 09:47:58 CDT 2017
Git repo info: branch = master, commit = 821c750de356
Python 3.6.1, PyQt version 5.6.2
Windows 10 AMD64 (build 10.0.15063) SP0
isPython3: True
caching enabled

@vitalije
Copy link
Contributor

@xgid

Python 3.6.1, PyQt version 5.6.2

suggests that you have copied version data from Leo log. Please provide copy of all output generated by your script from command line. When starting leoBridge there is no PyQt version but dummy version of nullGui.

I can't reproduce behavior that you have reported. Perhaps you are not using same version in leoBridge and normal Leo application. Otherwise I have no idea what to check else.

@xgid
Copy link

xgid commented Aug 1, 2017

Please provide copy of all output generated by your script from command line.

Well, the problem is that the output I provided in my previous comment is all the output I get from running my script. Even if I set verbose=True.

How can I get any other output for my script?

@vitalije
Copy link
Contributor

vitalije commented Aug 1, 2017

Please pay attention to the line that I quoted. It seems that the log you have copied did not originate from console and script that was using gui='nullGui', but instead from something that was using PyQt5.6.2. As I can see in your script you did provide keyword argument gui='nullGui'. Is this argument still in your script? Please, check line by line the output and what have you posted here and confirm that they are the same.

@vitalije
Copy link
Contributor

vitalije commented Aug 1, 2017

When I say all the output, I mean Leo version details. You have send them here in another message not the one you pointed in your last message.

@xgid
Copy link

xgid commented Aug 1, 2017

I'm sorry I have not been clear enough, there was too much info in the message I pointed to.

To make it clear again:

With this exact script:

import sys
import leo.core.leoBridge as leoBridge

def main():
    """ Module entry point
    """

    bridge = leoBridge.controller(gui='nullGui', silent=False,
        verbose=True, loadPlugins=True, readSettings=True)
    g = bridge.globals()
    print('g: ' + repr(g))
    print('bridge.isOpen: ' + str(bridge.isOpen()))

    c = bridge.openLeoFile(sys.argv[1])
    print('bridge.openLeoFile({0}) returned {1}'.format(sys.argv[1], c))

    print(c.p)
    
if __name__ == "__main__":
    main()

This is exactly the output I get in the command line:

D:\proj\code\leo\scripts>python.exe leoBridgeOpen.py D:\proj\code\leo\scripts\hello.leo
g: False
bridge.isOpen: False
bridge.openLeoFile(D:\proj\code\leo\scripts\hello.leo) returned None
Traceback (most recent call last):
  File "leoBridgeOpen.py", line 21, in <module>
    main()
  File "leoBridgeOpen.py", line 18, in main
    print(c.p)
AttributeError: 'NoneType' object has no attribute 'p'

No more and no less!

@xgid
Copy link

xgid commented Aug 1, 2017

And this is my current version of Leo, of course copied from the Log pane of a normal Qt GUI instance of Leo because I cannot get if from the command line!

Leo Log Window
Leo 5.5, build 20170731192022, пон, 31. јул 2017.  19:20:22 CEST
Git repo info: branch = master, commit = 754cf6957439
Python 3.6.1, PyQt version 5.6.2
Windows 10 AMD64 (build 10.0.15063) SP0
isPython3: True
caching enabled

@vitalije
Copy link
Contributor

vitalije commented Aug 1, 2017

Please change your script to the following:

import sys
print('\n'.join(sys.path))
import leo
print('leo path', leo.__file__)
import leo.core.leoBridge as leoBridge

def main():
    """ Module entry point
    """

    bridge = leoBridge.controller(gui='nullGui', silent=False,
        verbose=True, loadPlugins=True, readSettings=True)
    g = bridge.globals()
    print('g: ' + repr(g))
    print('bridge.isOpen: ' + str(bridge.isOpen()))

    c = bridge.openLeoFile(sys.argv[1])
    print('bridge.openLeoFile({0}) returned {1}'.format(sys.argv[1], c))

    print(c.p)
    
if __name__ == "__main__":
    main()

And send here the output.

@xgid
Copy link

xgid commented Aug 2, 2017

Thanks @vitalije, this is the output of the new script:

D:\proj\code\leo\scripts
c:\dev\leo-5.4
C:\dev\Anaconda3.6\python36.zip
C:\dev\Anaconda3.6\DLLs
C:\dev\Anaconda3.6\lib
C:\dev\Anaconda3.6
C:\dev\Anaconda3.6\lib\site-packages
C:\dev\Anaconda3.6\lib\site-packages\Sphinx-1.5.6-py3.6.egg
d:\proj\code\xutil
d:\proj\external\starred
C:\dev\Anaconda3.6\lib\site-packages\win32
C:\dev\Anaconda3.6\lib\site-packages\win32\lib
C:\dev\Anaconda3.6\lib\site-packages\Pythonwin
leo path c:\dev\leo-5.4\leo\__init__.py
g: False
bridge.isOpen: False
bridge.openLeoFile(D:\proj\code\leo\scripts\hello.leo) returned None
Traceback (most recent call last):
  File "leoBridgeOpen.py", line 24, in <module>
    main()
  File "leoBridgeOpen.py", line 21, in main
    print(c.p)
AttributeError: 'NoneType' object has no attribute 'p'

As commented earlier, in order to have access to leo modules I had to set my PYTHONPATH env var to this value:

PYTHONPATH=c:\dev\leo-5.4

So I think that this is not the cause of the problem here... But don't know how to go further, so all the help is really welcome!

@xgid
Copy link

xgid commented Aug 2, 2017

FYI, I have also tried to run the script in a fresh new Python 3.5 conda enviroment, but with similar results:

(leo35) D:\proj\code\leo\scripts>python.exe leoBridgeOpen.py D:\proj\code\leo\scripts\hello.leo
D:\proj\code\leo\scripts
c:\dev\leo-5.4
C:\dev\Anaconda3\envs\leo35\python35.zip
C:\dev\Anaconda3\envs\leo35\DLLs
C:\dev\Anaconda3\envs\leo35\lib
C:\dev\Anaconda3\envs\leo35
C:\dev\Anaconda3\envs\leo35\lib\site-packages
leo path c:\dev\leo-5.4\leo\__init__.py
g: False
bridge.isOpen: False
bridge.openLeoFile(D:\proj\code\leo\scripts\hello.leo) returned None
Traceback (most recent call last):
  File "leoBridgeOpen.py", line 24, in <module>
    main()
  File "leoBridgeOpen.py", line 21, in main
    print(c.p)
AttributeError: 'NoneType' object has no attribute 'p'

@vitalije
Copy link
Contributor

vitalije commented Aug 2, 2017

@xgid, now we are getting somewhere. It appears that the Leo version that your script uses for launching bridge is in C:\dev\leo-5.4. I assume it is Leo5.4 not the latest git version. Check that you are not using old version for leo bridge, while newest version for normal use of Leo application. To check what is your Leo-git installation you may execute following script in Leo:

g.es(g.os_path_dirname(g.app.leoDir))

It should print in Log pane path to your Leo git installation that contains latest version. Then set PYTHONPATH to that path and try your script again.

Vitalije

@xgid
Copy link

xgid commented Aug 2, 2017

Thanks @vitalije. I'm sorry for the confusion, but despite the name the Leo version at C:\dev\leo-5.4 is the latest git version. The problem is that I first installed Leo using the windows installer when it was the official version 5.4 and latter on I changed to the latest git version in the same path without renaming it... so the name is now quite confusing!

Currently I only have the latest git version available anywhere in my system, so there's no possibility to be running an older version. How I wish that the cause of my problem was just that simple! What a shame! 😞

@vitalije
Copy link
Contributor

vitalije commented Aug 2, 2017

Ok. try to execute following command in D:\proj\code\leo\scripts:

python -c "import leo.core.leoGlobals as g;g.pr(g.gitInfo())"

And post here the output.

@xgid
Copy link

xgid commented Aug 2, 2017

D:\proj\code\leo\scripts>python -c "import leo.core.leoGlobals as g;g.pr(g.gitInfo())"
('master', '8d3c368f702b')

@vitalije
Copy link
Contributor

vitalije commented Aug 2, 2017

This is very strange! On my linux when I execute your leoBridgeOpen.py script, I have standard Leo signon message:

reading settings in /media/vitalije/.../trunk-git/leo/config/leoSettings.leo
reading settings in /media/vitalije/.../trunk-git/leo/config/myLeoSettings.leo

Leo 5.5, build 20170731192022, пон, 31. јул 2017.  19:20:22 CEST
Git repo info: branch = master, commit = 754cf6957439
Python 3.4.3, LeoGui: dummy version
linux
** isPython3: True
** caching enabled

g: <module 'leo.core.leoGlobals' from '/media/vitalije/.../leo/core/leoGlobals.py'>
bridge.isOpen: True
reading settings in /media/vitalije/.../leo/config/myLeoSettings.leo
bridge.openLeoFile(leo/config/myLeoSettings.leo) returned Commander 139728818663944: '/media/vitalije/.../leo/config/myLeoSettings.leo'
<pos 139728818726504 childIndex: 0 lvl: 0 key: 139728818782672:0 @chapters>

Maybe @edreamleo will have some idea what is going on.
I would suggest you to enable traces in leo/core/leoApp.py and in leo/core/leoBridge.py. Set loadPlugins=False in leoBridgeOpen.py script and if you can try using pdb to run debuger in your script and then go step by step until you find from where leoBridge.controller returns None.

@vitalije
Copy link
Contributor

vitalije commented Aug 2, 2017

@edreamleo, please have look at this discussion with @xgid . His output is missing usual Leo signon message, and leoBridge.controller returns None. What can prevent printing usual signon message? It seems that the problem happens before printing signon.

@xgid
Copy link

xgid commented Aug 2, 2017

Thanks @vitalije. At least I got some time to debug the code and I've finally found at least 3 bugs and have been able to fix 2 of them so my script is now working.

I'll try to explain the bugs as clearly as possible:

The problem was in the call to BridgeController.getLeoID() from BridgeController.initLeo() in leoBridge.py line 141:

if not self.getLeoID(): return

This call was returning None in my case, so the initialization did not complete.

This method happens to try get LeoID by calling three different functions:

  • getIDFromSys
  • getIDFromFile
  • getIDFromEnv

The first one tries to see if sys.LeoID has been defined, which according to the comments seems to be done by Python's sitecustomize.py file, which I do not have. Nothing to say about this.

The second one looks for a .leoID.txt file in the following locations:

  • g.app.homeDir
  • g.app.globalConfigDir
  • g.app.loadDir

But it is the case that I have the file in none of them! My .leoID.txt file is located in the .leo folder under my user home directory, so the function was not finding it.

I think that this is the first bug. To fix it, I have just added g.app.homeLeoDir in the list of paths to look for in leoBridge.py line 270:

for theDir in (g.app.homeDir, g.app.homeLeoDir, g.app.globalConfigDir, g.app.loadDir):

But that did not fully fix the problem, because there's still another bug in that function: a return is missing after line 283 to get out of the loop once g.app.leoID has been set. Otherwise, it's value is set back to None (at line 285) when checking the remaining paths.

And finally, the third function getIDFromEnv is just looking at the environment var USER which happens to be empty in windows. The environment var used in Windows for that purpose is USERNAME, so I think this is the third bug.

As a final reflection about all this "experience": I think that it should be easier to debug problems like this one without the need of a debugger. It would be REALLY useful to have a "verbose" flag which can be activated when running Leo that could make Leo show detailed traces of the code being run. The verbose argument in the call to leoBridge.controller() was not showing any trace at all. I find it too much complicated to have to set trace = True in every relevant function I find, run the code again, find another trace to be activated ... and so on.

But I don't have enough expertise in Python to say if this is an easy matter... I guess it is not.

Thanks again for all your support, @vitalije.

@xgid
Copy link

xgid commented Aug 2, 2017

For the record, my original script is now showing this other (expected) output:

D:\proj\code\leo\scripts>python.exe leoBridgeOpen.py D:\proj\code\leo\scripts\hello.leo
reading settings in C:\dev\leo-5.4\leo\config\leoSettings.leo
reading settings in C:\Users\sysadmin\.leo\myLeoSettings.leo
global config directory : C:\dev\leo-5.4\leo\config
home directory : C:\Users\sysadmin

Leo 5.5, build 20170731233644, Mon Jul 31 23:36:44 CDT 2017
Git repo info: branch = master, commit = 8d3c368f702b
Python 3.5.2, LeoGui: dummy version
Windows 10 AMD64 (build 10.0.15063) SP0
** isPython3: True
** caching enabled

g: <module 'leo.core.leoGlobals' from 'c:\\dev\\leo-5.4\\leo\\core\\leoGlobals.py'>
bridge.isOpen: True
reading settings in D:\proj\code\leo\scripts\hello.leo
bridge.openLeoFile(D:\proj\code\leo\scripts\hello.leo) returned Commander 67601840: 'D:\\proj\\code\\leo\\scripts\\hello.leo'
<pos 46581840 childIndex: 0 lvl: 0 key: 67686224:0 Setup>

vitalije added a commit that referenced this issue Aug 3, 2017
…getIDFromEnv

as suggested by XGiD from github. See #534 discussion.

Leo build: 20170803065322
@vitalije
Copy link
Contributor

vitalije commented Aug 3, 2017

@xgid, I am glad that you worked it out. Thanks to your comments and suggestions I have made some improvements in leoBridge. First of all instead of returning None silently, when unable to find LeoID, now bridge will throw ValueError complaining that it wasn't able to set LeoID.
Next, I have changed g.app.homeDir to g.app.homeLeoDir. In the documentation there is no mention of home directory containing .leoID.txt file. So I presume it was just typo. And finally in getIDFromEnv, I have added check for USERNAME also as you have suggested.

@edreamleo
Copy link
Member

edreamleo commented Aug 3, 2017 via email

@edreamleo
Copy link
Member

edreamleo commented Aug 3, 2017 via email

@edreamleo
Copy link
Member

edreamleo commented Aug 3, 2017 via email

@xgid
Copy link

xgid commented Aug 4, 2017

Thanks @vitalije and @edreamleo for your commits to fix this so fast! Now it's working perfectly, so I'm gonna try some scripts to see how far I can get.

@xgid
Copy link

xgid commented Aug 4, 2017

When run externally, "@test leoBridge init logic" test passes, after a minor tweak in the test code in leoImport.py. It fails, as expected, when leoID.txt does not exist. As always, this test is skipped when running tests locally.

Regarding this comment: Would it be possible to change that test so the code is run using a separate process (by means of the BackgroundProcessManager or using Popen directly, I don't know) and inspecting the expected output? This way it would not need to be skipped when run locally.

@edreamleo
Copy link
Member

edreamleo commented Aug 5, 2017 via email

@edreamleo
Copy link
Member

edreamleo commented Aug 5, 2017 via email

@edreamleo edreamleo self-assigned this Sep 6, 2017
@edreamleo edreamleo added this to the 5.6 milestone Sep 6, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants