including source maps, to allow for debugging python with in-browser debugger

[glyph]

I've been trying to debug #121 and it has been very challenging to do so because it's not possible to get a JavaScript debugger to set a breakpoint in a Python file.

If you could easily do ahead-of-time compilation (emitting, let's say, .pyc.js files next to the .py files), it would at least be possible to set breakpoints in the generated JavaScript source. Even better, with pre-generated source files we could potentially have source maps which would facilitate actually debugging Python code with a JavaScript debugger.

Is this already possible? I searched the documentation, but maybe I'm just not looking for the right string.

[ghost]

A better solution might be an optional debugger toolbar that you could attach to and pull out of a web page but right now that's a pipe dream IMO.

[glyph]

@JamesHutchison Do you mean that Brython would implement its own parallel debugging toolchain? All my browsers already have an optional debugger toolbar built into the browser itself :).

[ghost]

yeah, like the console but with debugging functionality. Kind of like the django debug toolbar.

[glyph]

@JamesHutchison This would be much, much worse than just supporting source maps. It would be years of work and a huge ongoing maintenance burden to maintain a parallel debugging and instrumentation infrastructure. Browsers have increasingly sophisticated tooling and Brython should just leverage it.

[ghost]

Source maps aren't actually an option. For one, they only work on pre-compiled javascript files (you can't use .html files that contain python or any dynamically generated code for example). Two, you're still debugging javascript, thus you would at best know where you are at in python. It would still take a lot of effort to investigate values or step through code.

Three, I hate the debuggers in browsers.

[glyph]

You can use source maps on dynamically generated code (at least with firefox) by setting a URL: http://blog.getfirebug.com/2009/08/11/give-your-eval-a-name-with-sourceurl/ - the docs for it look pretty thin though so I might be misinterpreting.

I'm not saying that browser debuggers are necessarily the best ever, but there are operations like "set a breakpoint" which would be next to impossible from within Brython itself if it isn't completely rewritten to support that, but the browser debugger gives you "for free". You're right that there are other issues, like sanely representing Python's variable scope from within the JS debugger's view.

[glyph]

Aah, here's something a bit more authoritative: http://stackoverflow.com/questions/14375567/getting-source-maps-working-with-evaluated-code

[ghost]

I'm not denying that a debug toolbar would be a lot of effort. That's why I said it was a pipe dream :)

[glyph]

@JamesHutchison - I suppose I might have overstated the case a bit. I actually think a "debug toolbar" project would be a super useful thing - there are lots of Brython-specific things one might want to debug; for example, visualizing sys.modules. I just wouldn't want to do that to the exclusion of making things work more nicely on browser debuggers :).

[jsbueno]

OTOH,
Brython is getting Python compatibility to the level of frame objects - it
is rather possible that in a couple
months, or even weeks, a proper debuger in python code (well, "brython)
could run and debug brython code in the browser.

On 4 February 2015 at 04:09, Glyph notifications@github.com wrote:

@JamesHutchison https://github.com/JamesHutchison - I suppose I might
have overstated the case a bit. I actually think a "debug toolbar" project
would be a super useful thing - there are lots of Brython-specific things
one might want to debug; for example, visualizing sys.modules. I just
wouldn't want to do that to the exclusion of making things work more
nicely on browser debuggers :).

—
Reply to this email directly or view it on GitHub
#123 (comment).

[glyph]

@jsbueno - Frame objects aren't the problem. The way that a debugger works involves interacting with the runtime, and brython still relies heavily on the browser runtime (it does not, and probably can not, have its own bytecode and instruction format). On native platforms you can work around this by simply using multiple threads and suspending some of them, but in the browser you only have the main event loop and "workers" which can't draw any UI.

[ghost]

Being able to analyze the state of the python environment when a you have an uncaught exception would be nice.

But yes, I don't really see it being possible to pause/step/resume execution unless Brython is re-written to use a state machine with byte code. Which might end up being molasses slow.

[glyph]

I should probably not under-emphasize the "optimization" aspect of this. Loading my pretty basic fork of Deferred takes on the order of 5 seconds of CPU time on a fairly fast, modern computer; most of that seems to be spent translating the Python to JavaScript. Being able to do this ahead of time and cache it would be really great. Perhaps Brython could even do this automatically using LocalStorage, looking at caching headers and only re-requesting Python source if it's changed, just like real Python does automatically with PYC files? (Being able to do it truly ahead of time is still useful for production websites that want the first visit to be snappy, of course.)

[glyph]

On the performance side of things I think this is closely related to #117 .

[ghost]

The thing is, when you import a module it is stored as a bunch of objects, which when converted to json takes up a lot of space. On top of that you have to remove the circular references and then re-build them again. The optimal solution for .py files seems to be to leave them as-is and just use #117 so the most common ones aren't a daisy chain of ajax calls.

[glyph]

I don't think I understand this comment. In other Python runtimes, .pyc files are not serialized representations of the objects within a module, they're just an optimized form of the code. It sounds like what you're talking about would be importing the module and then saving the imported module. What I'm talking about is translating the javascript without executing it, and then putting it into a file which, when executed, results in the same module again.

In other words, optimizing only the compilation time, not the import time. Optimizing import time could then be further accomplished by simply concatenating the .pyc.js files together.

[ghost]

I think I get you. When I looked into this earlier I was focusing on saving the actual module object that gets created when you do an import, but if I understand what you're saying, you mean changing this:

       <set-up code for root node>

        var js = root.to_js()
        eval(js)

into something like this:

        var js
        if (js_cache_exists(module)) {
            js = get_js_cache(module)
        } else {
            <set-up code for root node>
            ...
            js = root.to_js()
        }
        eval(js)

That might provide some useful savings, especially for the re module.

[glyph]

Yeah, something like that.

I'm actually thinking something more like:

var module_function = eval(js);
var module_globals = {};
module_function(module_globals);

so that things like reload() can be made to seamlessly work.

While it would definitely help with the re module, I'm mostly concerned about 3rd-party libraries. If every additional module requires an extra second or so of translation time, Brython users are going to strongly feel that they should optimize by writing everything themselves in one giant module and avoiding imports. This is obviously a terrible pressure to put on software design; basically the opposite of good practice :-). It's fine if it's slow to reload the page while developing, but when deploying to a production site, loading that page should be as quick as possible. In the process of developing software to be quick, of course, you should want to use as many modules as makes sense :).

[ghost]

Sounds like we need to extend the VFS functionality:

• Python modules are compiled to JS and saved to a single VFS file
• A dict maps module names to VFS URI (sys.vfs_map?)
• VFS files are downloaded on-demand

[ghost]

For the first suggestion, I have found out that compiled code is between 5x-10x larger than the python version. The parser/compiler is very fast, so the download time might take longer than the time it takes to parse and compile the python code.

for the third suggestion, I have been working on a brython_loader.js file, which can be used to dynamically load brython_dict.js OR brython.js (and possibily py_VFS) OR all the py_*.js file (and possibly py_VFS). This could be expanded by providing extra loader options.

[glyph]

For the first suggestion, I have found out that compiled code is between 5x-10x larger than the python version.

You could minify and gzip the compiled code for better results. Likely it compresses very well :).

The parser/compiler is very fast, so the download time might take longer than the time it takes to parse and compile the python code.

This has not been my experience; on localhost, where the transfer is instant, it still takes me about 5s to load a page with 16 or so modules on it, which is 300ms per module. That is not a very fast compiler :-).

Plus, you pay the execution cost of the transpiler on every page-load, but you only pay the download cost of cached files once.

[ghost]

I agree with glyph.

Bill does the code in node_bridge.js work for you? When I try it, I get this:

var scripts=document.getElementsByTagName('script')
                     ^
TypeError: Object #<Object> has no method 'getElementsByTagName'

So far the only way I have been able to get compiling of js to work is to inject javascript in selenium on a running browser instance.

[glyph]

@JamesHutchison - Have you tried PhantomJS? If you want the environment of a running browser instance but the convenience of a command-line tool it's nice.

[ghost]

It has been 6 months or so since I've worked on node_bridge.js. It is
possible that it is broken. We may want to see if PhantomJS is a better
solution than node. I haven't tried it, so it might be better.

On Fri, Feb 13, 2015 at 2:21 PM, Glyph notifications@github.com wrote:

@JamesHutchison https://github.com/JamesHutchison - Have you tried
PhantomJS? If you want the environment of a running browser instance but
the convenience of a command-line tool it's nice.

—
Reply to this email directly or view it on GitHub
#123 (comment).

[ghost]

On Thu, Feb 12, 2015 at 10:23 PM, Glyph notifications@github.com wrote:

For the first suggestion, I have found out that compiled code is between
5x-10x larger than the python version.

You could minify and gzip the compiled code for better results. Likely it
compresses very well :).

The python version compresses much better, and its size is even smaller.

The parser/compiler is very fast, so the download time might take longer
than the time it takes to parse and compile the python code.

This has not been my experience; on localhost, where the transfer is
instant, it still takes me about 5s to load a page with 16 or so modules on
it, which is 300ms per module. That is not a very fast compiler :-).

Plus, you pay the execution cost of the transpiler on every page-load,
but you only pay the download cost of cached files once.

Most of the 300ms per module is not the execution cost of the transpiler,
but the execution of the compiled code. At least that is what I've
discovered. Not everything that gets compiled is written to javascript
code. Some (I believe generators, and other similar things) are put into
objects in memory. I had trouble getting some things to execute properly
when I stored the javascript code. Just an FYI..

—
Reply to this email directly or view it on GitHub
#123 (comment).

[ghost]

Most of the 300ms per module is not the execution cost of the transpiler, but the execution of the compiled code.

Are you taking into account that the execution time of the module includes the transpiler time of submodules?

I did some testing and of the ~2.2 seconds it took to import the re module, it would likely be reduced to ~1.2 seconds if it didn't have to transpile any of the modules. For the small modules the transpiler time was significantly larger than the execution time.

compile re:103.62999996868894
compile sre_compile:142.3359999898821
compile _sre:268.7690000166185
compile operator:94.05900002457201
run operator:19.53799999319017
compile sre_constants:160.5190000263974
run sre_constants:35.77499999664724
run _sre:929.4510000036098
compile sre_parse:130.5609999690205
run sre_parse:49.559000006411225
run sre_compile:1433.0970000009984
compile copyreg:77.1200000308454
run copyreg:29.37300002668053
run re:2171.67100001825

[glyph]

@JamesHutchison - My measurement has not been terribly scientific, but this is completely consistent with my subjective experience of loading progressively larger numbers of small modules and watching things get slower without loading an appreciable amount more data.

[glyph]

Although this issue existed first, its discussion is somewhat muddled, so I will close it as a duplicate of both #151 and #222.