New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

documentation on how to package libraries for brython #119

Closed
glyph opened this Issue Jan 29, 2015 · 26 comments

Comments

Projects
None yet
9 participants
@glyph
Copy link
Contributor

glyph commented Jan 29, 2015

If I want to use a package from PyPI in Brython, what do I do? Right now I am manually adding packages or symlinks to a fork of brython, but it would be nice to be able to use Pip for this.

Until Brython has Pip (or something like it), at least some documentation around conventions for how to write and use libraries would be nice.

@kikocorreoso

This comment has been minimized.

Copy link
Member

kikocorreoso commented Jan 29, 2015

Not all Python libs are Brython compatible. It wouldn't be beneficial to let people install whatever from Pypi if it is not working. Maybe a Brypi (Brython Pypi) and a dedicated Brip (Brython pip management package).

Brypi could be a list of user cheched packages and versions and Brip a tool that checks first that list, if the package is in the list, download it on site-pagckages of the Brython distribution.

Ideas? Thoughts?

@glyph

This comment has been minimized.

Copy link
Contributor

glyph commented Jan 29, 2015

Not all Python libs are PyPy compatible, or Jython compatible, or Python 3 compatible, or, for that matter, CPython compatible. Nevertheless, PyPI is the shared repository for everything we call "python". I would strongly advise that you use the same infrastructure, since operating a package index is pretty hard, and a lot of effort has gone into making PyPI reliable, fast, and secure.

@glyph

This comment has been minimized.

Copy link
Contributor

glyph commented Jan 29, 2015

You could add a new Trove identifier to specifically indicate Brython compatibility, and then have "brip" refuse to download or install things without that identifier unless a special option, say, --force, is passed.

@jsbueno

This comment has been minimized.

Copy link
Contributor

jsbueno commented Jan 29, 2015

As for package managing:
I had not thought of using PyPi itself - but Glyph's enphatic suggestion
makes sense.
The registering of packages into PyPi does allow the script to register
which
interpreter versions it is compatible with - it is a matter of allowing
"Brython"
in that register, and specifying parameters to be used in Brython packages
"setup.py" file. (Note that this setup.py" file would be used by
distutils to upload the package into PyPi using a traditional command line
Python
(or eventually Brython over node.js :-) ) )

Also, if Brython does get a nice "setup.py" file, it can put in PyPi -
that would allow one to install say, browser_invaders, by just typing
"pip install browser_invaders" in a virtualenv. That would fetch both that
code, and the latest stable Brython, and arrange them in a working directory
structure ready to be served.

(Personally I like keeping Brython in a "brython/" folder relative to
www root)

On 29 January 2015 at 07:50, Glyph notifications@github.com wrote:

You could add a new Trove identifier to specifically indicate Brython
compatibility, and then have "brip" refuse to download or install things
without that identifier unless a special option, say, --force, is passed.


Reply to this email directly or view it on GitHub
#119 (comment).

@glyph

This comment has been minimized.

Copy link
Contributor

glyph commented Jan 29, 2015

@jsbueno - exactly. There are many different ways of arranging your brython files right now, and they are not all compatible with each other. If there were some pip idiom that worked with, let's say, data_files, that would make it a lot easier for new users to get started!

@PierreQuentel

This comment has been minimized.

Copy link
Contributor

PierreQuentel commented Jan 30, 2015

I agree that PyPI is the best choice. Brython is hopefully popular enough now to ask the PyPI team if they would consider adding a Trove classifier "Programming Language :: Python :: Implementation :: Brython"

Joao, would you be ok to take the lead on this task of handling Brython packages through PyPI ? I'm sure others on the Brython group would be willing to help

@jsbueno

This comment has been minimized.

Copy link
Contributor

jsbueno commented Jan 30, 2015

On 30 January 2015 at 14:53, Pierre Quentel notifications@github.com
wrote:

I agree that PyPI is the best choice. Brython is hopefully popular enough
now to ask the PyPI team if they would consider adding a Trove classifier
"Programming Language :: Python :: Implementation :: Brython"

Joao, would you be ok to take the lead on this task of handling Brython
packages through PyPI ? I'm sure others on the Brython group would be
willing to help

Yes, I think I can do that - I am taking a read on distutils to find out
the best way to describe a
Brython package with a "setup.py" - and meanwhile I could contact the Pypi
folks as well.

Since Brython even makes use of a "site-packages" folder, I think that is
the way to go: have a modified setup.py that installs the pack in Brython's
site-packages.

Is there any enviroment variable which might optionally point to Brython's
root?
(Like Python's "$VIRTUAL_ENV", not PYTHONPATH). If not, what about
"$BRYTHON_ROOT" ?
(it would point to the directory where bryhton.js is, and
ordinary install would go under $BRYTHON_ROOT/Lib/site-packages")

js
-><-

Reply to this email directly or view it on GitHub
#119 (comment).

@earney

This comment has been minimized.

Copy link
Contributor

earney commented Jan 31, 2015

One thing to keep in mind, is that Brython (is currently) just a subset of python3. The differences between Cpython 3 and Brython is smaller each day, but at this time, I am not sure it is a good idea to just assume that a package will work with Brython. Most packages eventually do work directly out of the box, but that is because we try to import a library/module, find what is not implemented (or broken) and then implement (fix it) in Brython.

I feel that if we provide a pip like tool, that is giving people permission to download whatever and run it in Brython. I believe one of these days, 99% of packages will just work (when downloading via pip), but today, I do not think Brython is mature enough to just throw any package at it. (IMHO)

I feel that once we get all/most of the standard library modules working, and that most of the unittests work, then we can provide a pip tool to the masses. (but if someone wants to create a pip like tool, and it is used by brython development staff for the time being, I'd be all for it.)

Am I too conservative?

Billy

@kikocorreoso

This comment has been minimized.

Copy link
Member

kikocorreoso commented Jan 31, 2015

Am I too conservative?

I don't think so, I think you are just cautious. Maybe we could maintain a document with links to tested libraries working in Brython and a pip like tool to install libraries contained in that document.

@earney

This comment has been minimized.

Copy link
Contributor

earney commented Feb 3, 2015

For now, I like Kiko's recommendation. we could create a tool named brip, and maybe have a "location" for these modules. As time goes by, and Brython supports 99% of CPython 3x, then we could modify it to point to pypi.

What do others think?
Billy

@jsbueno

This comment has been minimized.

Copy link
Contributor

jsbueno commented Feb 3, 2015

I think such a tool could be alright - if it is needed at all - but that we
might point at pypi right now - for Brython-specifc packages. Setuptools
allows enough meta-information for even an ordinary setup.py file to "know"
certain files are supposed to be served as static content and live with
Brython.

Being able to install and use ordinary, not intenteded for Brython, Python
modules is another history - -
Any module containing no native code or ctypes calls, and making no I/O
would potentially work (but for some bug fixes) - we just have to think of
a way to signal the module should be installed in a Brython's file
structure. (Maybe could be as simple as passing some install directory
parameters to PIP).

Anyway, there are two different things going on:

  1. a way to package and install Brython itself, and packages specifically
    designed to work with it
    (like Brython's Pygame currently in the source tree, the Browser Invader
    games, some html5-using widget and application toolkits, a package to
    easily make RPC calls to Python functions on the server side, and so on)

  2. A way to install generic Python packages in Brython's structure.

I will be focusing on (1) as it is even needed in a way to be able to
assert a file structure where packages available for Brython will end up.
(Although, since there is a "site-packages" directory, it will likely be
just
making use of it for both types of packages).

js
-><-

On 2 February 2015 at 23:55, Billy Earney notifications@github.com wrote:

For now, I like Kiko's recommendation. we could create a tool named brip,
and maybe have a "location" for these modules. As time goes by, and Brython
supports 99% of CPython 3x, then we could modify it to point to pypi.

What do others think?
Billy


Reply to this email directly or view it on GitHub
#119 (comment).

@glyph

This comment has been minimized.

Copy link
Contributor

glyph commented Feb 3, 2015

A brip tool might make sense as a thin wrapper around pip, so that all the options were set correctly to put stuff into a particular brython environment. Also to pre-compile any installed packages to .js, just like pip would normally pre-compile any installed packages to .pyc.

@earney

This comment has been minimized.

Copy link
Contributor

earney commented Feb 3, 2015

+1 placing the modules/packages in site-packages

I like @glyph 's idea, but here is an example that might cause some issues. What about brython custom packages of popular python packages such as pygame? pygame will be implemented very different for brython then for CPython. How would this work with pypi and pip?

@jsbueno

This comment has been minimized.

Copy link
Contributor

jsbueno commented Feb 3, 2015

On 3 February 2015 at 11:25, Billy Earney notifications@github.com wrote:

+1 placing the modules/packages in site-packages

I like @glyph https://github.com/glyph 's idea, but here is an example
that might cause some issues. What about brython custom packages of popular
python packages such as pygame? pygame will be implemented very different
for brython then for CPython. How would this work with pypi and pip?

We name it "brygame" instead? :-)

But seriously - I think it would be nice for this kind of packages to have
a prefix for the package name in PyPi
(this does not affect the actual module names) -
something like brython-pygame, brython-twisted, brython-tkinter .

It does not have to be "brython-", neither should it be mandatory - but
something along that
could be recommended.


Reply to this email directly or view it on GitHub
#119 (comment).

@glyph

This comment has been minimized.

Copy link
Contributor

glyph commented Feb 4, 2015

packages to havea prefix for the package name in PyPi (this does not affect the actual module names) - something like brython-pygame, brython-twisted, brython-tkinter

"package name in PyPI" technically is "distribution name"

Also, it is possible that as Brython becomes more popular, PyGame (or other related packages) may want to actually support Brython in the main distribution! Although lots of the core of PyGame would be very different on Brython, it could perhaps be split into a private low-level backend interface and then a public high-level interface, with all utility and convenience logic written in pure, implementation-agnostic Python :).

I know that I would want to make brip install twisted do something useful :-).

@dstufft

This comment has been minimized.

Copy link

dstufft commented Feb 11, 2015

For what it's worth, if there's something pip can do to support brython better, as long as it doesn't end up regressing non Python things, there's a good chance we can get it in. I don't know what brip would actually need over pip (this was linked to me and I've never used brython) so maybe it's just too different to sanely fit into pip itself I don't know.

@dstufft

This comment has been minimized.

Copy link

dstufft commented Feb 11, 2015

Oh, and by the Way, wheel supports arbitrary implementations in the tag name, so you can have a compiled Wheel which only contains code for Brython. You'd get something like foo-1.0-brythonXY-none-any.whl or something like that. We'd probably have to teach tooling to understand it, but the framework is there to support "binary" artifacts which are specific to a particular implementation.

@olemis

This comment has been minimized.

Copy link
Contributor

olemis commented Jul 27, 2015

IMHO we do not need brip or anything alike . After the work made in #189 the steps to install libs should be similar to this :

  • Create a Python virtual env in the server (optional)
  • Install your libs (easy_install , pip , ...)
  • Publish your virtual env folder via HTTP
  • Add virtual env base URL to sys.path

Therefore I'm closing this ticket . Feel free to reopen it otherwise .

@olemis olemis closed this Jul 27, 2015

@olemis

This comment has been minimized.

Copy link
Contributor

olemis commented Jul 27, 2015

There is another approach consisting in publishing via HTTP .vfs.js files (like this one ) containing code for several modules . It is necessary to add VFS file URL in sys.path .

@glyph

This comment has been minimized.

Copy link
Contributor

glyph commented Sep 12, 2015

One of the problems with the approach listed above is that pip can't do its compilation step to ensure that everything is compiled to JS, in the way that it would normally compile everything to .pyc. (If brython shipped a node.js compatible compileall module, this would work with no changes to pip.)

Another problem is that it's still not documented, even if it should work this way :). So I think this ticket should be reopened until there's clear documentation that you are at least supposed to do things this way (pip install with the python runtime of your choice).

@vikrantsanghvi

This comment has been minimized.

Copy link

vikrantsanghvi commented Dec 1, 2016

@glyph actually i am using brython for my data science project which is written in ipython jupyter notebook and the logic is in python. For getting it in web app form ,i am facing an error on the import of libraries like pandas which are basic ones for data sciences. it searches in src/Lib/site-packages/ but is not able to find. I believe because the package is not present in brython python standard library. How can i fork it to use it in my brython project? if you can provide some help or any documentation link or workaround, it would be great.

Thank you in advance. I am really looking forward to hearing from you regarding this query.

@kikocorreoso

This comment has been minimized.

Copy link
Member

kikocorreoso commented Dec 2, 2016

@vikrantsanghvi I think you are confusing the frontend and the backend. In the Jupyter notebook the python code runs in a backend kernel (the IPython kernel). The notebook is kernel agnostic, it means it can run Python, Julia, Haskell,..., code but using the same user interface (Jupyter) for all of them. In this context brython will run on the frontend (Jupyter notebook) and pandas et al data analytics related is running in the IPython kernel (backend).

I've made a library, brythonmagic, that runs in the Jupyter notebook and allows to write brython code in a code cell. It also allows to pass info from the backend to the frontend (no dataframes 'serialization'). The way to pass info is creating a string representation of the data that is injected in the brython code and the final brython code runs with the data from the backend.

In the repo linked above there are several notebooks showing how to work with brython in the notebook using brythonmagic.

I hope it helps.

@russoturisto

This comment has been minimized.

Copy link

russoturisto commented Apr 30, 2017

Hi,

We are in early stages of writing a cordova based federated server system, where every phone is it's own server with data specific for it's user. We are looking into running statistical analysis right on the device (on the data for it's user). I'm just starting to research our options but it looks like the only choice we have is bundling kivy along with cordova. Would compatibility with libraries like pandas be on the road-map in the foreseeable future?

Thanks, :)

@kikocorreoso

This comment has been minimized.

Copy link
Member

kikocorreoso commented Apr 30, 2017

Hi,

I think you are confused. If you use kivy you wouldn't need cordova. OTOH, brython is a python implementation written in javascript that allows you to run python on the frontend.

If you want to make a kivy app it is better to ask in their forums.

@russoturisto

This comment has been minimized.

Copy link

russoturisto commented Apr 30, 2017

@kikocorreoso

This comment has been minimized.

Copy link
Member

kikocorreoso commented Apr 30, 2017

Pandas has compiled dependencies and it is not an easy target.

If your stats are simple maybe it would be better to use Java/android, Swift/iOS, Javascript/Cordova, Kivy/Buildozer,..., and create your own statistical functions or use something like sqlite sql operations to do the work.

Brython doesn't support pandas and neither is a backend technology.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment