pip module #39
Comments
|
@briarfox My opinion is to use stash modules as much as possible to avoid repetition. I do realise that the current shell script is quite limited due to lack of program control constructs. So I am not sure how much functionality the pip module would have. I am still thinking about the possibility to interleave python code in a shell script so the shell script can be more useful. |
|
Ok currenly pyhonisa sie-packages does no load .pth files ( it's not in the site path list) I am wondering what the best way to handle packges is. When downloading a pypi package there is the root folder which usually contains documentation, examples and the acual package. The acual package needs to be in site-packages direcly. So the question is do we keep he documentation/examples and if so where? Options:
|
|
Another option: for each top level package (can a pypi contain more than one?), create a .py file in site-packages with that name. That py would then redirect any imports to the proper location, under the root wherever you place it. See below for an example.. In this example, typing first imports # site-packages/myModule.py
# import myModule will import this script first, but then will end up importing
def _do_import(p):
import sys, os, imp
m=imp.find_module(__name__)
path=os.path.expanduser(p)
module_path = imp.find_module(__name__,[path])
if module_path:
try:
module = imp.load_module(__name__,*module_path)
except ImportError:
raise #maybe fake traceback?
sys.modules[__name__]=module
else:
pass
#. What to do if package not found anymore? Haven't tested...
# maybe del sys.modules[__name__] so it doesn't point back here?
# maybe raise an ImportError?, which maybe does the same thing
_do_import('~/Documents/myModule-1.0/')pip could write out this script, just substituting the proper path. |
|
@briarfox I would prefer to simply use Option 1, i.e. move the actual package folder directly under
Stash can manage its own python path. In fact, I am going to add a @jsbain your approach is indeed quite clever. But for this issue, I kinda think a simple approach is viable. |
|
@ywangd Going with Option 1. It turns out that pypi does use .zip as well. I should have that added in tonight and will get a working pull request up. I agree that it would be useful to have pip keep track of packages installed through pip. Having the ability to update, list and remove packages would be handy. I'll work on that next. @jsbain Clever script, thanks. |
|
Current pull request of pip uses the common package naming conventions but I'm finding packages that do not use the standard. It looks like I'll need to parse setup.py to find the package folders. |
|
@briarfox I noticed the same issue when trying to install github3.py. So it is probably necessary to somehow parse the setup function (mainly to get the @jsbain Regarding to your idea about letting With that being said, I do agree that since both pypi and github manage packages, they should be somehow unified to simply the overall user-experience. So maybe add a wrapper function , e.g. Thanks! |
|
@ywangd @dgelessus I'm running into issues with parsing setup.py. The average setup.py is easy to parse: setup(name='somepackage',
packages = [somepackage],
etc
However there are may ways to pass params to setup.py. example: NAME = 'somepackage' PACKAGES = [somepackage'] params = dict(name=NAME,packages=PACKAGES) setup(**params) How can we handle all the different syntax? If I could just mock setup to return a dict of the params I'd be set. However I cant import or execfile setup.py becuase of the setuptools/distutils imports. |
|
How about |
|
@ywangd I was thinking specifically of pypi packages that are hosed on github, so have a setup.py, etc, and look identical to the real pypi modules except their source ( and in many cases I think pypi points to a github archive for the actual download!). The use case would be forked versions needed to run on pythonista. I agree a separate mechanism might be needed for true github modules that don't have setup.py. @dgelessus That is similar to the approach used by @pudquick for pipista v2, though he served an actual slimmed down version of setuptools with mods for the pythonista environment. Note that this script would need a lot of updates for pythonista 1.5+, since xmlrpclib is already installed, etc. also, as I recall there were issues in using Another question.... What prevents running setup.py in general? 1) missing _osx_support.py 2) some errors in getting some sysconfig parameters, because pythonista lies about some paths. 3) And maybe some other issues. |
|
Neat. Didn't know this project existed. Thanks for the mention. I did a custom setuptools with shortcircuited code in it because it expected an environment where some kind of alternate process spawning was possible. Not all python modules are pure python and setuptools didn't have native support for detecting the (jailed/sandboxed) Pythonista environment, thereby knowing to fail if compilation was required. I tried to wanted to just "let it fail", but the exec bug didn't let a core piece of the logic work. So I resorted to basic heuristics where most modules contained a single modulename.py or modulename/ folder and just extract -> installed them. It was on the user to know if it would work or not. I may have to look into this again :) |
|
@jsbain I get your point now. I agree this specific use case of GitHub repo shares a lot similarity with pypi. This is indeed the situation of the external libraries that the |
|
@briarfox A few things to note are:
The sample code is as follows: import sys
import os
from types import ModuleType
def make_module(new_module, doc="", scope=locals()):
"""
modified from
http://dietbuddha.blogspot.com.au/2012/11/python-metaprogramming-dynamic-module.html
make_module('a.b.c.d', doc="", scope=locals()]) -> <module built-in="built-in" d="d">
* creates module (and submodules as needed)
* adds module (and submodules) to sys.modules
* correctly nests submodules as needed
* not overwritting existing modules (my modification to the original function)
"""
module_name = []
for name in new_module.split('.'):
m = ModuleType(name, doc)
parent_module_name = '.'.join(module_name)
if parent_module_name:
if parent_module_name not in sys.modules.keys(): # do not overwrite existing modules
print 'creating parent', parent_module_name
setattr(sys.modules[parent_module_name], name, m)
else:
if m not in scope.keys(): # do not overwrite existing modules
scope[name] = m
module_name.append(name)
name_path = '.'.join(module_name)
if name_path not in sys.modules.keys(): # do not overwrite existing modules
sys.modules[name_path] = m
return sys.modules['.'.join(module_name)]
def setup(*args, **kwargs):
print kwargs['packages']
# NOTE: pip installation code here
# e.g. _stash('mv package_folder ~/Documents/site-packages/package_folder') etc.
pass
saved_modules = sys.modules
saved_cwd = os.getcwd()
try:
# pip code here (download and unzip)
# Fake the setuptools modules so setup.py can run
# fake from setuptools import setup
st = make_module('setuptools')
st.__dict__['setup'] = setup # the custom setup function will perform the installation
# more sub-modules to fake
# fake from setuptools.command.test import test
test = make_module('setuptools.command.test')
test.__dict__['test'] = type('test', (), {}) # dummy class so setup can run
os.chdir('TOP_LEVEL_FOLDER_of_the_UNZIPPED_PACKAGE')
execfile('setup.py')
finally: # remove the fake modules
sys.modules = saved_modules
os.chdir(saved_cwd) |
|
I do realise above code has some limitations:
|
|
@ywangd looks good, thanks. I should have an update today. |
|
@pudquick Did you want to handle pip for stash? I used pipista for reference. I've been sick and have a new born so my time coding has been greatly reduced. If you'd like to take it over let me know, no issues if you scrap my pip version. |
|
@briarfox Please take time to rest and look after the baby. No pressure here at all. |
|
@yawngd Thanks. I'll keep working at it, but its slow going. Figured if @pudquick wanted to work on it, that be fine. Do we want pip handleing setup.py or would that be more appropriate for the python module? Since users might like to install a script without pip by python setup.py install. |
|
@briarfox I would prefer to let I just realised @pudquick is the original author of shellista. Thanks for the inspiration on getting me started on StaSh. |
|
@ywangd I'm finding a wierd issue with python cwd. it seems that using the cd module is different then using os.chdir(). I'm finding that I need to use _stash('cd') to manipulate paths when jusing stash modules and os.chdir() for any paths needed outside of a stash module. example: #example 1 cwd ~/Documents
#Change directory with cd module
_stash('cd ~/Documents/site-packages')
_stash('tar -xzf ~/Documents/site-packages/some_file.tar.gz) #tar extracts to current dir
#This results in the unarchived file in Documents/site-packages
#example 2 cwd ~/Documents
os.chdir(os.path.expanduser('~/Documents/site-packages'))
_stash('tar -xzf ~/Documents/site-packages/some_file.tar.gz)
#This results in tar extracting into Documents/
|
|
@ywangd The setup.py is running with mixed results. Simple setup.py files work great but more complicated ones seem to run into a lot of import errors, even when the package works in pythonista. This is my current thought, let me know what you think. First we look for common module naming. Such as mymodule, mymodule.py. If found, simpley move folder/script. If not found, then try to run setup.py |
|
@briarfox The answer to the In your two examples, they actually both correctly extract archives to The reason is stash keeps track of its own cwd for each sub-shell. If no This behavior is to simulate the sub-shell environment where cwd changes in sub-shell does not affect its parent shell. I have to admit this may not be the best design as I sometimes get confused by my own code. But if you stick to use |
|
@briarfox I agree with you that we can try simple module names first before resorting to execute Could you please given me an example of the more complicated |
|
@ywangd Would I be better off sticking to using os.chdir() instead of _stash('cd ...')? Two of the modules that I've tried are scapy and Twisted. The current Twisted does not work in pythonista but I believe it will after next pythonista update. We could try to stub in other modules that are missing, but it seems that most cases can be solved by looking for the package directory. |
|
@briarfox As long as you use the same function to manipulate directory, it should be fine. I personally would prefer to use I had a look at the two modules. Yes the import processes really vary quite a bit. It is probably too much bandage work to cater all of them. |
|
@briarfox Another reason to use |
|
Does this version of pip attempt to handle dependencies in any way? If it is possible to read the required dependencies from the setup.py file, it might be useful to at least check if the dependency is part of default Pythonista or found in |
|
@dgelessus It currently does not install dependencies. It will attempt to load the module and print out a warning on possible dependencies if failed. I'd really like to grab dependencies but there are many ways to use setup.py so simplely parsing it is not a guarantee. If we could just find a way to handle majority of setup.py imports it would be easy to rely on setup.py working. I'll look into at least displaying the needed dependencies, if i can't auto install them. I'm still working on adding 3rd party github packages. |
|
@ywangd re: shellista inspiration - you're welcome :) |
|
So. Diving back into this (whee). pip is an exceedingly complex tool. But honestly, I only ever use a fraction of the functionality of it. Really what I want is the ability to: find, download, unpack, and install a package from PyPI When I was attempting to do this before, I came to the realization that pip (or at least the pieces I was using of it) only was really handling the search/download/unpack part. The install was still being handled by setuptools. To see what it's doing in the handoff, do the following:
if command_desc is None:
print cmd.__repr__()
if command_desc is None:
You will get the commands that pip (after finding, downloading, unpacking, and analyzing the package) sends to python to kick off the install process. Here's example output from beautifulsoup4: As you can see, pip essentially "bows out" of the process. It's generated a temporary script that's passed to python as the -c argument which relies on only setuptools from that point onwards. Basically, if you're building a pip equivalent - or at least trying to cover 80%+ of the functional code involving downloading from PyPI, unpacking, and installing - then really you only need:
The hard part I was running into before and I'm seeing again (now that I'm revisiting it) is controlling the installation so that it doesn't muck up your globals() and locals(), since we only have one running python interpreter instance. pip on a normal system (aka not Pythonista) solves this by running a new separate instance of python. If we can keep that segmented (if we care about that, even), we can probably patch around the rest. For instance: I already know how to control / redirect the directory that setuptools will attempt to install the package into :) |
|
@pudquick I do not believe we have access to setuptools. We had tried to mock distutils, setuptools but the list is long. It works great on simple packages but it was getting rather cumbersome on more complet setup.py files. The current pip handling:
It would be great to get a working setuptools and distutils. |
|
@pudquick Thanks for your explanation. I haven't spent much time on this topic. So I don't quite understand what exactly you mean by "muck up globals() and locals()". Is it possible to run the script with a copy of globals() and locals()? e.g. I did notice that pip is a wrapper of setuptools, which in turn is a wrapper of distutils. So only implementing pip is not going to solve the whole problem as Pythonista does not have full functional What I think is that if we can intercept the call to |
|
@briarfox I like your pip module so far :) Here's a different take on it: https://gist.github.com/pudquick/cb7b6413a54162fd1c00 Try it out as 'pip2.py' if you want to see it in action. It's not a complete implementation, but vanilla 'install' is functional. Mine's not stash-specific, so it carries its own support for handling gzip / tar / zip files (makes up about 50% of the code in itself). But it's got progress indicators for download (with current speed), its own method for parsing setup.py (I split out the 'setup(...)' call and then dummy out up to a max of 10 undefined object names), plus it safely contains modifications to global/local variables by setup.py. I've got additional ideas for handling setup.py that can probably increase its parsing resilience, but as it's written right now, it's confirmed as able to install simple packages. If anything in there is helpful, grab what you like. |
|
@pudquick
|
|
@ywangd Sorry about that (the first error)! Just updated the code, should be fine now. It intentionally makes temp files in __pip_tmp because the intent, later, is to provide an option to preserve the temporary directory when module download succeeds but parsing fails (allowing for a manual option / seeing where it failed). By all means grab what code you like. As for the other error you ran into - it indicates there's code outside of the setup(...) block in setup.py necessary for building the 'packages' dict key. I want to try some other tricks before resorting to naively executing setup.py. Pretty sure I can tweak better performance out of it. |
|
@pudquick Great! I'll have a look at it when I get to work. |
|
@ywangd @briarfox Updated mocking/dummy capabilities. There is now a '_dummy' global object it creates that can act like an object with attributes or even like a function. You can do any of the following with a _dummy object:
This stubs out most "key = " actions in setup(...). Also replaced the built-in open(...) with a version that, if unable to open a particular file it returns a _dummy object instead. @ywangd: This makes it successfully install 'uritemplate' now, for example, which had setup(...) lines like:
|
|
Looking at the pip module lines 281 ff, I wondered why you chose to prefer manual unpacking of modules (for instance I would like to suggest to prefer the setup method to the others if a setup.py file can be found. |
|
if you look through the history, pip has gone through multiple iterations. at one time the fake setup was used.... the problem is that pythonista has a funny folder system, and some of the places setup would try to install things are not writable. that said, pip does routinely fail on some modules... if you can figure out ways to make it more reliable, i think pull requests are welcome! |
|
@jsbain Thanks, I think I'll have a look at how to improve pip. No promises though... One more questions: pip has code snippets such as Georg |
|
You can use |
|
Finally come back to this topic ... I have just largely rewrite the The major change is that setup(**kwargs)or setup(**get_my_kwargs())or setup(packages=find_packages())This is made possible by parsing the setup file into AST and modify AST to hijack the setup call. So the setup file pretty much runs normally until it reaches the Try This change is now available in both |
|
Thank you for your efforts with pip. I installed oletools (installed within /site-packages/oletools/), but when executing a python script (such as mraptor.py), I am presented with the following error:
The thirdparty directory exists within the oletools folder, suggesting that the toolkit was correctly installed. I suspect this is more likely to be a path issue. Can anyone offer any guidance please? |
|
@ssnkhan stash pip does not handle path configuration for sub-packages. It is however easy to fix by moving the |
|
Thank you @ywangd. This makes some scripts work (such as olevba.py), however, some tools, existing at the same location as /olevba.py (such as /mraptor.py) throw up the following error:
I also find that the auto-complete for scripts only works within the active directory in which the tool is stored - is there anyway I can link these somehow, so that I can accept the script from any directory? I am taking a foundation in Python at present, so I apologise for what would otherwise be considered a very newbie question. EDIT: Call script from any location -- solved, I used alias.py and linked the script from there. However, the alias does not persist after Pythonista is closed. How can the alias be made more permanent? |
|
The errors can be fixed by add a few settings in Add following contents to the file: PYTHONPATH=~/Documents/site-packages/oletools:$PYTHONPATH
BIN_PATH=~/Documents/site-packages/oletools:$BIN_PATHThe first line should solve the import issue and the second line allows you to call the script anywhere with auto-completion. The aliases can also be persistent by add them to this file. To create this file, you can issue following commands: Now you should have this file opened in an editor tab. You'll need to restart StaSh or even Pythonista after you finish typing. |
|
Thank you, that works perfectly! Supposing I have other scripts in other directories which I also wish to call from anywhere, how is that best achieved? For instance, I installed balbuzard via pip, and adding the following lines to .stashrc didn't work: oletools scripts still auto-complete, but balbuzard ones don't. I have googled this too, but struggling to find the answer, so I appreciate your continued patience with me! |
|
It worked for me. Did you try restart Pythonista (i.e. double click Home button and swipe up to remove Pythonista from the task list)? Also it is possible to add multiple entries in one line as: |
|
@ywangd oletools works, balbuzard refuses to autocomplete. I've tried placing the instructions on seperate lines as well in their short hard form as you've suggested, but while oletools continues to work, balbuzard does not autocomplete. I have killed the app from the dock and checked via cat and the file has saved the commands. |
|
Resolved — pardon my stupidity, I was editing the file in ~Documents, rather than $STASH_ROOT! All working now, many thanks :) |
|
What are the outstanding tasks to close this issue out? |
#About this commit/PR This is a commit/PR to close a large number of old/fixed issues at once using the github `fix #<issuenumber>` syntax. Below is a list of all issues which will automatically be closed by merging this commit/PR into `ywangd/master`. Behind each issue number is a short summary about the issue or why it can be closed. #Issues which can be closed **Py3 related:** - fix ywangd#281 (py3 had not yet been supported) - fix ywangd#261 (python3 pip installation) - fix ywangd#246 (python3 code in python2) - fix ywangd#222 (python3 command in stash) - fix ywangd#200 (pip now works (at least partially) with python3) - close ywangd#197 (general pythonista3 compatibility thread) **Questions which have been answered / unfixable bugs:** - close ywangd#310 (pip installation with c code) - close ywangd#292 (StaSh is not dead; also a question regarding py3) - close ywangd#290 (pip installation, problem solved) - close ywangd#289 (answer provided, no further activity by reporter) - close ywangd#279 (pip install protects standard distribution) - close ywangd#245 (answered) - close ywangd#244 (not possible (unless someone finds a workaround) - close ywangd#243 (c-code in pip installation) - close ywangd#208 (question answered) **implemented suggestions/changes:** - fix ywangd#273 (partially implemented as part of py2and3) - fix ywangd#193 (implemented by now) - fix ywangd#63 (subcommand completion; implemented long ago) - fix ywangd#39 (pip; implemented long ago) **fixed bugs:** - fix ywangd#308 (https error in pip; fixed) - fix ywangd#223 (old beta-related bug) - fix ywangd#211 (old beta-related bug) - fix ywangd#209 (seems to be fixed, can not reproduce) - fix ywangd#206 (git did not clone into subdirectory; fixed) - fix ywangd#98 (telnet did not send \r\n; fixed) **other stuff:** - close ywangd#260 (no useful bug description) - close ywangd#258 (no useful bug description) - close ywangd#251 (not a bug but an announcement, no longer relevant) - close ywangd#241 (works now) - close ywangd#219 (no further acivity by reporter, likely user error) #Issues which can be closed, but where the underlying issue has not been fixed - close ywangd#285 (pip command too complex) - close ywangd#291 (pip installation question, but not c-code related. But seems package-related) - close ywangd#274 (pip install problem, at least partial related to c-code, no further activity by reporter)
The pip module is almost done. Does anyone know if pypi modules are only tar.gz or are .zip used as well? I have not been able to find a .zip example.
Currently pip is using a .sh script to install the pypi package since setup.py will not work in pythonista. The .sh uses common stash modules. Is this exceptable or should pip handle ungziping, mkdir and rm by itself? Currently the .sh uses: tar, mkdir, mv and rm stash modules.
The text was updated successfully, but these errors were encountered: