Remote, in-memory Python package/module importing via HTTP/S, FTP, or SMB
Inspired by httpimport
.
od_import
enables Python packages and modules to be imported within a Python interpreter's process memory via remote URIs
Able to import packages with c extensions! (Windows only -- with _memimporter)
Able to import packages via HTTP/S, SMB, or FTP
Leverages path caching to avoid noisy bruteforcing (404s)
Written in modular way to allow for expanding supported protocols
Supports dynamic function patching to support packages with complex requirements
Only imports dependencies as required (for example, won't load smb dependencies unless smb handler is used)
od_import.add_remote_source('http://my-packages.local/site-packages', INSECURE=True)
import package
with od_import.remote_source('http://my-packages.local/site-packages', INSECURE=True):
import package
config = {"verify": False}
with od_import.remote_source('http://my-packages.local/site-packages', config=config):
import package
crt = open("my-ca-crt.crt").read()
config = {"ca_data": crt}
with od_import.remote_source('http://my-packages.local/site-packages', config=config):
import package
config = {"type": "s3", "access_key": "MYACCESSKEY", "bucket": "mybucket", "secret_key": "MYSECRETKEY", "region": "us-east-1"}
with od_import.remote_source("https://amazonaws.com", config=config):
import mypackage
Load package/module via GITHUB repo (uses github.com and raw.githubusercontent.com) - github example
config={"type": "git", "git": "github", "user": "rkbennett", "repo": "py3memimporter"}
with od_import.remote_source('https://github.com', config=config):
import py3memimporter
config={"type": "git", "git": "gitlab", "group": "mygroup/mysubgroup", "project": "myproject", "api_key": "glpat_1234567890"}
with od_import.remote_source('https://my-gitlab.local', config=config):
import project_package
config={"type": "git_zip", "git": "github", "user": "naksyn", "repo": "PythonMemoryModule"}
with od_import.remote_source('https://github.com', config=config):
import pythonmemorymodule
config={"type": "git_zip", "git": "gitlab", "group": "mygroup/mysubgroup", "project": "myproject", "api_key": "glpat_1234567890"}
with od_import.remote_source('https://gitlab.local', config=config):
import myrepo_package
Load package/module via repo API (would recommend you use the api_key config or you will be severely rate limited) - github example
config={"api_key":"github_pat_somelongapikeystring", "type": "git_api", "git": "github", "user": "myuser", "repo": "myrepo"}
with od_import.remote_source('https://api.github.com', config=config):
import py3memimporter
Load package/module via repo API (would recommend you use the api_key config or you will be severely rate limited) - gitlab example
config={"api_key":"glpat_1234567890", "type": "git_api", "git": "gitlab", "group": "mygroup/mysubgroup", "project": "myproject"}
with od_import.remote_source('https://my-gitlab.local', config=config):
import mypackage
config={"type": "pypi", "verify": False, "package": [{"name": "psutil", "release": "5.9.5"}]}
with od_import.remote_source('https://pypi.org/pypi', config=config):
import psutil
config={"type": "dropbox", "verify": False, "access_token": "sl.mylongdropboxaccesstoken"}
with od_import.remote_source('https://dropbox.com/pybof', config=config):
import bof
config={"package": ["psutil"]}
with od_import.remote_source('pip', config=config):
import psutil
config = {'type':'pastebin', 'visibility': 'passworded', 'paste_key': 'PaSt3Key', "module": "foo", "developer_key": "mydeveloperkey", "user_key": "myuserkey", "paste_password": "foobarbaz"}
with od_import.remote_source('https://pastebin.com', config=config):
import foo
config = {"access_key": "MYACCESSKEY", "bucket": "mybucket", "secret_key": "MYSECRETKEY", "region": "us-east-1"}
with od_import.s3(**config):
import my_package
with od_import.github("rkbennett", "py3memimporter", branch="main", git_type="git_zip", api_key="github_pat_somelongapikeystring"):
import py3memimporter
with od_import.gitlab("https://my-gitlab.local", "mygroup", "myproject", branch="main", git_type="git_api", api_key="glpat_1234567890", verify=False):
import py3memimporter
with od_import.gitea("http://my-gitea.local", "rkbennett", "py3memimporter", branch="main", git_type="git", INSECURE=True):
import py3memimporter
with od_import.pypi(package=[{"name": "psutil", "release": "5.9.5"}], verify=False):
import psutil
with od_import.dropbox("sl.mylongdropboxaccesstoken", path="/pybof", verify=False, timeout=30):
import bof
with od_import.remote_source('ftp://my-packages.local/site-packages', INSECURE=True):
import ftp_package
config = {"user":"rkbennett","password":"1234567"}
with od_import.remote_source('smb://my-packages.local/site-packages', INSECURE=True, config=config):
import smb_package
No files touch disk during import
with od_import.remote_source('https://my-packages.local/package.zip'):
import zip_package
No files touch disk during import
with od_import.remote_source('https://my-packages.local/package.zip', zip_password=b"SuperSecretPassword"):
import zip_package
Importing modules/packages with c extensions (uses py3memimporter
)
import py3memimporter
import od_import
od_import.add_remote_source('http://my-packages.local/site-packages', INSECURE=True)
import psutil
Configs are a dict of attributes associated with the remote source for packages/modules
username
password
timeout
(allows for setting the timeout of a connection)http_version
(allows for downgrading http version toHTTP/1.0
on requests)headers
(dictionary of headers for requests, user-agent defaults to Python-urllib/3.x)proxy
(Currently supports unauthenticated only)verify
(enable/disable certificate verification for https certificates, defaults toTrue
)ca_file
(path to ca file for certificate trust)ca_data
(string containing one more concatinated ca certificates)type
(one of dir, s3, git, git_zip, or git_api; currently defaults to dir)api_key
(only used for git, git_zip, and git_api types)git
(only acceptsgitea
,gitlab
, andgithub
currently; only used for git_zip and git_api types)package
(only used for pypi, can be a str of the package name, a dict of the package name and release or a list of package dicts)user
(only used for github and gitea, owner of the target repo)repo
(only used for github and gitea, the target repo)group
(only used for gitlab, the full group path for the target project -- this includes subgroups)project
(only used for gitlab, the target project)branch
(only used for gitlab, github, and gitea, the desired branch of the target repo/project)visibility
(only used for pastebin, must be one ofpublic
,private
,unlisted
,burn
, orpassworded
)module
(only used for pastebin, the fully-qualified name of the module you wish to import the paste as)paste_key
(only used for pastebin, the id of the paste)developer_key
(only used for pastebin whenvisibility
set toprivate
,burn
orpassworded
)user_key
(only used for pastebin whenvisibility
set toprivate
,burn
orpassworded
)paste_password
(only used for pastebin whenvisibility
set topassworded
)access_token
(only used for dropbox)http_provider
(only acceptswinhttp
currently; instructs web requests to use winhttp api calls instead of urllib -- only works on Windows)access_key
(only used for s3 type)secret_key
(only used for s3 type)bucket
(only used for s3 types)region
(only used for s3 types)
port
(defaults to 445)user
(defaults to guest)password
(defaults to guest)proxy
(Not currently used)client
(what the client wants to be called, defaults to localhost)nbname
(netbios name for remote source, defaults to hostname from URL)smb2
(enables smb2 support, defaults to True)
port
(defaults to 21)user
(defaults to anonymous)password
(defaults to "")proxy
(Not currently used)
package
(can be list of package name strings or package name string)
This is a boolean arg that is supplied to the hook constructor, which enables the use of insecure protocols (http, smb, ftp)
If you are importing from a zip which requires a password, you must provide the zip_password keyword arg to the hook constructor
bitbucket helper
add multiple gitlab token types to git helpers
proxy support
authenticated testing (should work, only tested anonymous so far)
proxy support
Add obfuscator framework
NFS handler
Packaging
Testing for other python distros
- This has currently only been tested on windows 10 and 11 with python 3.10-3.12, but in theory should work on any version 3.6+
- Download and install the Git client and Python on your device.
- Open your terminal. (Make sure you have the necessary permissions to install packages on your system).
- Execute the following command:
pip install git+https://github.com/rkbennett/od_import.git
- boludoz - gitignore and setup.py
- natesubra - For challenging me to learn this stuff :D
- operatorequals - For httpimport, which is where I cut my teeth working on import hooks
- desty2k - For Paker, which helped me figure out how to support packages with C extensions
- py2exe - For the hooking logic for dynamic patching
- SeaHOH - For _memimporter