Scripts that help build a mirror for PyBOMBS repos.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore cleaning up Jun 16, 2016
.travis.yml cleaning up Jun 16, 2016
README.md Update README.md Nov 30, 2016
ignore.urls ignore/passthrough all svn URLs for now. Jun 13, 2016
nginx.conf nginx.conf Jun 16, 2016
pre-replace-upstream.urls Update README Jun 16, 2016
pybombs-mirror.sh generator of readme.txt: bladerf -> bladeRF Nov 30, 2016
upstream-recipe-repos.urls cleaning up Jun 16, 2016

README.md

PyBOMBS mirror site builder

Build Status

Sites that running with this script

As Mirror Maintainer

sudo apt-get install fcgiwrap nginx git svn wget

sudo mkdir /pybombs
sudo gpasswd -a yourid www-data  # then logout and in
sudo chown www-data:www-data /pybombs

export PYBOMBS_MIRROR_BASE_URL="http://yoursite.example.com/pybombs"
export DRY_RUN=false 
#export DRY_RUN=true  # Or test drive without actually fetching with DRY_RUN=true
export PYBOMBS_MIRROR_WORK_DIR=/pybombs

cp upstream-recipe-repos.urls ${PYBOMBS_MIRROR_WORK_DIR}/
cp pre-replace-upstream.urls ${PYBOMBS_MIRROR_WORK_DIR}/
cp ignore.urls ${PYBOMBS_MIRROR_WORK_DIR}/

./pybombs-mirror.sh

cp ./nginx.conf /etc/nginx/sites-available/default

sudo /etc/init.d/nginx restart
  • upstream-recipe-repos.urls Recipes repos to be fetched.
  • (optional) ignore.urls defines urls that will be ignored. Users of these URLs will be passed through upstream.
  • (optional) pre-replace-upstream.urls defines custom upstreams replacement. In order to gain better syncing speed according to your network condition.

As Mirror User

rm -rf ~/.pybombs
pybombs recipes add gr-recipes git+http://yoursite.example.com/pybombs/git/gr-recipes.git 
pybombs recipes add gr-etcetera git+http://yoursite.example.com/pybombs/git/gr-etcetera.git 
mkdir gnuradio-prefix
cd gnuradio-prefix
pybombs prefix init
pybombs install gnuradio
. ./setup_env.sh
gnuradio-companion

pybombs install rtl-sdr hackrf bladeRF gr-osmosdr gr-bluetooth gr-ieee-80211

PyBOMBS Update issue

Unfortunately, PyBOMBS only support git as remote repo. Your patched recipe repos will make a complex git tree. So we don't want to mess up with a lot of git-cherry-pick git-merge or something.

For now, every time we fetch recipes from upstream, we drop our previously patched commits, generate a new repo. Thus user need to do a refresh git clone, because the mirror recipe git repo we generated doesn't have a continous git tree.

As a result, users cannot use pybombs recipes update directly.

Instead, users can:

pybombs recipes remove gr-etcetera
pybombs recipes remove gr-recipes

pybombs recipes add gr-recipes git+http://yoursite.example.com/pybombs/git/gr-recipes.git 
pybombs recipes add gr-etcetera git+http://yoursite.example.com/pybombs/git/gr-etcetera.git 

I think using a local directory is another option, such as:

git clone --depth=1 http://yoursite.example.com/pybombs/git/gr-recipes.git /path/to/your/git/cache
pybombs recipes add gr-recipes /path/to/your/git/cache

# Updating
rm -rf /path/to/your/git/cache
git clone --depth=1 http://yoursite.example.com/pybombs/git/gr-recipes.git /path/to/your/git/cache

Maybe there exists another git trick to help me solve this. sigh..

See also: http://lists.gnu.org/archive/html/discuss-gnuradio/2016-06/msg00170.html

Technical Details


10-retrieve-urls-from-recipes....................................................
                            upstream-recipe-repos.urls  
                                [Upstream recipes]
                                       |
                                       | git clone/update
                                       v
                                    recipes-origin/
                                       |
                                       | Copy
                                       v
                                    _recipes/
                                       |
    pre-replace-upstream.urls  -->     + grep & sed
                                       v
                                    _recipes/
                                       |
                                       | grep "(wget|svn|git)+..."
                                       v
                               recipes-origin.urls
                                       |
         ignore.urls           -->     + sed, delete line
                                       v
                               recipes-origin.urls
                                       |
20-fetch...............................|.........................................
                                       |             +--> svn/
                                       v             |
                                    Fetch...  -------+--> wget/
                                       |             |
                                +------+------+      +--> git/
                                |             | 
                                V             V 
       _recipes-mirror-replacement.urls     failed.log
(PYBOMBS_MIRROR_BASE_URL as placeholder)
                                |
30-replace-recipes..............|................................................
                                |
                     [sed] PYBOMBS_MIRROR_BASE_URL -> $PYBOMBS_MIRROR_BASE_URL
                                |
                                |
                                |                    _recipes/
                                v                       |
        recipes-mirror-replacement.urls --------->------+ grep & sed 
                                                        |
40-deploy...............................................|........................
                                                        | git commit -am  
                                                        | git clone --bare
                                                        v
                                                      recipes/
                                                        |
                                                        |
                                                        v
                                                      README.txt

Build a PyBOMBS totally from upstream PyBOMBS mirror

wget http://mirrors.tuna.tsinghua.edu.cn/pybombs/pre-replace-upstream.urls -O 1.urls
wget http://mirrors.tuna.tsinghua.edu.cn/pybombs/recipes-mirror-replacement.urls -O 2.urls
cat 1.urls 2.urls > pre-replace-upstream.urls

It's very convenient to choose a upstream mirror site with stable network speed.

Change a URL to another URL while your mirror site don't want to fetch it

  1. Add that URL replacement pair into pre-replace-upstream.urls
  2. Add the latter URL into ignore.urls

For example:

echo "wget+https://download.qt.io/archive/qt/4.6/qt-everywhere-opensource-src-4.6.2.tar.gz  wget+http://mirrors.tuna.tsinghua.edu.cn/qt/archive/qt/4.6/qt-everywhere-opensource-src-4.6.2.tar.gz" >> pre-replace-upstream.urls
echo "wget+http://mirrors.tuna.tsinghua.edu.cn/qt/archive/qt/4.6/qt-everywhere-opensource-src-4.6.2.tar.gz" >> ignore.urls

TODO

  • SVN fetch method
  • remove PYBOMBS_MIRROR_ROOT_DIR
  • add mirror usage
  • auto deploy: gr-recipes and gr-etcetera automatically moved to /pybombs/git/
  • add travis-ci configuration
  • add support for multi repos.