New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARM (i.e. Raspberry Pi) install problematic #43

Closed
cdubz opened this Issue May 5, 2018 · 15 comments

Comments

Projects
None yet
2 participants
@cdubz
Copy link
Owner

cdubz commented May 5, 2018

#41 and #42 seem to indicate some issues with doing a manual install on ARM devices. I'm going to work through testing this on a Model B from Raspbian Stretch Lite to see if things can be smoothed out.

@cdubz cdubz self-assigned this May 5, 2018

@cdubz

This comment has been minimized.

Copy link
Owner

cdubz commented May 6, 2018

I am getting random kernel panics and segfaults with one of my Model B systems, but the other is getting closer. Perhaps the first one has memory or disk corruption.

Anyway, on my first run through I got an sha256 error similar to #41 but with a different package. I have found some random clues here and there about similar issues using pipenv's Pipfile.lock file between operating systems (e.g. bennuttall/piwheels#47; pypa/pipenv#1191) so I'm not sure there is much to be done here (other than maybe advising pipenv install --skip-lock for the Pi specifically). I think this happens because the Pi is using piwheels for some packages.

Reference re: piwheels - piwheels: building a faster Python package repository for Raspberry Pi users

I am also testing running pipenv lock first, but that seems to run in to issues with the psycopg-binary requirement. I think maybe it doesn't have a binary for arm and fails to build from source because the system lacks some pg dev packages.

@novski Did you ever get things up and running on your Pi?

@cdubz

This comment has been minimized.

Copy link
Owner

cdubz commented May 6, 2018

Just adding some additional notes from my testing -

With my Pi, pipenv install [...] is consistently failing to build numpy, which is a dependency of pandas. Even if I rebuild the lock file on the Pi or use --skip-lock (which makes sense). The error I get is:

Command "/home/pi/.local/share/virtualenvs/public-BuvyXxxq/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-c10b86m4 https://files.pythonhosted.org/packages/1b/d2/22cde5ea9af055f81814f9f2545f5ed8a053eb749c08d186b369959189a8/wheel-0.31.0-py2.py3-none-any.whl#sha256=9cdc8ab2cc9c3c2e2727a4b67c22881dbb0e1c503d592992594c5e131c867107 https://files.pythonhosted.org/packages/8c/10/79282747f9169f21c053c562a0baa21815a8c7879be97abd930dbcf862e8/setuptools-39.1.0-py2.py3-none-any.whl#sha256=0cb8b8625bfdcc2d43ea4b9cdba0b39b2b7befc04f3088897031082aa16ce186 https://www.piwheels.org/simple/cython/Cython-0.28.2-cp35-cp35m-linux_armv7l.whl#sha256=d3680f79ae233275265917936a4f47f0bb0136d81c50aaee5ff5b7de11c03644 https://www.piwheels.org/simple/numpy/numpy-1.9.3-cp35-cp35m-linux_armv7l.whl#sha256=b542e1f6c4dd53e18c42eb102214078279cb4bca1d183eeccd957da1edcfd024 https://www.piwheels.org/simple/numpy/numpy-1.12.1-cp35-cp35m-linux_armv7l.whl#sha256=473f951e6103c0f01dbaec2ef2298e8828659e64635a0cb4d0a03a17aa295ab3 https://www.piwheels.org/simple/numpy/numpy-1.13.1-cp35-cp35m-linux_armv7l.whl#sha256=b698328af899dcd647711e1a835ee0b7ea01247a1ef7f2512fe044465ac2b303" failed with error code 1 in None

I can get around this by changing pandas in Pipfile (which has numpy as a dependency) to look for the latest version. Baby Buddy has pandas locked at a lower version because it had build issues with Python 3.4. I barely ended up using pandas anyway, so I may refactor that out or drop Python 3.4 support.

However, I still end up with build issues for psycopg2-binary. Because there doesn't seem to be an ARM binary, both psycopg2 and psycopg2-binary attempt to build from source and can't because:

Looking in indexes: https://pypi.python.org/simple, https://www.piwheels.org/simple
Collecting psycopg2 
  Using cached https://files.pythonhosted.org/packages/74/83/51580322ed0e82cba7ad8e0af590b8fb2cf11bd5aaa1ed872661bd36f462/psycopg2-2.7.4.tar.gz
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/psycopg2.egg-info
    writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt
    writing pip-egg-info/psycopg2.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'
    
    Error: pg_config executable not found.
    
    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:
    
        python setup.py build_ext --pg-config /path/to/pg_config build ...
    
    or with the pg_config option in 'setup.cfg'.
    
    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.
    
    For further information please check the 'doc/src/install.rst' file (also at
    <http://initd.org/psycopg/docs/install.html>).

piwheels does have wheels for psycopg2, but not for the version pip is attempting to install (2.7.4). There is some history around psycopg2 and wheel packages in the psycopg 2.7.4 release notes.

So, ultimately I got the pipenv step working by:

  1. changing pandas = "<0.22.0" to pandas = "*" in Pipfile,
  2. changing psycopg2-binary = "*" to psycopg2 = "<2.7.4" in Pipfile, and
  3. executing pipenv install --three --dev --skip-lock.
@cdubz

This comment has been minimized.

Copy link
Owner

cdubz commented May 6, 2018

Lastly, sudo apt-get install libopenjp2-7-dev is necessary for the easy-thumbnails package on ARM (see #42).

After all that (plus 674da36 and e6d1205), Baby Buddy runs on a Pi! Now to document this...

@cdubz cdubz referenced this issue May 6, 2018

Closed

modul Image missing #42

@novski

This comment has been minimized.

Copy link
Contributor

novski commented May 6, 2018

Yes! After pulling your changes (thanks for reverse engineering!) and applying the dependency sudo apt-get install libopenjp2-7-dev , i ran step 7-10 successfully but then caught an error because of missing a db.
Does the migrate command normaly create a db, or is the manual creation of an Sqlite db not documented?

i have the default code in my babybuddy/settings/production.py file:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, '../data/db.sqlite3'),
    }
}

I have to say, am not verry experienced with django, but i have done one project with it in school once and this first steps: https://tutorial.djangogirls.org/en/django_start_project/
In that resource its described that python manage.py migrate is used to initialize the db.
Seeing gulp migrate, i assume, in this project gulp is takeing care of this?

how come the db.sqlite3 is not accessable for gulp?

here is my full Terminal log of stef 7 to 10: https://pastebin.com/ZnHeNHNJ

edit: giving python3 manage.py migrate a shot unveiled a different error:
i had to install dotenv because of ImportError: No module named 'dotenv'
i did with pip3 install dotenv --user and then got this error:

pi@babybuddy:/var/www/babybuddy/public $ python3 manage.py migrate
Traceback (most recent call last):
  File "manage.py", line 4, in <module>
    from dotenv import load_dotenv, find_dotenv
ImportError: cannot import name 'load_dotenv'

found this: https://stackoverflow.com/a/47393778/4061870 but i do not know where to search for this circular dependency...

@cdubz

This comment has been minimized.

Copy link
Owner

cdubz commented May 6, 2018

Re: the migrate command - I just made a change in 674da36 to address this. What you can do now is sudo chown pi:pi /var/www/babybuddy/data and then follow the current steps from gulp migrate. Should be good to go after that.

Re: dotenv - Did you execute that pip3 install command from within a pipenv shell? If not, you can do that or pipenv install dotenv from /var/www/babybuddy/public. If all else fails, you can nuke that environment (pipenv uninstall -all) and try to start fresh with the modifications from my above comments.

@novski

This comment has been minimized.

Copy link
Contributor

novski commented May 6, 2018

the manual section does not mention to pipenv shell is that a workaround?

anyway it still does not work:

pi@babybuddy:/var/www/babybuddy/public $ pipenv shell
Spawning environment shell (/bin/bash). Use 'exit' to leave.
. /home/pi/.local/share/virtualenvs/public-BuvyXxxq/bin/activate
pi@babybuddy:/var/www/babybuddy/public $ . /home/pi/.local/share/virtualenvs/public-BuvyXxxq/bin/activate
(public-BuvyXxxq) pi@babybuddy:/var/www/babybuddy/public $ pip3 install dotenv
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting dotenv
  Using cached https://www.piwheels.org/simple/dotenv/dotenv-0.0.5-py3-none-any.whl
Installing collected packages: dotenv
Successfully installed dotenv-0.0.5
(public-BuvyXxxq) pi@babybuddy:/var/www/babybuddy/public $ gulp migrate
[18:50:58] Using gulpfile /var/www/babybuddy/public/gulpfile.js
[18:50:58] Starting 'migrate'...
Traceback (most recent call last):
  File "manage.py", line 4, in <module>
    from dotenv import load_dotenv, find_dotenv
ImportError: cannot import name 'load_dotenv'
[18:50:59] 'migrate' errored after 1.41 s
[18:50:59] Error: 1
    at formatError (/usr/lib/node_modules/gulp-cli/lib/versioned/^3.7.0/format-error.js:20:10)
    at Gulp.<anonymous> (/usr/lib/node_modules/gulp-cli/lib/versioned/^3.7.0/log/events.js:41:15)
    at emitOne (events.js:116:13)
    at Gulp.emit (events.js:211:7)
    at Gulp.Orchestrator._emitTaskDone (/var/www/babybuddy/public/node_modules/orchestrator/index.js:264:8)
    at /var/www/babybuddy/public/node_modules/orchestrator/index.js:275:23
    at finish (/var/www/babybuddy/public/node_modules/orchestrator/lib/runTask.js:21:8)
    at ChildProcess.cb (/var/www/babybuddy/public/node_modules/orchestrator/lib/runTask.js:29:3)
    at emitTwo (events.js:126:13)
@cdubz

This comment has been minimized.

Copy link
Owner

cdubz commented May 6, 2018

Oops, sorry about that. The package is actually python-dotenv!

pipenv shell is just a way to access the virtual environment that pipenv creates. Normally, this stuff is all covered by the regular pipenv install command. Just trying to save you having to reinstall the entire environment here.

@novski

This comment has been minimized.

Copy link
Contributor

novski commented May 6, 2018

Thanks @cudbz i realy apreciate.

It still breaks:

(public-BuvyXxxq) pi@babybuddy:/var/www/babybuddy/public $ pip3 install python-dotenv
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: python-dotenv in /home/pi/.local/share/virtualenvs/public-BuvyXxxq/lib/python3.5/site-packages (0.8.2)
(public-BuvyXxxq) pi@babybuddy:/var/www/babybuddy/public $ gulp migrate
[22:09:47] Using gulpfile /var/www/babybuddy/public/gulpfile.js
[22:09:47] Starting 'migrate'...
Traceback (most recent call last):
  File "manage.py", line 4, in <module>
    from dotenv import load_dotenv, find_dotenv
ImportError: cannot import name 'load_dotenv'
[22:09:50] 'migrate' errored after 2.47 s
[22:09:50] Error: 1
    at formatError (/usr/lib/node_modules/gulp-cli/lib/versioned/^3.7.0/format-error.js:20:10)
    at Gulp.<anonymous> (/usr/lib/node_modules/gulp-cli/lib/versioned/^3.7.0/log/events.js:41:15)
    at emitOne (events.js:116:13)
    at Gulp.emit (events.js:211:7)
    at Gulp.Orchestrator._emitTaskDone (/var/www/babybuddy/public/node_modules/orchestrator/index.js:264:8)
    at /var/www/babybuddy/public/node_modules/orchestrator/index.js:275:23
    at finish (/var/www/babybuddy/public/node_modules/orchestrator/lib/runTask.js:21:8)
    at ChildProcess.cb (/var/www/babybuddy/public/node_modules/orchestrator/lib/runTask.js:29:3)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
(public-BuvyXxxq) pi@babybuddy:/var/www/babybuddy/public $ 

Maybe its not worth all the time i (and you) spend besides our kids.
A docker container is not that much fun, but will maybe help's. I will give it a try.

@cdubz

This comment has been minimized.

Copy link
Owner

cdubz commented May 6, 2018

Hmmm. That is very odd :\

You could give Docker a try, but otherwise I would suggest nuking the environment and starting fresh with the changes I noted in my two comments above: one, two. I still need to figure out how I want to document this or change things up.

But yes! Spend time with the little one(s) (: I have been peppering these responses in during naps while my wife has been out of town, hah.

@novski

This comment has been minimized.

Copy link
Contributor

novski commented May 7, 2018

docker breaks at the same place. I pasted the output here: https://pastebin.com/BLzL6dkW
i used this skript to install: https://get.docker.com/

last few lines:

Step 20/28 : RUN pipenv install --deploy --system
 ---> Running in 14134433a2f7
Installing dependencies from Pipfile.lock (85fe31)…
Ignoring enum34: markers 'python_version == "3.3"' don't match your environment
Looking in indexes: https://pypi.python.org/simple
An error occurred while installing pandas==0.21.1! Will try again.
An error occurred while installing pillow==5.1.0! Will try again.
Installing initially–failed dependencies…
Looking in indexes: https://pypi.python.org/simple
Collecting pandas==0.21.1 
  Using cached https://files.pythonhosted.org/packages/4c/7e/96aaf955f2d8a829b3ca637368c726b5030bac01f22b6e66da3da361f467/pandas-0.21.1.tar.gz

  Could not find a version that satisfies the requirement Cython (from versions: )
No matching distribution found for Cython

ERROR: Service 'app' failed to build: The command '/bin/sh -c pipenv install --deploy --system' returned a non-zero code: 1
@cdubz

This comment has been minimized.

Copy link
Owner

cdubz commented May 7, 2018

I’m not terribly surprised that didn’t work. I’m not particularly familiar with Docker, but I would guess it won’t use piwheels and therefore won’t be able to build some of the requirements for ARM.

@novski

This comment has been minimized.

Copy link
Contributor

novski commented May 7, 2018

yes it worked now with a fresh manual try. I think the failure was to not pienv shell so i added that to the manual and made a PR.
what i did in complete is:

sudo apt-get install python3 python3-pip nginx uwsgi uwsgi-plugin-python3 git libopenjp2-7-dev &&
sudo -H pip3 install pipenv &&
curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash - &&
sudo apt-get install -y nodejs && sudo apt-get update &&
sudo npm install -g gulp-cli &&
sudo mkdir /var/www/babybuddy &&
sudo chown pi:pi /var/www/babybuddy &&
mkdir -p /var/www/babybuddy/data/media &&
git clone https://github.com/cdubz/babybuddy.git /var/www/babybuddy/public &&
cd /var/www/babybuddy/public 
nano Pipfile
1.    changing pandas = "<0.22.0" to pandas = "*" in Pipfile,
2.     changing psycopg2-binary = "*" to psycopg2 = "<2.7.4" in Pipfile, and

pipenv install --three --dev --skip-lock &&
npm install &&
gulp build &&
cp babybuddy/settings/production.example.py babybuddy/settings/production.py &&
editor babybuddy/settings/production.py 

export DJANGO_SETTINGS_MODULE=babybuddy.settings.production &&
gulp collectstatic &&
gulp migrate &&
sudo chown -R www-data:www-data /var/www/babybuddy/data &&
sudo chmod 640 /var/www/babybuddy/data/db.sqlite3 &&
sudo chmod 750 /var/www/babybuddy/data &&
pipenv --venv              --> copy line
sudo editor /etc/uwsgi/apps-available/babybuddy.ini

sudo ln -s /etc/uwsgi/apps-available/babybuddy.ini /etc/uwsgi/apps-enabled/babybuddy.ini &&
sudo service uwsgi restart &&
sudo editor /etc/nginx/sites-available/babybuddy 

sudo ln -s /etc/nginx/sites-available/babybuddy /etc/nginx/sites-enabled/babybuddy &&
sudo service nginx restart

what i do not see and would need to read is how to config the nginx server_name to accept something localhost like "baby.local" or somethin easy. I see a "welcome to nginx" by now. not more...
I don't need a connection from outside of my network just a localhost...
If you know how, maybe you can tell me? Else i will google tomorrow.
I Need some sleep before my baby wakes up again 😇 ...

Regards, Novski

@cdubz

This comment has been minimized.

Copy link
Owner

cdubz commented May 8, 2018

Glad you got it working, too! Check out my notes on #45 and I'll merge it after you make changes.

I'm going to sit on this for a little while before I decide how to improve this situation beyond just those changes to the install process.

@cdubz

This comment has been minimized.

Copy link
Owner

cdubz commented May 12, 2018

bennuttall/piwheels#68 is addressing the psycopg2-binary issue. The next thing I’ll do is see where the building of pandas stands on other platforms. If changes to Pipfile can resolve those two issues there won’t be any real need to provide special instructions for ARM users.

@cdubz

This comment has been minimized.

Copy link
Owner

cdubz commented May 20, 2018

478526a removes the pandas version lock (and drops support for Python 3.4). I'll add a note to the README about --skip-lock and that should wrap this issue up.

cdubz added a commit that referenced this issue May 20, 2018

@cdubz cdubz closed this May 20, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment