Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Lokole's Python dependencies #3276

Merged
merged 8 commits into from Sep 27, 2022

Conversation

mabuelhagag
Copy link
Contributor

@mabuelhagag mabuelhagag commented Jun 30, 2022

Fixes bug:

#2833,#2986, ascoderu/lokole#585

Description of changes proposed in this pull request:

  • Python dependencies are upgraded in the upstream PR.
  • This PR uses a fork until the upstream PR is merged.

/etc/iiab/local_vars.yml:

lokole_install: True
lokole_enabled: True
lokole_client_id: <client ID>
lokole_commit: hotfix/upgrade_python_deps
lokole_sim_type: Ethernet

Smoke-tested on which OS or OS's:

Ubuntu 20.04 Server (Vagrant ubuntu/focal64 box)

Mention a team member @username e.g. to help with code review:

@jvonau @holta

@mabuelhagag
Copy link
Contributor Author

mabuelhagag commented Jun 30, 2022

@nzola Please test this PR by executing the following command:

curl iiab.io/install.txt | sudo bash -s 3276

@mabuelhagag mabuelhagag marked this pull request as ready for review June 30, 2022 15:34
- name: "OPTIONAL: pip install opwen_email_client (Lokole, git commit {{ lokole_commit }}) from GitHub to {{ lokole_venv }}, if lokole_commit is defined"
pip:
name: "git+https://github.com/ascoderu/lokole.git@{{ lokole_commit }}#egg=opwen_email_client"
name: "git+https://github.com/mabuelhagag/lokole.git@{{ lokole_commit }}#egg=opwen_email_client"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Temporary during testing, right?

(Presumably account 'mabuelhagag' will be changed back to 'ascoderu' later?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, until upstream is merged

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might have to drop the 'git+' part and tinker with the rest of the string to get it right, but that is the general idea.

Copy link
Contributor

@jvonau jvonau Jun 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One could move the definition from hardcoded to softcoded and use local_vars to override.

pip:
  name: "{{  lokole_repo  }}" 

then define in defaults/main.yml
lokole_repo: git+https://github.com/ascoderu/lokole.git@{{ lokole_commit }}#egg=opwen_email_client
and then use the override in local_vars.yml
lokole_repo: git+https://github.com/mabuelhagag/lokole.git@{{ lokole_commit }}#egg=opwen_email_client

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check if 191e757 does it

@jvonau
Copy link
Contributor

jvonau commented Jun 30, 2022

Think 'environment=OPWEN_SETTINGS={{ lokole_settings }}' is intended to read a single file holding the environment variables that starts out as templates/settings.env.j2, perhaps 'FLASK_APP="opwen_email_client.webapp:app"' could reside in settings.env.j2?

@jvonau
Copy link
Contributor

jvonau commented Jun 30, 2022

Current PR as of e0ca1f2 works through to login with Admin, looks good so far.

@jvonau
Copy link
Contributor

jvonau commented Jun 30, 2022

The registration webpage where lokole_client_id and auth token could be entered is gone.

@mabuelhagag
Copy link
Contributor Author

Think 'environment=OPWEN_SETTINGS={{ lokole_settings }}' is intended to read a single file holding the environment variables that starts out as templates/settings.env.j2, perhaps 'FLASK_APP="opwen_email_client.webapp:app"' could reside in settings.env.j2?

Flask CLI uses this environment var first to find the app, using the flask manage custom commands need this to initialize.
settings.env is loaded within the app so it needs to precede that.

@mabuelhagag
Copy link
Contributor Author

The registration webpage where lokole_client_id and auth token could be entered is gone.

What does pointing to http;//box/lokole/admin/register gives you?

@jvonau
Copy link
Contributor

jvonau commented Jun 30, 2022

lol.. forgot 'Admin' and 'admin' makes a difference when logging in but only 'admin' is displayed on the logout button... logging in as 'admin' takes me straight to the registration page. Nice work.

@jvonau
Copy link
Contributor

jvonau commented Jun 30, 2022

image

@mabuelhagag
Copy link
Contributor Author

image

Did you set lokole_client_id: <client ID>? None.lokole.ca could indicate that you haven't.

@nzola
Copy link

nzola commented Jul 1, 2022

TASK [lokole : start supervisor] ************************************************************************************************************************************************************************************************************
ok: [127.0.0.1]

TASK [lokole : Create Lokole admin user with password, for http://box/lokole] ***************************************************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": "while read envvar; do export "$envvar"; done < /home/lokole/state/settings.env\ncd /library/lokole/venv/lib/python${python_version}/site-packages/\nexport FLASK_APP="opwen_email_client.webapp:app"\n/library/lokole/venv/bin/flask manage createadmin --name='admin' --password='changeme'\n", "delta": "0:00:00.548691", "end": "2022-07-01 01:31:53.796314", "msg": "non-zero return code", "rc": 1, "start": "2022-07-01 01:31:53.247623", "stderr": "/bin/sh: 2: cd: can't cd to /library/lokole/venv/lib/python/site-packages/\nTraceback (most recent call last):\n File "/library/lokole/venv/bin/flask", line 5, in \n from flask.cli import main\n File "/library/lokole/venv/lib/python3.9/site-packages/flask/init.py", line 14, in \n from jinja2 import escape\nImportError: cannot import name 'escape' from 'jinja2' (/library/lokole/venv/lib/python3.9/site-packages/jinja2/init.py)", "stderr_lines": ["/bin/sh: 2: cd: can't cd to /library/lokole/venv/lib/python/site-packages/", "Traceback (most recent call last):", " File "/library/lokole/venv/bin/flask", line 5, in ", " from flask.cli import main", " File "/library/lokole/venv/lib/python3.9/site-packages/flask/init.py", line 14, in ", " from jinja2 import escape", "ImportError: cannot import name 'escape' from 'jinja2' (/library/lokole/venv/lib/python3.9/site-packages/jinja2/init.py)"], "stdout": "", "stdout_lines": []}

TASK [lokole : SEE ERROR ABOVE (skip_role_on_error: False)] *********************************************************************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": ""}

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
127.0.0.1 : ok=271 changed=148 unreachable=0 failed=1 skipped=58 rescued=1 ignored=2

pi@raspberrypi:~ $

@nzola
Copy link

nzola commented Jul 1, 2022

@mabuelhagag @jvonau @holta

@holta
Copy link
Member

holta commented Jul 1, 2022

Thanks @nzola

In general please also post the sprunge.us pastebin URL — after you run:

iiab-diagnostics

@mabuelhagag
Copy link
Contributor Author

Looks like python_version variable wasn't picked up. Pushed a fix at 6a55a11 . Try it now.

@holta
Copy link
Member

holta commented Jul 1, 2022

Looks like python_version variable wasn't picked up. Pushed a fix at 6a55a11 . Try it now.

@nzola should try the following 3 lines to try to rescue his current install attempt:

cd /opt/iiab/iiab

sudo git pull https://github.com/mabuelhagag/iiab bugfix/upgrade_lokole

sudo iiab

No need to start over with a completely clean/fresh OS (hopefully!)

@nzola
Copy link

nzola commented Jul 1, 2022

@mabuelhagag @holta I did as recommended by @holta , but end up with the same error. See details:

pi@raspberrypi:~ $ cd /opt/iiab/iiab
pi@raspberrypi:/opt/iiab/iiab $ sudo git pull https://github.com/mabuelhagag/iiab bugfix/upgrade_lokole
hint: Pulling without specifying how to reconcile divergent branches is
hint: discouraged. You can squelch this message by running one of the following
hint: commands sometime before your next pull:
hint:
hint: git config pull.rebase false # merge (the default strategy)
hint: git config pull.rebase true # rebase
hint: git config pull.ff only # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
remote: Enumerating objects: 28, done.
remote: Counting objects: 100% (28/28), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 20 (delta 15), reused 16 (delta 11), pack-reused 0
Unpacking objects: 100% (20/20), 1.57 KiB | 40.00 KiB/s, done.
From https://github.com/mabuelhagag/iiab

  • branch bugfix/upgrade_lokole -> FETCH_HEAD
    Merge made by the 'recursive' strategy.
    roles/lokole/defaults/main.yml | 3 ++-
    roles/lokole/tasks/install.yml | 1 -
    roles/lokole/tasks/setup.yml | 1 +
    3 files changed, 3 insertions(+), 2 deletions(-)
    pi@raspberrypi:/opt/iiab/iiab $ sudo iiab

@holta
Copy link
Member

holta commented Jul 1, 2022

@nzola if you try this again, run:

cd /opt/iiab/iiab

sudo git pull --no-rebase https://github.com/mabuelhagag/iiab bugfix/upgrade_lokole

sudo iiab

@nzola
Copy link

nzola commented Jul 1, 2022

@holta Ok I am going to run what you have just sent. But I have a question

I originally used this command to install iiab : curl iiab.io/install.txt | sudo bash -s 3276 as directed.There is no d. before iiab does it matter?

@nzola
Copy link

nzola commented Jul 1, 2022

same error after I run this:

pi@raspberrypi:/opt/iiab/iiab $ cd /opt/iiab/iiab
pi@raspberrypi:/opt/iiab/iiab $ sudo git pull --no-rebase https://github.com/mabuelhagag/iiab bugfix/upgrade_lokole
From https://github.com/mabuelhagag/iiab

  • branch bugfix/upgrade_lokole -> FETCH_HEAD
    Already up to date.
    pi@raspberrypi:/opt/iiab/iiab $ sudo iiab

@holta
Copy link
Member

holta commented Jul 1, 2022

There is no d. before [iiab.io] does it matter?

It does not matter.

@holta
Copy link
Member

holta commented Jul 1, 2022

same error after I run this:

Thanks @nzola.

This should hopefully help @mabuelhagag figure out his next steps.

@holta
Copy link
Member

holta commented Jul 1, 2022

@nzola: Can you run iiab-diagnostics (and post the sprunge.us URL result) to give everyone a bit more context?

@nzola
Copy link

nzola commented Jul 1, 2022

@holta Done!
PUBLISHING TO URL...
http://sprunge.us/rnDwnQ

@holta
Copy link
Member

holta commented Jul 1, 2022

FYI @mabuelhagag add ?en to make iiab-diagnostics more readable in browser:

http://sprunge.us/rnDwnQ?en

@jvonau
Copy link
Contributor

jvonau commented Jul 5, 2022

Did you set lokole_client_id: ? None.lokole.ca could indicate that you haven't.

No, I did not set the variable, should really be assigned by nzola to keep duplicates out of the picture when a request for the required auth-token is made.

@jvonau
Copy link
Contributor

jvonau commented Jul 5, 2022

Think 'environment=OPWEN_SETTINGS={{ lokole_settings }}' is intended to read a single file holding the environment variables that starts out as templates/settings.env.j2, perhaps 'FLASK_APP="opwen_email_client.webapp:app"' could reside in settings.env.j2?

Flask CLI uses this environment var first to find the app, using the flask manage custom commands need this to initialize. settings.env is loaded within the app so it needs to precede that.

https://flask.palletsprojects.com/en/2.1.x/cli/#dotenv suggest that .flaskenv could hold the FLASK_APP= part but would need python-dotenv in the virtual environment, which looks to be installed, most likely dragged in by some other python library as a dependency.

ls /library/lokole/venv/bin

Activate.ps1   alembic          flask        normalizer  pybabel     watchmedo
activate       celery           gunicorn     pip         python
activate.csh   dotenv           mako-render  pip3        python3
activate.fish  email_validator  mkwvconf     pip3.10     python3.10

cd /library/lokole/venv
ubuntu@libraryimg:/library/lokole/venv$ source bin/activate
(venv) ubuntu@libraryimg:/library/lokole/venv$ pip list

Package               Version
--------------------- -----------
alembic               1.8.0
amqp                  5.1.1
apache-libcloud       3.6.0
Babel                 2.10.3
bcrypt                3.2.2
beautifulsoup4        4.11.1
billiard              3.6.4.0
blinker               1.4
cached-property       1.5.2
cachelib              0.9.0
celery                5.2.7
certifi               2022.6.15
cffi                  1.15.1
charset-normalizer    2.1.0
click                 8.1.3
click-didyoumean      0.3.0
click-plugins         1.1.1
click-repl            0.2.0
dnspython             2.2.1
email-validator       1.2.1
environs              9.5.0
fasteners             0.17.3
Flask                 2.1.2
Flask-BabelEx         0.9.4
Flask-Caching         2.0.0
Flask-Cors            3.0.10
Flask-Login           0.6.1
Flask-Mail            0.9.1
Flask-Migrate         3.1.0
Flask-Principal       0.4.0
Flask-Script          2.0.6
Flask-Security        3.0.0
Flask-SQLAlchemy      2.5.1
Flask-Testing         0.8.1
Flask-WTF             1.0.1
greenlet              1.1.2
gunicorn              20.1.0
idna                  3.3
itsdangerous          2.1.2
Jinja2                3.1.2
kombu                 5.2.4
Mako                  1.2.1
MarkupSafe            2.1.1
marshmallow           3.17.0
mkwvconf              0.1.1
opwen-email-client    0.0.0
packaging             21.3
passlib               1.7.4
Pillow                9.1.1
pip                   22.0.2
prompt-toolkit        3.0.30
pycparser             2.21
pyparsing             3.0.9
python-crontab        2.6.0
python-dateutil       2.8.2
python-dotenv         0.20.0
pytz                  2022.1
pytz-deprecation-shim 0.1.0.post0
requests              2.28.1
setuptools            59.6.0
six                   1.16.0
soupsieve             2.3.2.post1
speaklater            1.3
SQLAlchemy            1.4.39
types-requests        2.28.0
types-urllib3         1.26.15
typing                3.7.4.3
tzdata                2022.1
tzlocal               4.2
urllib3               1.26.9
vine                  5.0.0
watchdog              2.1.9
wcwidth               0.2.5
Werkzeug              2.1.2
WTForms               3.0.1
xtarfile              0.1.0
zstandard             0.18.0

@mabuelhagag
Copy link
Contributor Author

Lokole 0.8.2 is now released.
This PR can be tested now using the following vars on /etc/iiab/local_vars.yml::

lokole_install: True
lokole_enabled: True
lokole_client_id: <client ID>
lokole_sim_type: Ethernet

@jvonau @holta
@nzola Please check if everything works as it should.

@nzola
Copy link

nzola commented Aug 9, 2022

Ok I am going to test now.

@nzola
Copy link

nzola commented Aug 9, 2022

TASK [lokole : Create Lokole admin user with password, for http://box/lokole] ******************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": "while read envvar; do export "$envvar"; done < /home/lokole/state/settings.env\n/library/lokole/venv/bin/manage.py createadmin --name='admin' --password='changeme'\n", "delta": "0:00:00.009542", "end": "2022-08-09 09:28:22.794083", "msg": "non-zero return code", "rc": 127, "start": "2022-08-09 09:28:22.784541", "stderr": "/bin/sh: 2: /library/lokole/venv/bin/manage.py: not found", "stderr_lines": ["/bin/sh: 2: /library/lokole/venv/bin/manage.py: not found"], "stdout": "", "stdout_lines": []}

TASK [lokole : SEE ERROR ABOVE (skip_role_on_error: False)] ************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": ""}

PLAY RECAP *************************************************************************************************************************************************************
127.0.0.1 : ok=40 changed=7 unreachable=0 failed=1 skipped=18 rescued=1 ignored=0

pi@raspberrypi:~ $

@nzola
Copy link

nzola commented Aug 9, 2022

@mabuelhagag The install error I have posted here above had been fixed by you before. In fact this was the first problem you fixed when you joined the project.
This problem also affects Lokole server status board. See print screen :

image

@mabuelhagag
Copy link
Contributor Author

mabuelhagag commented Aug 9, 2022

@nzola It seems that you're not using the PR code.
Please re-run with:

curl iiab.io/install.txt | sudo bash -s 3276

@nzola
Copy link

nzola commented Aug 9, 2022

@mabuelhagag You are right I used this curl iiab.io/install.txt | sudo bash instead
It is 4 am I am going to bed but I will leave it installing and will check it in the morning to tell you the result before going to work and I will do testing in the afternoon after work

@mabuelhagag
Copy link
Contributor Author

No worries.
FYI. I did a quick test and testing and receiving plain text emails worked for me.

@holta
Copy link
Member

holta commented Aug 9, 2022

@mabuelhagag if this PR #3276 is carefully tested to be safe to merge, please let us know, Thanks!

@mabuelhagag
Copy link
Contributor Author

@holta If Nzola reports back that it's working fine then we can merge.

@nzola
Copy link

nzola commented Aug 9, 2022

@mabuelhagag
@mabuelhagag
@jvonau
The install went thru without any problem as indicated. The failure I reported here above was my fault. I did not include -s 3276 at the end of the script. I did not have chance to test the Lokole mail operation as I had only 2 hours of the night left before going to work in the morning. I am at work right now. I will do tests this afternoon when I return home and report as soon as I can.

@nzola
Copy link

nzola commented Aug 9, 2022

@holta
@mabuelhagag
@jvonau

I have just finished testing Lokole email system.
Mail transactions between users and admin, Ok
Users registrations, Ok

Client cannot register onto the main server
Main main server status dashboard http://ascoderu.ca/lokole/ is not functioning.

@nzola
Copy link

nzola commented Aug 9, 2022

Because the client cannot register to the main server, mail transaction doesn't flow between the client users and the internet email addresses.

@nzola
Copy link

nzola commented Aug 27, 2022

I have extensively tested Lokole email functionality under PR 3276 and the result is positive.
Client registration and mail transactions between local users and transfer to the internet all are flowing without any problems.
I would say it is good to merge.

@holta
Copy link
Member

holta commented Sep 27, 2022

@nzola:

I just got a message from @mabuelhagag saying merge this, so I'm doing that now, and further polish can be added later.

Thanks to @jvonau who's helped along the way, keeping an eye on this.

Related:

@holta holta merged commit 40db09c into iiab:master Sep 27, 2022
@mabuelhagag
Copy link
Contributor Author

Thank you guys for your support. Feel free to report any issues that might occur and assign it to me.

@nzola
Copy link

nzola commented Sep 27, 2022 via email

@nzola
Copy link

nzola commented Sep 28, 2022

@mabuelhagag @jvonau @holta
PUBLISHING TO URL...
http://sprunge.us/wIwPKB

iiab install when very well nonstop from start to end. I selected size 1 install and it took about 45 minutes to complete on PI4 B 2 GIG. I used PI OS 64 bit Lite.
1 Lokole emails transaction between local users within local network is working without any problems
2 Lokole admin cannot register the client to the main server (Azure cloud). This means Lokole box will not be able to exchange emails with the world. though it exchanges emails within the local network. as I mentioned here above.
3 The Lokole newer logo does not show on the iiab main menu, only he old generic logo
4 Now I am in the process to install French bouquet and I will report.

@holta
Copy link
Member

holta commented Sep 28, 2022

3 The Lokole newer logo does not show on the iiab main menu, only he old generic logo

Likely because you did not force a full SHIFT-reload (refresh) in your browser. Hold down the SHIFT key on your keyboard — as you click reload in your browser. (You need to clear your browser cache.)

Use a completely different client device and completely different browser to verify.

(As you can see below, the logo was changed in late August...)

https://github.com/iiab-share/js-menu-files/blob/master/menu-defs/en-lokole.json

https://github.com/iiab-share/js-menu-files/blob/master/images/lokole-logo-200x105.png

@nzola
Copy link

nzola commented Sep 28, 2022 via email

@nzola
Copy link

nzola commented Sep 29, 2022

3 The Lokole newer logo does not show on the iiab main menu, only he old generic logo

Likely because you did not force a full SHIFT-reload (refresh) in your browser. Hold down the SHIFT key on your keyboard — as you click reload in your browser. (You need to clear your browser cache.)

Use a completely different client device and completely different browser to verify.

(As you can see below, the logo was changed in late August...)

https://github.com/iiab-share/js-menu-files/blob/master/menu-defs/en-lokole.json

https://github.com/iiab-share/js-menu-files/blob/master/images/lokole-logo-200x105.png

@holta
I did SHFT-refresh . it has fixed the Logo problem.

@nzola
Copy link

nzola commented Sep 29, 2022

@holta French bouquet install completed without any problem and nonstop from start to finish

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants