Upgrade installation scripts and documentation #1332

return42 · 2022-06-16T17:06:15Z

A preview of the documentation about the upgrade is available from here:

https://return42.github.io/searxng/admin/update-searxng.html#migrate-and-stay-tuned

HINT: last patch on top needs to be removed before this PR is merged!

[brand] branch: searxng-install https://github.com/return42/searxng

This brand is needed to test the installation procedures as long this PR is not merged into master.

What does this PR do?

- [utils/searxng.sh] implement new script to install SearXNG
- [doc] update documentation of the installation procedures
- [clean up] drop obsolete searx, filtron and morty install scripts
- [docs] add section "Migrate and stay tuned!"

Why is this change important?

The infrastructure of SearXNG has changed dramatically over the past year. The installation scripts and documentation have not been adapted so far.

How to test this PR locally?

First read the admin documentation, to see what have been changed ..

make docs.live

Test new ./utils/searxng.sh script ..

in LXCs, first build containers and install SearXNG by install all:

sudo -H ./utils/lxc.sh build
sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install all

Choose to install nginx:

sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx

or install apache HTTP server:

sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install apache

Run instance check:

sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh instance check

Now you can test the SearXNG instances from the conatiners, to get the URLs of the containers use:

sudo -H ./utils/lxc.sh show suite | grep SEARXNG_URL
...
[searxng-ubu2004]    SEARXNG_URL          : http://10.146.148.203/searxng
[searxng-ubu2110]    SEARXNG_URL          : http://10.146.148.219/searxng
[searxng-fedora35]    SEARXNG_URL          : http://10.146.148.64/searxng
[searxng-archlinux]    SEARXNG_URL          : http://10.146.148.135/searxng

Settings see https://github.com/return42/searxng/blob/searxng-install/utils/templates/etc/searxng/settings.yml#L3-L22

Features:

the limiter should work (except on fedora35, see "Pitfalls of the Tyrant mode" in the uWSGI admn-doc)
image_proxy should work on all instances
cache busting (aka static_use_hash) should work

closes: #1112

return42 · 2022-06-21T17:13:09Z

FYI: rebased branch and append

b5ecade [docs] corrections from @tiekoetter's review

Thank you, @tiekoetter 👍

dalf · 2022-06-25T11:45:12Z

Running:
sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install all

There are warning about the virtualenv:

sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install all


[searxng-ubu2004] Create virtualenv (python)
[searxng-ubu2004] --------------------------
[searxng-ubu2004] 
[searxng-ubu2004] ERROR: ./utils/searxng.sh: line 474: To create pyenv for SearXNG, first install searxng-src.
WARN:  [searxng-ubu2004] exit code (42) from FORCE_TIMEOUT=0 ./utils/searxng.sh install all
INFO:  [searxng-ubu2110] FORCE_TIMEOUT=0 ./utils/searxng.sh install all


[searxng-ubu2110] Create virtualenv (python)
[searxng-ubu2110] --------------------------
[searxng-ubu2110] 
[searxng-ubu2110] ERROR: ./utils/searxng.sh: line 474: To create pyenv for SearXNG, first install searxng-src.
WARN:  [searxng-ubu2110] exit code (42) from FORCE_TIMEOUT=0 ./utils/searxng.sh install all
INFO:  [searxng-fedora35] FORCE_TIMEOUT=0 ./utils/searxng.sh install all


[searxng-fedora35] Create virtualenv (python)
[searxng-fedora35] --------------------------
[searxng-fedora35] 
[searxng-fedora35] ERROR: ./utils/searxng.sh: line 474: To create pyenv for SearXNG, first install searxng-src.
WARN:  [searxng-fedora35] exit code (42) from FORCE_TIMEOUT=0 ./utils/searxng.sh install all
INFO:  [searxng-archlinux] FORCE_TIMEOUT=0 ./utils/searxng.sh install all


[searxng-archlinux] Clone SearXNG sources
[searxng-archlinux] ---------------------
[searxng-archlinux] 
[searxng-archlinux] WARN:  missing local branch searxng-install
[searxng-archlinux] INFO:  create local branch searxng-install from start point: origin/searxng-install
[searxng-archlinux] fatal: not a valid object name: 'origin/searxng-install'
[searxng-archlinux] WARN:  take into account, installing branch searxng-install while current branch is return42-searxng-install
[searxng-archlinux] INFO:  clone into: /usr/local/searxng/searxng-src
[searxng-archlinux] mkdir -p "/usr/local/searxng"
[searxng-archlinux] cd "/usr/local/searxng"
[searxng-archlinux] git clone --branch "searxng-install" --origin "origin" "/share/searxng" "searxng-src"
[searxng-archlinux]   |searxng| Cloning into 'searxng-src'...
[searxng-archlinux]   |searxng| fatal: Remote branch searxng-install not found in upstream origin
[searxng-archlinux] ./utils/searxng.sh: line 459: pushd: /usr/local/searxng/searxng-src: No such file or directory
[searxng-archlinux] cd "/usr/local/searxng/searxng-src"
[searxng-archlinux]   |searxng| -bash: line 1: cd: /usr/local/searxng/searxng-src: No such file or directory
[searxng-archlinux] git remote set-url origin https://github.com/return42/searxng
[searxng-archlinux]   |searxng| fatal: not a git repository (or any of the parent directories): .git
[searxng-archlinux] git config user.email "root@searxng-archlinux"
[searxng-archlinux]   |searxng| fatal: not in a git directory
[searxng-archlinux] git config user.name "root"
[searxng-archlinux]   |searxng| fatal: not in a git directory
[searxng-archlinux] git config --list
[searxng-archlinux]   |searxng| safe.directory=/usr/local/searxng/searxng-src
[searxng-archlinux] ./utils/searxng.sh: line 467: popd: directory stack empty
[searxng-archlinux] ** press any [KEY] to continue **
[searxng-archlinux] 
[searxng-archlinux] Create virtualenv (python)
[searxng-archlinux] --------------------------
[searxng-archlinux] 
[searxng-archlinux] ERROR: ./utils/searxng.sh: line 474: To create pyenv for SearXNG, first install searxng-src.
WARN:  [searxng-archlinux] exit code (42) from FORCE_TIMEOUT=0 ./utils/searxng.sh install all

The command ends with the line above and the exit code 42. Is it normal ?

https://github.com/dalf/searxng-install : the scripts runs in a GitHub Action. "Of course" there is a network issue: the gateway is reachable but not further. I wish to fix this issue so we can run these tests automatically.

return42 · 2022-06-25T16:07:18Z

the scripts runs in a GitHub Action.

Would be cool to have an automated test 👍

"Of course" there is a network issue:

Without a network no packages can be installed and the SearXNG brand (repo) can't be cloned.

The command ends with the line above and the exit code 42. Is it normal ?

Yes, the function searxng.install.pyenv check if a clone exists, if not, it ends with exit code 42 ..

https://github.com/return42/searxng/blob/searxng-install/utils/searxng.sh#L470-L474

You will see the same exit code, if you do not install all but calling ./utils/searxng.sh install pyenv without having cloned before (./utils/searxng.sh install searxng-src).

https://github.com/return42/searxng/blob/searxng-install/utils/searxng.sh#L133

dalf · 2022-06-25T17:23:09Z

Yes, the function searxng.install.pyenv check if a clone exists, if not, it ends with exit code 42 ..

I've tried the next step: sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx.

The full log:

INFO:  [searxng-ubu2004] FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
[searxng-ubu2004] 
[searxng-ubu2004] Install nginx site searxng.conf
[searxng-ubu2004] ===============================
[searxng-ubu2004] 
[searxng-ubu2004] This installs SearXNG's uWSGI app as Nginx site.  The Nginx site is
[searxng-ubu2004] located at: /etc/nginx/default.apps-available/searxng.conf and requires
[searxng-ubu2004] a uWSGI.
[searxng-ubu2004] 
[searxng-ubu2004] To install uWSGI use::
[searxng-ubu2004] 
[searxng-ubu2004]     searxng.sh install uwsgi
[searxng-ubu2004] 
[searxng-ubu2004] ERROR: ./utils/searxng.sh: line 684: SearXNG's uWSGI app not available
WARN:  [searxng-ubu2004] exit code (42) from FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
INFO:  [searxng-ubu2110] FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
[searxng-ubu2110] 
[searxng-ubu2110] Install nginx site searxng.conf
[searxng-ubu2110] ===============================
[searxng-ubu2110] 
[searxng-ubu2110] This installs SearXNG's uWSGI app as Nginx site.  The Nginx site is
[searxng-ubu2110] located at: /etc/nginx/default.apps-available/searxng.conf and requires
[searxng-ubu2110] a uWSGI.
[searxng-ubu2110] 
[searxng-ubu2110] To install uWSGI use::
[searxng-ubu2110] 
[searxng-ubu2110]     searxng.sh install uwsgi
[searxng-ubu2110] 
[searxng-ubu2110] ERROR: ./utils/searxng.sh: line 684: SearXNG's uWSGI app not available
WARN:  [searxng-ubu2110] exit code (42) from FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
INFO:  [searxng-fedora35] FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
[searxng-fedora35] 
[searxng-fedora35] Install nginx site searxng.conf
[searxng-fedora35] ===============================
[searxng-fedora35] 
[searxng-fedora35] This installs SearXNG's uWSGI app as Nginx site.  The Nginx site is
[searxng-fedora35] located at: /etc/nginx/default.apps-available/searxng.conf and requires
[searxng-fedora35] a uWSGI.
[searxng-fedora35] 
[searxng-fedora35] To install uWSGI use::
[searxng-fedora35] 
[searxng-fedora35]     searxng.sh install uwsgi
[searxng-fedora35] 
[searxng-fedora35] ERROR: ./utils/searxng.sh: line 684: SearXNG's uWSGI app not available
WARN:  [searxng-fedora35] exit code (42) from FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
INFO:  [searxng-archlinux] FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
[searxng-archlinux] 
[searxng-archlinux] Install nginx site searxng.conf
[searxng-archlinux] ===============================
[searxng-archlinux] 
[searxng-archlinux] This installs SearXNG's uWSGI app as Nginx site.  The Nginx site is
[searxng-archlinux] located at: /etc/nginx/default.apps-available/searxng.conf and requires
[searxng-archlinux] a uWSGI.
[searxng-archlinux] 
[searxng-archlinux] To install uWSGI use::
[searxng-archlinux] 
[searxng-archlinux]     searxng.sh install uwsgi
[searxng-archlinux] 
[searxng-archlinux] ERROR: ./utils/searxng.sh: line 684: SearXNG's uWSGI app not available
WARN:  [searxng-archlinux] exit code (42) from FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx

return42 · 2022-06-26T10:39:26Z

I've tried the next step: sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx.

Sorry, I do not understand the intention of your question.

Isn't it clear, that all is just a command that calls other commands in the needed order:

https://github.com/return42/searxng/blob/searxng-install/utils/searxng.sh#L256-L287

If one command fails it does not make sense to continue with the next; When there is no clone, you can't install a pyenv, when there is no uwsgi service installed, you can't install a HTTP service for this uWSGI service .. and when there is a network issue it makes no sense to try an installation, since nothin can be loaded and installed from package manager repositories.

dalf · 2022-06-26T18:29:09Z

and when there is a network issue it makes no sense to try an installation

The network issue is from this GitHub workflow (disabled for now)

The log above comes from my try on my laptop: there is no network issue.
To make sure, I've tried again, so I've stopped all the containers, remove them and run the script again.

sudo -H ./utils/lxc.sh build went well
sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install all stops with some errors:

[searxng-ubu2004] Finalize installation
[searxng-ubu2004] =====================
[searxng-ubu2004] 
[searxng-ubu2004] Do you want to run some checks? [YES/no] Yes
[searxng-ubu2004] INFO:  wrapper:  utils/searxng.sh instance _call searxng.check
[searxng-ubu2004] -bash: /usr/local/searxng/searxng-src/utils/searxng.sh: No such file or directory
WARN:  [searxng-ubu2004] exit code (127) from FORCE_TIMEOUT=0 ./utils/searxng.sh install all


[searxng-ubu2110] Finalize installation
[searxng-ubu2110] =====================
[searxng-ubu2110] 
[searxng-ubu2110] Do you want to run some checks? [YES/no] Yes
[searxng-ubu2110] INFO:  wrapper:  utils/searxng.sh instance _call searxng.check
[searxng-ubu2110] -bash: line 1: /usr/local/searxng/searxng-src/utils/searxng.sh: No such file or directory
WARN:  [searxng-ubu2110] exit code (127) from FORCE_TIMEOUT=0 ./utils/searxng.sh install all


[searxng-fedora35] Finalize installation
[searxng-fedora35] =====================
[searxng-fedora35] 
[searxng-fedora35] Do you want to run some checks? [YES/no] Yes
[searxng-fedora35] INFO:  wrapper:  utils/searxng.sh instance _call searxng.check
[searxng-fedora35] -bash: line 1: /usr/local/searxng/searxng-src/utils/searxng.sh: No such file or directory
WARN:  [searxng-fedora35] exit code (127) from FORCE_TIMEOUT=0 ./utils/searxng.sh install all
INFO:  [searxng-archlinux] FORCE_TIMEOUT=0 ./utils/searxng.sh install all
[searxng-archlinux] ERROR: missing command wget
WARN:  [searxng-archlinux] exit code (42) from FORCE_TIMEOUT=0 ./utils/searxng.sh install all

In addition to the test above I have updated https://github.com/dalf/searxng-install :

It is not surprising reading the doc, however I'm not sure how to solve that.

return42 · 2022-06-27T09:22:14Z

The network issue is from this GitHub workflow (disabled for now)
The log above comes from my try on my laptop: there is no network issue.
To make sure, I've tried again, so I've stopped all the containers, remove them and run the script again.

Ah, OK .. thats good .. if you have problems building all containers, I recommend to run your first test in a single container, e.g. I often use archlinux for a first test; bevore remove all containers and all images (I have seen some issues building all containers due to the limited size of the lxc storage where all the containers and images are stored)

sudo -H ./utils/lxc.sh remove containers
sudo -H ./utils/lxc.sh remove images
sudo -H ./utils/lxc.sh cmd searxng-archlinux FORCE_TIMEOUT=0 ./utils/searxng.sh install all

Does this work on your laptop ..

If yes, install ngingx:

 sudo -H ./utils/lxc.sh cmd searxng-archlinux FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx

I not, please read the log thoroughly, are there any ERROR messages .. please post it here to see where it might fail.

The message you posted above:

  [searxng-ubu2004] -bash: /usr/local/searxng/searxng-src/utils/searxng.sh: No such file or directory
  [searxng-ubu2110] -bash: line 1: /usr/local/searxng/searxng-src/utils/searxng.sh: No such file or directory

shows me that the clone was not built .. and the nessage

 [searxng-archlinux] ERROR: missing command wget

shows me, that the wget command from the archlinix package manager was not installed for some reason, I assume you have no more space left on your lxc storage. And this is the reason why the installation in the containers fails at different points (do you see a "no space left" message anywhere in the log?).

About CI & LXC ..

It is not surprising reading the doc, however I'm not sure how to solve that.

:-) .. after the clone ..

https://github.com/dalf/searxng-install/blob/main/.github/workflows/test2.yml#L17

and before the install all add a chmod ugo+rwx -R searxng .. does it help?

dalf · 2022-06-27T17:05:26Z

Ok I figured out the issue: I created a new branch from master and then merge the changes from this branch.
Once I checked out return42/searxng-install it works as intended.

the log about "Clone SearXNG sources" with the previous configuration (new branch from master and then merge searxng-install)

Clone SearXNG sources
---------------------

WARN:  missing local branch searxng-install
INFO:  create local branch searxng-install from start point: origin/searxng-install
fatal: not a valid object name: 'origin/searxng-install'
WARN:  take into account, installing branch searxng-install while current branch is return42-searxng-install
INFO:  clone into: /usr/local/searxng/searxng-src
mkdir -p "/usr/local/searxng"
cd "/usr/local/searxng"
git clone --branch "searxng-install" --origin "origin" "/share/searxng" "searxng-src"
  |searxng| Cloning into 'searxng-src'...
  |searxng| fatal: Remote branch searxng-install not found in upstream origin
./utils/searxng.sh: line 459: pushd: /usr/local/searxng/searxng-src: No such file or directory
cd "/usr/local/searxng/searxng-src"
  |searxng| -bash: line 1: cd: /usr/local/searxng/searxng-src: No such file or directory
git remote set-url origin https://github.com/return42/searxng
  |searxng| fatal: not a git repository (or any of the parent directories): .git
git config user.email "root@searxng-archlinux"
  |searxng| fatal: not in a git directory
git config user.name "root"
  |searxng| fatal: not in a git directory
git config --list
  |searxng| safe.directory=/usr/local/searxng/searxng-src
./utils/searxng.sh: line 467: popd: directory stack empty
** press any [KEY] to continue **

Create virtualenv (python)
--------------------------

ERROR: ./utils/searxng.sh: line 474: To create pyenv for SearXNG, first install searxng-src.
WARN:  [searxng-archlinux] exit code (42) from FORCE_TIMEOUT=0 ./utils/searxng.sh install all

and before the install all add a chmod ugo+rwx -R searxng .. does it help?

It seems to be a chicken / egg problem: after then git clone, the searxng user does not exist.

why doesn't the script do the chown?

See the log: https://github.com/dalf/searxng-install/runs/7062375666?check_suite_focus=true#step:2:180

return42 · 2022-07-07T16:48:06Z

why doesn't the script do the chown?

The script should not change permissions of objects that are not a part of the installation itself .. the git clone in this folder is a "tool" to install, but it is not a part of the installation itself. I hope it was understandable what I wanted to express.

return42 · 2022-07-22T16:17:10Z

FYI: in preparation to merge this PR, the branch has been rebased on master and the brand-commit has been removed (see HINT)

A preview of the documentation about the upgrade is available from here:

https://return42.github.io/searxng/admin/update-searxng.html#migrate-and-stay-tuned

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

Since ./utils/searxng.sh is implemented, the old installation procedures from filtron, morty and searx can be removed. For users who want to upgrade, the procedures for removing old installations have still been retained. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

Related: searxng#1332 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

[mod] fix minor leftovers from PR #1332

return42 force-pushed the searxng-install branch 4 times, most recently from 00a532b to d381ac4 Compare June 17, 2022 17:24

return42 marked this pull request as ready for review June 17, 2022 17:35

return42 requested review from dalf, mrpaulblack and tiekoetter June 17, 2022 17:36

return42 mentioned this pull request Jun 21, 2022

Added more detailed searxng.conf for apache #1355

Closed

return42 force-pushed the searxng-install branch 2 times, most recently from 3e0c305 to 13c5ac9 Compare June 21, 2022 17:11

This was referenced Jul 4, 2022

Bump github.com/valyala/fasthttp from 1.37.0 to 1.38.0 searxng/filtron#12

Closed

bug fix: uwsgi log file from searx to searxng #1447

Closed

dalf approved these changes Jul 8, 2022

View reviewed changes

return42 mentioned this pull request Jul 11, 2022

limiter plugin http error Too Many Requests and debug log shows searx.plugins.limiter: got connection=close when using via uwsgi Unix Domain Socket #1492

Closed

return42 force-pushed the searxng-install branch from 13c5ac9 to 25dc5ad Compare July 22, 2022 16:14

return42 force-pushed the searxng-install branch from 25dc5ad to 58b7e67 Compare July 22, 2022 16:27

return42 mentioned this pull request Jul 23, 2022

pass morty key as env variable #1538

Closed

return42 added 2 commits July 30, 2022 13:39

[install scripts] rename SEARX_<name> variables to SEARXNG_<name>

81bba44

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

[utils/searxng.sh] implement new script to install SearXNG

782f735

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

return42 and others added 4 commits July 30, 2022 13:39

[doc] update documentation of the installation procedures

ed8a169

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

[docs] add section "Migrate and stay tuned!"

6fbffe9

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

[docs] corrections from @tiekoetter's review

e5323b8

return42 force-pushed the searxng-install branch from 58b7e67 to e5323b8 Compare July 30, 2022 11:39

return42 merged commit 645c2a2 into searxng:master Jul 30, 2022

return42 deleted the searxng-install branch July 30, 2022 11:45

return42 added a commit to return42/searxng that referenced this pull request Jul 30, 2022

[mod] fix minor leftovers from PR searxng#1332

bded875

Related: searxng#1332 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

return42 mentioned this pull request Jul 30, 2022

[mod] fix minor leftovers from PR #1332 #1589

Merged

return42 added a commit that referenced this pull request Jul 30, 2022

Merge pull request #1589 from return42/searxng-install-additions

ededaab

[mod] fix minor leftovers from PR #1332

return42 mentioned this pull request Oct 5, 2024

Remove morty and filtron leftovers #3888

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade installation scripts and documentation #1332

Upgrade installation scripts and documentation #1332

return42 commented Jun 16, 2022 •

edited

Loading

return42 commented Jun 21, 2022

dalf commented Jun 25, 2022 •

edited

Loading

return42 commented Jun 25, 2022

dalf commented Jun 25, 2022

return42 commented Jun 26, 2022

dalf commented Jun 26, 2022 •

edited

Loading

return42 commented Jun 27, 2022

dalf commented Jun 27, 2022 •

edited

Loading

return42 commented Jul 7, 2022

return42 commented Jul 22, 2022

Upgrade installation scripts and documentation #1332

Upgrade installation scripts and documentation #1332

Conversation

return42 commented Jun 16, 2022 • edited Loading

What does this PR do?

Why is this change important?

How to test this PR locally?

return42 commented Jun 21, 2022

dalf commented Jun 25, 2022 • edited Loading

return42 commented Jun 25, 2022

dalf commented Jun 25, 2022

return42 commented Jun 26, 2022

dalf commented Jun 26, 2022 • edited Loading

return42 commented Jun 27, 2022

dalf commented Jun 27, 2022 • edited Loading

return42 commented Jul 7, 2022

return42 commented Jul 22, 2022

return42 commented Jun 16, 2022 •

edited

Loading

dalf commented Jun 25, 2022 •

edited

Loading

dalf commented Jun 26, 2022 •

edited

Loading

dalf commented Jun 27, 2022 •

edited

Loading