What searx option is a part of the brand and what is a part of instance settings.yml ? #2394

dalf · 2020-12-16T10:06:35Z

Currently:

Lines 1 to 6 in 002d53a

    
           GIT_URL = 'https://github.com/searx/searx' 
        
           GIT_BRANCH = 'master' 
        
           ISSUE_URL = 'https://github.com/searx/searx/issues' 
        
           SEARX_URL = 'https://searx.me' 
        
           DOCS_URL = 'https://searx.github.io/searx' 
        
           PUBLIC_INSTANCES = 'https://searx.space'

SEARX_URL is very similar to

searx/searx/settings.yml

Line 16 in 002d53a

base_url : False # Set custom base_url. Possible values: False or "https://your.custom.host/location/"
GIT_URL usage:

searx/utils/makefile.sphinx

Line 180 in 002d53a

[ -d "gh-pages/.git" ] || git clone $(GIT_URL) gh-pages

searx/utils/searx.sh

Line 398 in 002d53a

git remote set-url origin ${GIT_URL}
GIT_BRANCH usage: utils/searx.sh

searx/utils/searx.sh

Line 329 in 002d53a

git checkout -B "$GIT_BRANCH"
DOCS_URL is displayed in the themes, referenced in setup.py and utils/morty.sh
PUBLIC_INSTANCES, DOCS_URL, ISSUE_URL are displayed in the themes and referenced in setup.py

When a setting should go into brand.py ? (brand.py but it is actually Makefile )

The text was updated successfully, but these errors were encountered:

return42 · 2020-12-16T10:22:07Z

What is the difference between band.py and settings.yml

Yes, this needs some discussion .. but I would use a different title simmilar:

What searx option is a part of the brand and what is a part of instance settings.yml ..

.. and and what are the intersections.

dalf · 2020-12-16T10:50:50Z

Let's start with SEARX_URL:

in https://searx.github.io/searx/admin/settings.html#global-settings , when the admin modifies base_url, should we ask to update SEARX_URL too ?
or should we remove base_url from settings.yml ? If yes, why not the other settings in the server section ?
- Note: utils/morty.sh and utils/searx.sh rely on this value. So should searx.sh creates a settings.yml according to SEARX_URL ? In this case SEARX_URL is a setup parameter for the installation scripts.
or should we remove SEARX_URL from brand ?

return42 · 2020-12-18T23:40:26Z

I think the brand is needed for the buildenv while settings are needed for a instance.

Values like SEARX_URL and CONTACT_URL are needed at build-time and also are needed at instance's run-time.

One brand might maintain one or more instances.

I think we need some hierarchy: if a value is not given in the settings.yml file, it is taken from the brand.py

return42 · 2020-12-19T11:15:38Z

@dalf I created a draft to demonstrate & test what I mean : #2404

dalf · 2020-12-19T20:53:51Z

In the current master branch:

# git grep "CONTACT_URL"

# set value
Makefile:# export CONTACT_URL=mailto:contact@example.com

# boilerplate
Makefile:       @echo  '  CONTACT_URL = $(CONTACT_URL)'
Makefile:       $(Q)echo "CONTACT_URL = '$(CONTACT_URL)'" >> searx/brand.py
Makefile:       $(Q)echo "export CONTACT_URL='$(CONTACT_URL)'" >> utils/brand.env

# output
searx/brand.py:CONTACT_URL = 'mailto:contact@example.com'

# real usage: templates
searx/templates/oscar/base.html:                    <a href="{{ brand.PUBLIC_INSTANCES }}">{{ _('Public instances') }}</a>{% if brand.CONTACT_URL %} |
searx/templates/oscar/base.html:                    <a href="{{ brand.CONTACT_URL }}">{{ _('Contact instance maintainer') }}</a>{% endif %}
searx/templates/simple/base.html:        <a href="{{ brand.PUBLIC_INSTANCES }}">{{ _('Public instances') }}</a>{% if brand.CONTACT_URL %} |
searx/templates/simple/base.html:        <a href="{{ brand.CONTACT_URL }}">{{ _('Contact instance maintainer') }}</a>{% endif %}

# git grep "SEARX_URL"

# set value
Makefile:export SEARX_URL=https://searx.me

# boilerplate
.config.sh:PUBLIC_URL="${SEARX_URL}"
Makefile:       @echo  '  SEARX_URL = $(SEARX_URL)'
Makefile:       $(Q)echo "SEARX_URL = '$(SEARX_URL)'" >> searx/brand.py
Makefile:       $(Q)echo "export SEARX_URL='$(SEARX_URL)'" >> utils/brand.env
docs/conf.py:from searx.brand import SEARX_URL
manage.sh:    SEARX_URL:        ${SEARX_URL}

# output
searx/brand.py:SEARX_URL = 'https://searx.me'
utils/brand.env:export SEARX_URL='https://searx.me'

# false positive: documentation
docs/dev/makefile.rst::SEARX_URL:  Changes this, to point to your searx instance.

# doc really uses SEARX_URL
docs/conf.py:extlinks['search'] = (SEARX_URL + '/%s', '#')

# usage
utils/filtron.sh:    SEARX_URL_PATH=$("${REPO_ROOT}/utils/searx.sh" --getenv SEARX_URL_PATH)
utils/morty.sh:SEARX_URL="${PUBLIC_URL:-http://$(uname -n)/searx}"
utils/morty.sh:PUBLIC_URL_MORTY="${PUBLIC_URL_MORTY:-$(echo "$SEARX_URL" |  sed -e's,^\(.*://[^/]*\).*,\1,g')${PUBLIC_URL_PATH_MORTY}}"
utils/morty.sh:    SEARX_URL_PATH=$("${REPO_ROOT}/utils/searx.sh" --getenv SEARX_URL_PATH)
utils/searx.sh:SEARX_URL_PATH="${SEARX_URL_PATH:-$(echo "${PUBLIC_URL}" \
utils/searx.sh:[[ "${SEARX_URL_PATH}" == "${PUBLIC_URL}" ]] && SEARX_URL_PATH=/
utils/searx.sh:  SEARX_URL_PATH      : ${SEARX_URL_PATH}
utils/templates/etc/httpd/sites-available/searx.conf:uwsgi:# SetEnvIf Request_URI "${SEARX_URL_PATH}" dontlog
utils/templates/etc/httpd/sites-available/searx.conf:uwsgi:<Location ${SEARX_URL_PATH}>
utils/templates/etc/nginx/default.apps-available/searx.conf:filtron:location ${SEARX_URL_PATH} {
utils/templates/etc/nginx/default.apps-available/searx.conf:filtron:    proxy_set_header   X-Script-Name    ${SEARX_URL_PATH};
utils/templates/etc/nginx/default.apps-available/searx.conf:filtron:location ${SEARX_URL_PATH}/static {

So SEARX_URL has two usages (and according extlinks documentation ):

installation scripts
documentation

About the documentation:

# grep "search:"
docs/user/search_syntax.rst:modifiers are chainable and inclusive (e.g. with :search:`!it !ddg !wp qwer
docs/user/search_syntax.rst:See the :search:`/preferences page <preferences>` for the list of engines,
docs/user/search_syntax.rst:- :search:`!wp qwer <?q=%21wp%20qwer>` or
docs/user/search_syntax.rst:- :search:`!wikipedia qwer :search:<?q=%21wikipedia%20qwer>`
docs/user/search_syntax.rst:Image search:
docs/user/search_syntax.rst:- :search:`!images Cthulhu <?q=%21images%20Cthulhu>`
docs/user/search_syntax.rst:- :search:`:hu !wp hackerspace <?q=%3Ahu%20%21wp%20hackerspace>`

Sum of the sum up, SEARX_URL has two usages:

installation scripts
this page https://searx.github.io/searx/user/search_syntax.html and actually the links do not work, since searx.me is no more an instance, and there is no official instance.

My feeling is that:

the runtime configuration is settings.yml
the installation configuration is inside Makefile

return42 · 2020-12-19T22:03:21Z

My feeling is that:

the runtime configuration is settings.yml

the installation configuration is inside Makefile

Thanks for your analysis, your conclusion seems (more or less) the same than mine:

Values like SEARX_URL and CONTACT_URL are needed at build-time and also are needed at instance's run-time.

I think there are values needed at installation (build) time and also needed at runtime, e.g. CONTACT_URL.

In my draft #2404 I used inheritance to handle both usages of CONTACT_URL

https://github.com/return42/searx/blob/e2d40cc6eacdc68d437fb0e53627fbb6f329bd55/searx/settings_loader.py#L145-L152

In commit e2d40cc CONTACT_URL is used in the project links of the docs.

What do you think, is the draft a way we can go, or I am wrong?

and actually the links do not work, since searx.me is no more an instance, and there is no official instance.

Yes, this is a problem of the searx brand but not if someone maintain its own brand.

dalf · 2020-12-20T10:14:06Z

In my draft #2404 I used inheritance to handle both usages of CONTACT_URL
https://github.com/return42/searx/blob/e2d40cc6eacdc68d437fb0e53627fbb6f329bd55/searx/settings_loader.py#L145-L152

Meh, it starts to be complicated, three different locations for one setting.

Yes, this is a problem of the searx brand but not if someone maintain its own brand.

What is branding ? What is the process to brand a new version of searx ?

Here different searx forks / packages:

my-spot:
- source code: https://gitlab.e.foundation/e/cloud/my-spot
- one unique instance: https://spot.ecloud.global/
- here CONTACT_URL can be set to a value, but actually the documentation is neither modified and nor published.
- if the my-spot maintenainers want to display a contact information, they have to:
  * modify Makefile
  * make buildenv
  * commit
debian package:
- source code https://salsa.debian.org/debian/searx
- no "official" instance.
- the documentation is not modified, not published.
- here, how to modify the contact URL after apt-get install searx ?
nix-os package :
- description of the package: https://github.com/NixOS/nixpkgs/blob/nixos-20.09/pkgs/servers/web-apps/searx/default.nix#L39
- same question: how to modify the contact URL ?
docker image:
- to modify CONTACT_URL is to mount a volume to replace searx/brand.py
- (yes the admin can fetch the source code and build it, but it ruins the purpose of an official image for some settings).

Should these fork be changed in a way or another ?

I'm not saying branding should be removed, rather the process to brand a searx should be more documented and perhaps adjusted.

return42 · 2020-12-20T14:39:50Z

Should these fork be changed in a way or another ?

Package manger should ask for values which are individually or have some documentation.

A brand concept (however it looks like at the end) should help to maintain forks. Forks are created for package managers and for instances or (docker) images. To maintain a fork, a minimal build process will be needed,

I'm not saying branding should be removed, rather the process to brand a searx should be more documented and perhaps adjusted.

~~I vote to give up the existing brand concept for its first and~~ place all needed values in settings.yml. This was not possible in the past, but since there is a settings_loader I can imagine a implementation.

I can implement a draft ~~on top of #2386.~~

return42 · 2020-12-21T00:00:22Z

I send another draft #2408

The implementation moves all options to settings.yml, with there is no longer any need to touch the Makefile.

dalf · 2020-12-22T08:54:57Z

About GIT_URL

$ git grep "GIT_URL"
# Conf
Makefile:export GIT_URL=https://github.com/searx/searx

# output
searx/brand.py:GIT_URL = 'https://github.com/searx/searx'
utils/brand.env:export GIT_URL='https://github.com/searx/searx'

# Docker
# Purpose: documentation, descripbe the searx version
.github/workflows/integration.yml:        run: make -e GIT_URL=$(git remote get-url origin) docker.push
Dockerfile:ARG GIT_URL=unknown
Dockerfile:LABEL maintainer="searx <${GIT_URL}>" \
manage.sh:    GITHUB_USER=$(echo "${GIT_URL}" | sed 's/.*github\.com\/\([^\/]*\).*/\1/')
manage.sh:         --build-arg GIT_URL="${GIT_URL}" \
manage.sh:         --build-arg LABEL_VCS_URL="${GIT_URL}" \
manage.sh:    GIT_URL:          ${GIT_URL}

# API
# Purpose: documentation for searx-stats2
searx/webapp.py:            'GIT_URL': brand.GIT_URL,

# Installation scripts
# Purpose: which git repository to use to install searx. The value can be be completly different from the current git repository.
utils/searx.sh:  :searx-src:  clone $GIT_URL
utils/searx.sh:git remote set-url origin ${GIT_URL}

# Installation scripts: setup.py
# Purpose: documentation for pypi
setup.py:        "Code": brand.GIT_URL,

# Documentation
# Purpose: documentation
docs/conf.py:from searx.brand import GIT_URL
docs/conf.py:extlinks['origin'] = (GIT_URL + '/blob/' + GIT_BRANCH + '/%s', 'git://')
docs/conf.py:extlinks['patch'] = (GIT_URL + '/commit/%s', '#')
docs/conf.py:        ProjectLink("Source", GIT_URL),
docs/blog/lxcdev-202006.rst:The searx software is a clone of the ``GIT_URL`` (see :ref:`makefile setup`) and

# Usage at build time, for documentation
# Purpose: documentation
searx/static/themes/oscar/gruntfile.js:        banner: '/*! oscar/searx.min.js | <%= grunt.template.today("dd-mm-yyyy") %> | <%= process.env.GIT_URL %>  */\n'
searx/static/themes/simple/gruntfile.js:        banner: '/*! simple/searx.min.js | <%= grunt.template.today("dd-mm-yyyy") %> | <%= process.env.GIT_URL %> */\n',
searx/static/themes/simple/gruntfile.js:          banner: '/*! searx | <%= grunt.template.today("dd-mm-yyyy") %> | <%= process.env.GIT_URL %> */\n'
searx/static/themes/simple/gruntfile.js:          banner: '/*! searx | <%= grunt.template.today("dd-mm-yyyy") %> | <%= process.env.GIT_URL %> */\n'
searx/templates/__common__/about.html:    and <a href="{{ brand.GIT_URL }}">{{ brand.GIT_URL }}</a>
searx/templates/oscar/base.html:                    <a href="{{ brand.GIT_URL }}">{{ _('Source code') }}</a> |
searx/templates/simple/base.html:        <a href="{{ brand.GIT_URL }}">{{ _('Source code') }}</a> |

# Publish documentation
# Purpose: where to publish the documentation
utils/makefile.sphinx:  [ -d "gh-pages/.git" ] || git clone $(GIT_URL) gh-pages

# Boilerplate
Makefile:       @echo  '  GIT_URL   = $(GIT_URL)'
Makefile:       $(Q)echo "GIT_URL = '$(GIT_URL)'"  > searx/brand.py
Makefile:       $(Q)echo "export GIT_URL='$(GIT_URL)'"  > utils/brand.env

# False positive
docs/dev/makefile.rst::GIT_URL:    Changes this, to point to your searx fork.

Purpose: documentation

When GIT_URL is different from git remote get-url origin ?

IMO, this additional value tries to answer this question: which searx version are we talking about ?

0.18.0 is not enough
0.18.0-28-59217bb5 is better but we don't know the source
the GIT_URL allows to get the branding.

Purpose: Where to publish the documentation

There are two different ways to deploy the documentation:

searx/.github/workflows/integration.yml

Lines 77 to 84 in 59217bb

    
           - name: Deploy 
        
             if:  github.ref == 'refs/heads/master' 
        
             uses: JamesIves/github-pages-deploy-action@3.7.1 
        
             with: 
        
               GITHUB_TOKEN: ${{ github.token }} 
        
               BRANCH: gh-pages 
        
               FOLDER: gh-pages 
        
               CLEAN: true # Automatically remove deleted files from the deploy branch

searx/utils/makefile.sphinx

Lines 178 to 188 in 59217bb

    
           $(GH_PAGES):: 
        
           	$(MAKE) docs 
        
           	[ -d "gh-pages/.git" ] || git clone $(GIT_URL) gh-pages 
        
           	-cd $(GH_PAGES); git checkout gh-pages >/dev/null 
        
           	-cd $(GH_PAGES); git pull 
        
           	-cd $(GH_PAGES); ls -A | grep -v '.git$$' | xargs rm -rf 
        
           	$(MAKE) prepare-gh-pages 
        
           	cd $(GH_PAGES);\ 
        
           		git add --all . ;\ 
        
           		git commit -m "gh-pages: updated" ;\ 
        
           		git push origin gh-pages

In the second case, when GIT_URL is different from git remote get-url origin ?

Purpose: Which git repository to use to install searx. The value can be completly different from the current git repository.

When GIT_URL is different from git remote get-url origin ?

return42 · 2021-01-02T11:12:31Z

@dalf: here are my 5cent about GIT_URL:

Purpose: Which git repository to use to install searx. The value can be completly different from the current git repository.

Is it? .. If someone wants to install a instance from its fork (like I do it), he will clone from this fork and the 'origin' is set to fork's GIT_URL .. if I didn't missed anything, this should be true in common use cases

Vice-Versus: This rule will no match in use cases where the remote alias 'origin' does not point to GIT_URL, the URL of fork's repository. Should we cover (support) such naming issue of remote alias also? .. I don't think so.

There are two different ways to deploy the documentation:

I don't see any contradiction in this.

searx/utils/makefile.sphinx implement a default deployment method which can be used by searx forks
searx/.github/workflows/integration.yml is the deployment method which is used by by the searx project itself.

When GIT_URL is different from git remote get-url origin ?

ATM, I do not know any use case, where it is different. After #2408 has been merged I can try to replace GIT_URL by $(git remote get-url origin) in https://github.com/return42/searx/blob/rm-brand-make/utils/brand.env#L2 (and at other places as well).

May be we can also replace other environment (settings). But this will need more investigation from my side .. first, it might be good to merge #2408 to have one single point of definition.

return42 · 2021-03-03T13:26:12Z

@dalf what do you think, can we close this issue?

dalf added the question label Dec 16, 2020

dalf changed the title ~~What is the difference between band.py and settings.yml ?~~ What searx option is a part of the brand and what is a part of instance settings.yml ? Dec 16, 2020

dalf mentioned this issue Dec 18, 2020

Add link to contact instance maintainer to footer of each page #2391

Merged

return42 mentioned this issue Dec 19, 2020

inherit base_url and contact_url from brand #2404

Closed

return42 mentioned this issue Dec 20, 2020

[mod] move brand options from Makefile to settings.yml #2408

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

What searx option is a part of the brand and what is a part of instance settings.yml ? #2394

What searx option is a part of the brand and what is a part of instance settings.yml ? #2394

dalf commented Dec 16, 2020

return42 commented Dec 16, 2020

dalf commented Dec 16, 2020

return42 commented Dec 18, 2020

return42 commented Dec 19, 2020

dalf commented Dec 19, 2020

return42 commented Dec 19, 2020

dalf commented Dec 20, 2020 •

edited

Loading

return42 commented Dec 20, 2020 •

edited

Loading

return42 commented Dec 21, 2020

dalf commented Dec 22, 2020

return42 commented Jan 2, 2021

return42 commented Mar 3, 2021

What searx option is a part of the brand and what is a part of instance settings.yml ? #2394

What searx option is a part of the brand and what is a part of instance settings.yml ? #2394

Comments

dalf commented Dec 16, 2020

return42 commented Dec 16, 2020

dalf commented Dec 16, 2020

return42 commented Dec 18, 2020

return42 commented Dec 19, 2020

dalf commented Dec 19, 2020

return42 commented Dec 19, 2020

dalf commented Dec 20, 2020 • edited Loading

return42 commented Dec 20, 2020 • edited Loading

return42 commented Dec 21, 2020

dalf commented Dec 22, 2020

About GIT_URL

Purpose: documentation

Purpose: Where to publish the documentation

Purpose: Which git repository to use to install searx. The value can be completly different from the current git repository.

return42 commented Jan 2, 2021

return42 commented Mar 3, 2021

dalf commented Dec 20, 2020 •

edited

Loading

return42 commented Dec 20, 2020 •

edited

Loading