lets establish a build process for all the build products

[return42]

We should no longer commit build products to the repository. ATM we are committing build products like minified CSS and JS from themes, which is the cause of ongoing merge-conflicts.

Beside themes there are other topics we can generalize in a build phase. I think about the logo, there should be one logo image (e.g. a SVG used as the single source of definition) and derived logos in the themes.

More suggestions are welcome, add your toughs ...

[asciimoo]

I agree with the proposal but it raises new questions:
The most important one is: do we want to build the artifacts or let the administrators do it? Both have pros and cons.

If the build is done by the admins, it greatly increases the number of dependencies required to host searx (nodejs, npm, etc..), probably there are many administrators who would dislike installing nodejs to their servers just to be able to host searx.
And if we want to produce the artifacts, there are multiple questions we have to address:

1. Where should we build resources?
2. How can we distribute the artifacts with a privacy respecting manner?
3. How should we handle the versioning of the products?

[dalf]

In https://github.com/sethmlarson/hstspreload , there is a cron which build a pypi package automatically: https://pypi.org/project/hstspreload/#history

Can we do something similar ?

One idea, but frankly speaking I don't like too much because I'm afraid it will be burden rather than a help:

• a side project called searx-bin can contains the built files, so people can just pip install searx-bin (trigger: a commit in searx repository).
• searx-bin could contains a hash of the source file, so searx checks if the searx-bin are out of sync.

[asciimoo]

In https://github.com/sethmlarson/hstspreload , there is a cron which build a pypi package automatically: https://pypi.org/project/hstspreload/#history

Its not entirely clear to me how could we use this solution in our case. Would it mean that we have to build pypi packages for every PR to be able to test it?

a side project called searx-bin can contains the built files, so people can just pip install searx-bin (trigger: a commit in searx repository).

Yeah, I was thinking about a similar solution. Perhaps instead of a repo, we can use a build server (e.g. searx.me) to create and serve the artifacts - not sure if it would be better.

[return42]

Yeah, I was thinking about a similar solution. Perhaps instead of a repo, we can use a build server (e.g. searx.me) to create and serve the artifacts - not sure if it would be better.

I think same .. one word about searx versions .. We need a kind of releases to build and deploy artefacts ..Can we agree that every merge in the master is a version? At least as long as there is no concept for release management .. or should we better use a daily tag? .. any other suggestions about?

[asciimoo]

Can we agree that every merge in the master is a version?

Yup, this can be a good approach, but we still have to solve the build of the unmerged contributions to be able to run the CI and perform manual tests.

[dalf]

Another idea:

• Have a storage server somewhere ( like minio or anything that can store some files )
• In the CI:
  • hash source files ( = .less files, .js not minified, etc..)
  • build and store in the minio server, the key is the source files hash
• Add a script that:
  • download the binaries according to the source files hash.

But

• as soon the PR will involve change in the front-end files, node is required .... as before.
• it requires trust in this new server. Each bundle can have a cryptographic signature to improve that.
• searx-stats2:
  • can rely on these built files.
  • rebuild everything (since it is a source of trust in searx.space).

For information:

$ git log --pretty=format:"%h - %ar, %an : %s" searx/static | wc -l
282

[return42]

$ git log --pretty=format:"%h - %ar, %an : %s" searx/static | wc -l

In this context I want to mentioning, that 6abebc2 from PR #1938 tries to implement a model for StaticFiles.

i think we have to design and describe an architecture, my suggestion is a part of PR #1938 and I tried to document all the components of resources in the doc-strings (later I will add the doc-strings to the documentation build process).

[return42]

Unfortunately I have no more time if you want to discuss the topic further .. otherwise please close / Thanks!

[dalf]

Idea with pypi package

The searx/static directory exists to be easily server by a reverse proxy: all the static files are in other place, ready to be served.

The external plugin feature deploys the static files into the static directory: the searx/static directory must be writable (searx/static or whatever is configured in settings.yml).

Would it makes sense to deploy the themes in the same way?

In this case:

• a theme could be python package:
  • with a git URL requirements.txt.
  • with an actual version on pypi for the release of searx.
• when searx starts the static files from the themes are sync to searx/static.

Doing that for the oscar theme would show the way how to customize searx (which is different to a hack into the searx/static directory).

But:

• it breaks the whole in one repository spirit.
• git pull must be followed by pip intall -u -r requirements.txt to avoid incompatibility issue.

Idea using worktree

In this approach, themes are in different git repository:

• the themes source code is one directory ( .less files, .js).
• the build files (.min.css, .min.js) are pushed in another branch ( oscar-theme-static )

Some git worktree / git submodule provides an easy way to synchronize everything.

• everything in one repository.
• I'm not sure how to update everything in one git command?