Allow installing an instance of the producers platform through Docker

[stephanegigandet]

In production, we have 2 instances of Product Opener running:

• 1 for the public database https://world.openfoodfacts.org
• 1 for the producers platform https://world.pro.openfoodfacts.org

The code is the same, but the producers platform has a few differences:

• there is a different root path (in production: /srv/off-pro/ instead of /srv/off)
• in Config2.pm:

server constants

$server_domain = "pro.openfoodfacts.org";

server paths

$www_root = "/srv/off-pro/html";
$data_root = "/srv/off-pro";
$mongodb = "off-pro";

[..]

%server_options = (

    cookie_domain => "openfoodfacts.org",   # if not set, default to $server_domain
    private_products => 1,  # Make products visible only to the owner
    producers_platform => 1,        # Enable special features for producers
    minion_backend => {'Pg' => 'postgresql://off:Zo4pByAt2tt@localhost:5432/minion'},
    export_servers => { public => "off" },
    minion_local_queue => "pro.openfoodfacts.org",
    minion_export_queue => "openfoodfacts.org",
    export_data_root => "/srv/off",
    export_www_path => "/srv/off/html",

There are a few differences in directory structure:

the products and html/products directories have one extra level for the owner (e.g. org-nestle-france) of the products

In addition, there are also 2 minion server queues that are running, one for the producers platform (e.g. to import Excel file from producers, and to export data to the public platform), and one for the public platform (to import data from the producers platform).

The minion servers are run with systemctl:

root@off1:/srv/off-pro/products# cat /etc/systemd/system/minion-off-pro.service
[Unit]
Description=off-pro minion workers
After=postgresql.service

[Service]
Type=simple
#ExecStart=/srv/off-pro/scripts/minion.off-pro.sh
User=off
WorkingDirectory=/srv/off-pro/scripts
Environment="PERL5LIB=."
ExecStart=/srv/off-pro/scripts/minion_producers.pl minion worker -m production -q pro.openfoodfacts.org
KillMode=process

[Install]
WantedBy=multi-user.target

root@off1:/srv/off-pro/products#
root@off1:/srv/off-pro/products#
root@off1:/srv/off-pro/products# cat /etc/systemd/system/minion-off.service
[Unit]
Description=off minion workers
After=postgresql.service

[Service]
Type=simple
#ExecStart=/srv/off-pro/scripts/minion.off-pro.sh
User=off
WorkingDirectory=/srv/off/scripts
Environment="PERL5LIB=."
ExecStart=/srv/off/scripts/minion_producers.pl minion worker -m production -q openfoodfacts.org
KillMode=process

[Install]
WantedBy=multi-user.target

One issue may be memory: each instance of Product Opener takes a lot of memory, and the Minion processes also load the taxonomies.

[hangy]

The minion servers are run with systemctl:

You probably don't want a systemd service in docker. Start it as a separate Docker container - with the same image but a different entrypoint.

[ocervell]

Some updates here, here is what we need to deploy the producers platform:

• A new Proxmox container for the producers platform off-pro-net
• A new GitHub environment off-pro-net with the same values as off-net, except the following variables:
  • HOST
  • USERNAME
  • SSH_PRIVATE_KEY
  • PROXY_HOST
  • PRODUCERS_PLATFORM=1
• Update deployment strategy.matrix for container-deploy.yml to add off-pro-net and off-pro-org environments to the deployment
• Add a step to container-deploy.yml to run the minions

[stephanegigandet]

@alexgarel has added some support to run the producers platform in docker:

https://github.com/openfoodfacts/openfoodfacts-server/blob/main/docs/introduction/dev-environment-quick-start-guide.md#3-optional-review-product-openers-environment

[alexgarel]

Dev is working at the moment. We need staging environment deployability, but we would do it later this year.