Split configure.pl into installation and configuration scripts

[moisseev]

configure.pl is highly interactive and makes some modifications in source and configuraton files. For this reason porting of the BackupPC is complicated and nearly impossible to create "binary" packages without major changes in the source code.

To simplify packaging I propose to split configure.pl into two different scripts:

1. Installation script
   that will make all required modifications in source files, configure distribution specific parameters like directories locations and install files.
2. Configuration (and upgrade) script
   that will make modifications in configuration files only.

Il 16/05/2016 20:37, Alexander Moisseev ha scritto:

As FreeBSD sysutils/backuppc port maintainer I can say one of the first thing that should be done is to split up configure.pl in two parts: installation script and configuration/update script.
It's practically impossible to create pre-built package with monolithic configure.pl script, and I didn't find a way to do that. Finally I managed to solve that problem applying awful patch to configure.pl that removed installation part from it.

On 16.05.16 22:40, Mauro Condarelli wrote:

Sounds very reasonable.
Can You create an issue on github, please?
We need to capture all these issues even if we aren't in position to respond to them at first release.
Would You be interested in working to provide a patch?
Should we ask for debian maintainer input and/or support?
This could be handled together with production of a Docker Container.

I'm definitely interested in working to provide a patch. Otherwise I'll have to maintain that patch in the port anyway.
I believe we should invite debian backuppc package maintainer Ludovic Drolez not only to discussion on this particular issue, but to the backuppc organization. His contacts are available at the debian package tracker.
I'm not familiar with Docker and do not plan to use or study it in the nearest future. It there any specific requirements?

Just to illustrate what the FreeBSD sysutils/backuppc port is doing for now. This allows to create bre-built package and keep ability to automatically update configuration files on version update.

At staging (installation) phase it invokes configure.pl in --batch mode:

106     do-install:
107     cd ${WRKSRC} && ${PERL} configure.pl \
108     --batch \
109     --backuppc-user ${USERS} \
110     --bin-path perl=${PERL} \
111     --config-dir ${ETCDIR} \
112     --cgi-dir ${CGIDIR} \
113     --data-dir /var/db/BackupPC \
114     --dest-dir ${STAGEDIR} \
115     --fhs \
116     --html-dir ${WWWDIR} \
117     --html-dir-url /${PORTNAME} \
118     --install-dir ${PREFIX} \
119     --log-dir /var/log/BackupPC \
120     --no-set-perms \
121     --uid-ignore

After installation user should run update.sh script that just passes options to update.pl script:

1   #!/bin/sh
2   
3   perl %%PREFIX%%/libexec/backuppc/update.pl \
4   --bin-path perl=%%PREFIX%%/bin/perl \
5   --config-dir %%ETCDIR%% \
6   --cgi-dir %%CGIDIR%% \
7   --data-dir /var/db/BackupPC \
8   --fhs \
9   --html-dir %%WWWDIR%% \
10  --html-dir-url /backuppc \
11  --install-dir %%PREFIX%% \
12  --log-dir /var/log/BackupPC

update.pl is configure.pl script with stripped out installation directives:
https://svnweb.freebsd.org/ports/head/sysutils/backuppc/files/patch-update.pl?revision=377862&view=markup

[mcondarelli]

I contacted Ludovic Drolez. Let's hope he will be available.

As far as Docker is concerned the requirements are:

• a Docker Container is a stripped-down Virtual Machine, we need to decide it's underlying OS and Distribution (please avoid starting Holy Wars!!! ;) )
• a Docker Container ideally should run just one program, so normal setup would be to have two Containers, one for BackupPC and one for the Web Server.
• a Docker Container has no local persistent storage, so whatever needs to hold data (pool, config, etc.) needs to be in a well defined and separate (set of) directory (-ies) to be mounted as Dcker "Data volumes".

The latter is the only thing that has any impact on installation/configuration.
Other requirement, if we go with the "two container" configuration (recommended!) is anything accessed by the Web Server must be exported from the Container (and thus it would be better to have it grouped into a single place).

Following from ML:
on 17/05/2016 11:29, Juergen Harms wrote:

I agree that this is a very important issue. Can you also take the
aspect on board to facilitate the users work when he adapts his
customization to a new release? I have always found it extremely painful
to wade through more than 2000 lines of config.pl to find the 3 lines
that need to be considered for updating.

That is another, possibly huge, chapter.
Current setup (and especially installation setup, but not only) is not extremely user-friendly (or, as Linus once said about Linux: "it's very choosy about its friends").
It is OK (kind of) for sysadmin-type users with a good Perl background, but for the average user it's "a bit uphill".
I fully appreciate the intrinsic power of exposing the inner structure, but that is overwhelming for the trivial tasks.
Produce canned sequences for the most used setups should be fairly high priority.
What I mean is we should have a set of scripts (to be morphed into a real GUI later) covering the most frequent use-cases without requiring Perl source modification. I think of a script taking a few parameters and doing the appropriate changes into config.pl, much in the same spirit as configure.pl, but broken down into smaller pieces that can be reused after initial configuration.

This could be architected together with splitting work.
First step would be definition of use-cases we want to cover.
What needs to be covered beyond installation and configuration?

In general I have the feeling all user interface should be rethought from a different vantage point: what needs to be done instead of how to do it, but that is a very long term item

What do You think?

[ldrolez]

Or maybe add two options to configure.pl like --update-conf-only and --without-update-conf, to be able to skip the config phase?

[moisseev]

I agree, --update-conf-only is a good solution. On one hand separate configuration script is safer for a package than --update-conf-only option, but on the other hand there is a lot of options to pass to configure.pl, so we have to provide a wrapper script anyway.

I think --without-update-conf option is unnecessary. We should provide a file with default configuration for a new installation or as a reference for user anyway.
I am not aware of Debian package installation process, but on FreeBSD port/package building occurs not in place, but in stage directory. After executing configure.pl we'll get config.pl with default configuration in stage directory and then install it as config.pl.sample.

[moisseev]

I want to propose to distribute BackupPC as source code instead of an installer.
There is nothing new. Actually it seems to me a more simple and conventional way.

1. Replace configure.pl and makeDist with Makefile (or Makefile.PL) that should combine functionality of makeDist and installation part of current configure.pl .

The Makefile also should create and install the configure.pl (just configuration script).
The default options for configure.pl should be determined during build using Makefile command line options and/or user input (in interactive mode).

Since there is no distribution tarball, a new release is just a git tag and a constant (or variable) with version number (or might be 3 constants: major, minor, patch) in the Makefile. So users could install any backuppc version (release, HEAD or any commit) in the same way:

• clone or download source
• run Makefile to build and install
• run configure.pl to configure new installation or upgrade

2. Do we need the version number and release date in every file header? And change every file on each release? In my opinion it would be better to keep unmodified files untouched between releases.

3. Change tag format. I propose to use just dotted version number instead of underscore separated with preceding v: v4_1_0 -> 4.1.0 That is not important, but it can simplify things a little if a tag exactly matches a version number, tarball suffix, directory suffix, etc.
   The same is related to the backuppc-xs and rsync-bpc. Done.

[adferrand]

I would like to share with you some insights about configuration in the point of view of Docker image building, like @mcondarelli.

Recently I published a new Docker image adferrand/backuppc because:

• no Docker image was supporting the 4.x branch of BackupPC,
• none of them was dealing with the problem of upgrade, and that is the point here.

Basically, the use of a monolithic script for both installation (which is a compile phase), and upgrade (which is a runtime phase), can be problematic. Indeed, Docker clearly separates the compile time (building of an image) and runtime (instanciation of this image), and then it is a good study case.

To build a BackupPC Docker image, one can install BackupPC from the available package of the underlying distribution during the image building. Great, but what about the instantiation of a new version of the container ? As BackupPC has been installed during image building, configure.pl is never called on the existing BackupPC configuration/pool. Moreover configure.pl is not available any more except if you redownload the sources and call manually configure.pl.

For my case, I solved the problem by deferring the BackupPC installation on the first instantiation of the container in its entrypoint. Consequently configure.pl is always called, and :

• either it is a fresh install, no existing BackupPC configuration is detected and it will be installed from scratch,
• either a BackupPC configuration exists, configure.pl detects it and apply all upgrading logic on this config (in particular, activate the V3 Pool retrocompatibility if the upgrade comes from a v3.x version of BackupPC).

But this approach should be replaced by the more elegant solution proposed by @moisseev in its first post. The configure.pl script should only care about binary installation, and then would be invoked during image building phase (or package building phase). The update.pl script should be available in the BackupPC installation folder, ready to be called at any time : on container entrypoint for a Docker image, or as a post-installation script for a package.

And in fact, we have already this kind of runtime upgrade process for another matter, the pool : BackupPC_migrateV3toV4.

However, I disagree with the --update-conf-only solution, because it forces, as noticed after, to have the source code available during upgrade process. I think it is not acceptable to rely on sources to operate on an installed software.

As a side note, I have also some doubts about the two-containers approach of @mcondarelli : the admin web ui, which is a part of BackupPC, is a set of CGI scripts so it needs a web executor. So a web executor is mandatory requirement for BackupPC. Docker is not always "one process by container", but more "one purpose by container".

I think it is perfectly acceptable to execute an init system like supervisord to start few processes which servers one common purpose in a Docker container. In my case, I used supervisord to start BackupPC and lighttpd to expose the web admin ui. What you are doing of this web admin UI is indeed outside of the BackupPC container purpose, and should be delegated to other containers (like proxying, HTTPs on so on).

[moisseev]

However, I disagree with the --update-conf-only solution, because it forces, as noticed after, to have the source code available during upgrade process. I think it is not acceptable to rely on sources to operate on an installed software.

You don't need sources anymore. configure.pl with --config-only option (6aa5657) can use config.pl.sample and installed Lib.pm, but you have to specify its location as described in
configure.pl --man
It's not an elegant solution, but at least it's packageable.

[adferrand]

Thanks for your reply.

It is indeed a good start, because for the update process, only configure.pl need to be kept in the image.

However, problems still reside in the installation phase, because configure.pl is not stateless : it needs to know about the config dir and the pool dir.

But theses directories should not be accessed during docker image build, because they are part of the docker state : they may be mounted as data volumes, and their content will change between the image build process and the container instantiation.

Is there any way to make configure.pl acts as a pure install script? I mean copy only binaries, cgi, config template, without doing anything for the actual config and actual pool. Then config and pool would be handled at runtime with --config-only option.

[moisseev]

You can use --dest-dir to install BackupPC into a staging directory. If you specify this option any existing installation should be ignored, like it was installed in clean environment. After that you'll need to copy the installed files into real destination.