Move Configuration Backup directory out of critical path for Rockstor RPM updates

[Hooverdan96]

Thanks to forum member @marciopamplona it was discovered that as a side-effect pf the RPM build configuration, the config-backups directory is deleted during a release update, which causes the loss of existing configuration backups, as well as an error message in the WebUI, since that directory is never recreated as part of the update.

For reference:
In the RPM Build repository:
https://github.com/rockstor/rockstor-rpmbuild/blob/81bf62b3bfc14a12168b55a50d4f42aaebaf4213/rockstor.spec#L324-L329

during an update we’re deleting the static inventory, which also contains the config-backups.

The definition of the location for the config-backups happens here:

rockstor-core/src/rockstor/settings.py

Lines 96 to 97 in 4d60a2c

	# Absolute filesystem path where config backups are stored by default
	DEFAULT_CB_DIR = os.path.join(MEDIA_ROOT, 'config-backups')

where MEDIA_ROOT is:

rockstor-core/src/rockstor/settings.py

Line 94 in 4d60a2c

MEDIA_ROOT = os.path.join(BASE_DIR, 'static')

The initial proposal is to move the directory defined to a different location, as the deletion of the static directory is currently required to ensure that updates of the RPM release are sticking after the update. This also seems easier than trying to make the rm command too specific to avoid the deletion of one specific sub-directory out of multiples.

From the thread @FroggyFlox has suggested that a user configurable option could be introduced (or merely documented) and a future option to surface that configuration through the WebUI (but that would be for a later stage).

Finally, for reference here is the forum thread:
https://forum.rockstor.com/t/unknown-internal-error-doing-a-get-to-api-config-backup/8938

[phillxnet]

@Hooverdan96 I've linked to a potential fix/enhancement proposed on the basis of your exposition here, but in rockstor-rpmbuild. See what you think. It could well end up being a simple and robust fix to the core of this issue. And bring some redundancy along the way.

[Hooverdan96]

I reviewed it and left a comment over there.

[FroggyFlox]

Just briefly exploring the feasibility or amount of work needed to make the change to a different path than static for the config backups:
First, simply changing to use a MEDIA_ROOT different than static:

# Absolute filesystem path to the directory that will hold user-uploaded files.
# Example: "/home/media/media.lawrence.com/media/"
MEDIA_ROOT = os.path.join(BASE_DIR, "media")

# Absolute filesystem path where config backups are stored by default
DEFAULT_CB_DIR = os.path.join(MEDIA_ROOT, "config-backups")

Build, and intentionally do NOT create the /opt/rockstor/media/config-backups dir. Upon creating a config backup, the page reloads but no config backup is created and no error is displayed to the user. The logs, however, do show:

[17/Aug/2023 17:00:01] DEBUG [storageadmin.views.config_backup:659] backing up config...
[17/Aug/2023 17:00:01] ERROR [storageadmin.util:45] Exception: [Errno 2] No such file or directory: '/opt/rockstor/media/config-backups'
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 41, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/config_backup.py", line 661, in post
    cbo = backup_config()
  File "/opt/rockstor/src/rockstor/system/config_backup.py", line 66, in backup_config
    os.mkdir(cb_dir)
FileNotFoundError: [Errno 2] No such file or directory: '/opt/rockstor/media/config-backups'

We thus fail to create that folder. We do, however, have code in place intended to do just that in system.config_backup:

    if not os.path.isdir(cb_dir):
        os.mkdir(cb_dir)

This is simply failing because we now have also a parent folder to cb_dir to create: media/.
Let's thus use the makedirs() function from the os lib that can deal with that (unlike os.mkdir()):

    if not os.path.isdir(cb_dir):
        os.makedirs(cb_dir)

Now, we can create a config backup from the webUI and it is listed on the disk:

buildvm155:/opt/rockstor # ls -lah media/config-backups/
total 4.0K
drwxr-xr-x 1 root root  64 Aug 17 17:14 .
drwxr-xr-x 1 root root  28 Aug 17 17:14 ..
-rw-r--r-- 1 root root 933 Aug 17 17:14 backup-2023-08-17-171417.json.gz

Note that this is the easy part... Having Django serve the files in that dir should be a bit more involved.

[FroggyFlox]

Being able to download and re-upload a config backup would require adjusting the following as needed to ensure a consistency in the MEDIA_ROOT (testing only in DEBUG mode so far!!):

• src/rockstor/storageadmin/static/storageadmin/js/templates/cb/cb_table.jst
• etc/nginx/nginx.conf
• src/rockstor/urls.py
• src/rockstor/settings.py

Using the example above, for instance, we would need to have:

• src/rockstor/settings.py

MEDIA_URL = "/media/"
MEDIA_ROOT = os.path.join(BASE_DIR, "media")

• src/rockstor/urls.py

    url(r"^media/(?P<path>.*)$", serve, {"document_root": settings.MEDIA_ROOT}),

(note that this might need adjustments as it seems not recommended)

• etc/nginx/nginx.conf

		location /media  {
			root /opt/rockstor/;
		}

• src/rockstor/storageadmin/static/storageadmin/js/templates/cb/cb_table.jst

                <a href="/media/config-backups/{{this.filename}}" title="Download the config backup to your computer">

In any case, the MEDIA_ROOT and MEDIA_URL need to be defined in settings.py, so that limits the customization that we can offer.

[Hooverdan96]

based on that it seems way easier currently to handle this in the RPMBuild with a move, delete and move back approach. And the way it's integrated into django currently, just providing a different user-defined path does not seem to be trivial (if even possible in this framework setup) as you mentioned on the settings.py

[phillxnet]

@FroggyFlox @Hooverdan96 Thanks for the exploration on this one.
I'm inclined to carry on with my current rpmbuild copy-out-and-back approach for now give, as I'm a little unsure of the ramifications regarding establishing a new /media location. And even if we do go that route we can still modify the copy-out-and-back approach to serve a similar function regarding backing up these files between rpm version transition. Plus we need a fairly quick fix for this while we are still in early testing channel. And we have this same situation existing in our current stable channel - so the rpmbuild patch approach will at least manage to avoid loss of these configuration back-up files for current Stable users upgrading to the stable version that emerges from our current testing channel. Assuming I can get the brunt (or all) of the fix into the incoming scriptlets.

[phillxnet]

@Hooverdan96 & @FroggyFlox
Linking to a recent pull request that approaches, but does not fully address this issue:

rockstor/rockstor-rpmbuild#45

in that it maintains the current location of the config back files but manages their persistence over rpm update.
However this issue's focus is more on relocating these backup files entirely. So leaving this issue as no entirely resolved given we have a persistence mechanism but not a fully functional relocation (just a temporary one); by way of the linked pull request.

Note that in the rpm scriptlet approach (linked PR) I ended up using a more obvious directory name re the function implemented:

/opt/rockstor/config-backups-rpmsave

As that seemed to better fit the mechanism introduced there to address the main user experience down-fall we have as only part of this issue: that of lost config backups and a consequent disparity between db and directory contents.

[phillxnet]

Linking back to the spin-off issue within this repo to address the part approach referenced in the last comment:
Save and Restore config back-up files during rpm update #2660
Created to define the smaller scope addressed and to have a changelog reference here give all changes were in the rockstor-rpmbuild repo.