Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data deletion during 1.15.0 update #2619

Closed
RemiCab opened this issue Nov 1, 2019 · 16 comments
Closed

Data deletion during 1.15.0 update #2619

RemiCab opened this issue Nov 1, 2019 · 16 comments
Milestone

Comments

@RemiCab
Copy link

RemiCab commented Nov 1, 2019

Hello,

After receiving the rss post about the 1.15.0 update, I decided to do it (using the UI). After clicking on the "update" button, I was redirected to the 403 error file. According to the nginx logs, I discovered that the users directories (in data/users) and the config.php file had been deleted during the process.

I have restored these files using my backup and regardless this point the update seems to be correct.

Nevertheless, is this deletion a normal part of this particular update? If yes, it seems important to inform administrators of this behavior. Especially because the UI gives the impression of a smooth process and in this case it was not really the case.

If it is a bug, here are the technical details:

  • Raspbian 10
  • Nginx 1.14.2
  • php 7.3.11-1
  • PostgreSQL 11.5
  • update FreshRSS 1.14.3 -> 1.15.0

I can't provide any FreshRSS logs since they were in the deleted directories.

Nginx logs:

2019/11/01 15:18:51 [error] 6309#6309: *9798 FastCGI sent in stderr: "PHP message: PHP Warning:  touch(): Unable to create file /srv/FreshRSS/data/users/***_adm/log.txt because No such file or directory in /srv/FreshRSS/lib/lib_rss.php on line 317" while reading response header from upstream, client: 188.239.152.230, server: rss.xxx.yy request: "GET /i/?c=update&a=apply&post_conf=1 HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.3-fpm.sock:", host: "rss.xxx.yy"
@Frenzie
Copy link
Member

Frenzie commented Nov 1, 2019

I'm tagging this as important because it definitely sounds like a bug. However, nothing of the sort happened to me when I tested updating a few different systems.

Could there potentially have been a lack of diskspace or something of the kind involved?

@RemiCab
Copy link
Author

RemiCab commented Nov 1, 2019

I have checked it and the disk where FreshRSS is located is full at 51% so that is not the issue.

Could this be a git related problem? I mean, as the user data and config files are not tracked by git, is it possible that the update process has removed all to apply the repository state of the data directory?

@Frenzie
Copy link
Member

Frenzie commented Nov 1, 2019

It seems to execute a git clean since #2036 when using Git , which I believe would indeed remove untracked files.

exec('git clean -f -d -f', $output, $return);

@Alkarex
Copy link
Member

Alkarex commented Nov 1, 2019

@Reman38 No, those files are not supposed to be deleted.

Did you install by git?
If yes, could you please verify that you have the following files?

@RemiCab
Copy link
Author

RemiCab commented Nov 1, 2019

Yes, git is installed on the server.

Moreover, I have checked my backups and both of them are missing (hidden files are part of the backup since .htaccess file is present) at least since September 1st which is my oldest backup (at this time the version was already 1.14.3). Thus, the first gitignore file has probably been added by the current update.

For your information I followed the documentation to install FreshRSS and since this I always used the GUI to update without problem until today. I never moved this installation or manually removed filed in the FreshRSS directory.

@marienfressinaud
Copy link
Member

By taking a quick glance at updateController, I would say that we must at least execute a git checkout -- . before executing git clean

@Alkarex
Copy link
Member

Alkarex commented Nov 1, 2019

@marienfressinaud Cleaning is needed before checking out, otherwise if a file exist or has been modified, checking out would fail. See #2036
Furthermore, if the .gitignore is removed, then cleaning after checking out would have the same bad effects.

@Reman38 Any idea of what could have deleted https://github.com/FreshRSS/FreshRSS/blob/dev/data/users/.gitignore in your setup?

@Alkarex
Copy link
Member

Alkarex commented Nov 1, 2019

P.S. @marienfressinaud I did not see your -- . Indeed, good idea

@Alkarex Alkarex added this to the 1.15.1 milestone Nov 1, 2019
@Alkarex
Copy link
Member

Alkarex commented Nov 2, 2019

A fetch + reset seems to be more robust https://stackoverflow.com/a/9589927 and with the additional benefit that we might be able to remove our git clean altogether.
I have just checked that an existing local untracked file is properly overwritten by the fetch + reset.

@marienfressinaud
Copy link
Member

How is data/do-install.txt managed with this solution? We need to be sure it is deleted at the end of the update. Btw, I think we shouldn't have this file anymore (but it's a different issue).

Alkarex added a commit to Alkarex/FreshRSS that referenced this issue Nov 2, 2019
Fix FreshRSS#2619
Avoid potentially dangerous git clean, and use more robust fetch + reset
strategy instead
@Alkarex
Copy link
Member

Alkarex commented Nov 2, 2019

@marienfressinaud #2625

@RemiCab
Copy link
Author

RemiCab commented Nov 2, 2019

@marienfressinaud Cleaning is needed before checking out, otherwise if a file exist or has been modified, checking out would fail. See #2036
Furthermore, if the .gitignore is removed, then cleaning after checking out would have the same bad effects.

@Reman38 Any idea of what could have deleted https://github.com/FreshRSS/FreshRSS/blob/dev/data/users/.gitignore in your setup?

No idea. Since no other processes than Nginx can write into the FreshRSS directory, the only idea that comes into my mind is another strange issue during the previous update that has removed the .gitignore.

@Alkarex
Copy link
Member

Alkarex commented Nov 2, 2019

@Reman38 Could you please show the output of the following commands?

cd ./FreshRSS/
git status -v
git remote -v
git branch -v

Alkarex added a commit that referenced this issue Nov 2, 2019
Fix #2619
Avoid potentially dangerous git clean, and use more robust fetch + reset
strategy instead
@Alkarex Alkarex closed this as completed Nov 2, 2019
@RemiCab
Copy link
Author

RemiCab commented Nov 3, 2019

@Reman38 Could you please show the output of the following commands?

cd ./FreshRSS/
git status -v
git remote -v
git branch -v

Sorry for the delay, I was AFK this week-end.

git status -v

Sur la branche master
Votre branche est à jour avec 'origin/master'.

Modifications qui ne seront pas validées :
  (utilisez "git add/rm <fichier>..." pour mettre à jour ce qui sera validé)
  (utilisez "git checkout -- <fichier>..." pour annuler les modifications dans la copie de travail)

	supprimé :        ../PubSubHubbub/feeds/.gitignore
	supprimé :        ../PubSubHubbub/feeds/README.md
	modifié :         ../PubSubHubbub/feeds/index.html
	modifié :         ../PubSubHubbub/index.html
	supprimé :        ../PubSubHubbub/keys/.gitignore
	supprimé :        ../PubSubHubbub/keys/README.md
	modifié :         ../PubSubHubbub/keys/index.html
	supprimé :        ../cache/.gitignore
	modifié :         ../cache/index.html
	supprimé :        ../do-install.txt
	supprimé :        ../extensions-data/.gitignore
	modifié :         ../extensions-data/index.html
	supprimé :        ../favicons/.gitignore
	modifié :         ../favicons/index.html
	supprimé :        ../fever/.gitignore
	modifié :         ../fever/index.html
	modifié :         ../index.html
	supprimé :        ../tokens/.gitignore
	modifié :         ../tokens/index.html
	supprimé :        .gitignore
	supprimé :        _/.gitignore
	modifié :         _/index.html
	modifié :         index.html

Fichiers non suivis:
  (utilisez "git add <fichier>..." pour inclure dans ce qui sera validé)

	../PubSubHubbub/feeds/aHR0cDovL3N5bmRpY2F0aW9uLnRoZWRhaWx5d3RmLmNvbS9UaGVEYWlseVd0Zg/
	../PubSubHubbub/feeds/aHR0cHM6Ly90aGVoZnRndXkuY29tL2ZlZWQv/
	../PubSubHubbub/feeds/aHR0cHM6Ly93d3cuYmxvZ2dlci5jb20vZmVlZHMvNDMwNDczOTY5NzcxNjE5MTk5OC9wb3N0cy9kZWZhdWx0/
	../PubSubHubbub/feeds/aHR0cHM6Ly9sdXN0dWJsb2cuY29tL2ZlZWQv/
	../PubSubHubbub/keys/2a8ffa0b0a8d8e03fc3e1b706d5bb701973022c4.txt
	../PubSubHubbub/keys/9a9990fd8638cac7ce137028683e5e39e540200c.txt
	../PubSubHubbub/keys/c4248c3a8815d3d9ab36938daea00725bfc42abf.txt
	../PubSubHubbub/keys/d6c9549d2ca32745867414008cb913ad85003397.txt
	../cache/0e331bd88fe7a5bf3fec0b315037b283.spc
	../cache/12a882fb319b82e8894618e4bbfebeb7.spc
	../cache/1b06a01e5716bf5b4bb5ec441e3e7734.spc
	../cache/1f0bb5fd73c13fcba8200c9f4f26caa0.spc
	../cache/1fb3f91fe7bdb7c7ec956137b5aad8b0.spc
	../cache/2780b9b82c15c5a6caf154e328aca6db.spc
	../cache/28c0903117a8ef6bbfcbdac78531cad1.spc
	../cache/3b30fb1833ad6c01895040c669cfcfff.spc
	../cache/3df9110ccf2b1e06cc637e9b6e32fefd.spc
	../cache/4f59e4c7048297f1ef116fa122d5bd60.spc
	../cache/58451a04e801f9433a3a4c872a8dc6de.spc
	../cache/69a4ef607a57dbc84600024a33df4f94.spc
	../cache/6d5d2bde19a40fc144508942f9c4a331.spc
	../cache/80a071d1b09d13a281d12e1e034687f7.spc
	../cache/890c8ef4cc7909eb65c612dbfc8d4fb0.spc
	../cache/8f2bbdc69e07653bdd21bd7bcb602950.spc
	../cache/a301ed7554e0e34872f956088e0ab1e1.spc
	../cache/b611e592a68383c652ce896b46f18824.spc
	../cache/b625d7ffd14c2e75cfa88976ebabddde.spc
	../cache/c5c89cc894cff78b68d587d165ae0e70.spc
	../cache/cca48d7507598d29790135a8b8012f73.spc
	../cache/cd33672a6d724daa5a9cb9a6c18ee62b.spc
	../cache/ce0b3ed0977e1c9320862963a4677c0f.spc
	../cache/d760a7e4788bb026eef972858602da32.spc
	../cache/d8db5e85d077f9fc983a4d62f2ed0aae.spc
	../cache/da591e30ecb03522290d8955b589ff4c.spc
	../cache/decbcfce175c4f60c17a40e11b4ddbf6.spc
	../cache/e5fb78b4e6c8cf898803fbc27f012784.spc
	../cache/f0c83300cdcbaae0b2e1d1d56605c7f5.spc
	../cache/f3459711a74fcc67f0f6916e5dc58257.spc
	../cache/f9135b88a996be6606c01c1da5b6b4d0.spc
	../cache/fc352aff6e397c27577e5b39685a68ce.spc
	../cache/fe000aba7ec40424286f808ac7293784.spc
	../favicons/05d8f488.ico
	../favicons/05d8f488.txt
	../favicons/075f3dfc.ico
	../favicons/075f3dfc.txt
	../favicons/089c2527.ico
	../favicons/089c2527.txt
	../favicons/1171bbfd.ico
	../favicons/1171bbfd.txt
	../favicons/1df94ba2.ico
	../favicons/1df94ba2.txt
	../favicons/2c99fbd3.ico
	../favicons/2c99fbd3.txt
	../favicons/31ecd1f9.ico
	../favicons/31ecd1f9.txt
	../favicons/3245352b.ico
	../favicons/3245352b.txt
	../favicons/4a0acad8.ico
	../favicons/4a0acad8.txt
	../favicons/4c85dc3c.ico
	../favicons/4c85dc3c.txt
	../favicons/574cd674.ico
	../favicons/574cd674.txt
	../favicons/58cf19dd.ico
	../favicons/58cf19dd.txt
	../favicons/604f55b8.ico
	../favicons/604f55b8.txt
	../favicons/6e97f911.ico
	../favicons/6e97f911.txt
	../favicons/6f561ff0.ico
	../favicons/6f561ff0.txt
	../favicons/6fd4beb0.ico
	../favicons/6fd4beb0.txt
	../favicons/8b58398c.ico
	../favicons/8b58398c.txt
	../favicons/8c8e32b6.ico
	../favicons/8c8e32b6.txt
	../favicons/90d50880.ico
	../favicons/90d50880.txt
	../favicons/98879613.ico
	../favicons/98879613.txt
	../favicons/9ad51aa7.ico
	../favicons/9ad51aa7.txt
	../favicons/ad93cfd5.ico
	../favicons/ad93cfd5.txt
	../favicons/be91c64f.ico
	../favicons/be91c64f.txt
	../favicons/c12190f7.ico
	../favicons/c12190f7.txt
	../favicons/cb0bd10c.ico
	../favicons/cb0bd10c.txt
	../favicons/cb8014f2.ico
	../favicons/cb8014f2.txt
	../favicons/d1376fca.ico
	../favicons/d1376fca.txt
	../favicons/d4106aba.ico
	../favicons/d4106aba.txt
	../favicons/e9a07854.ico
	../favicons/e9a07854.txt
	../favicons/ea800b4b.ico
	../favicons/ea800b4b.txt
	../favicons/ec8f6c3a.ico
	../favicons/ec8f6c3a.txt
	../favicons/faa0a16c.ico
	../favicons/faa0a16c.txt
	../tokens/6185e7a2535a6eb8e3d8b6187377cbf5ccb23132.txt
	_/log.txt
	_/log_api.txt
	_/log_pshb.txt
	user1/
	user2/

aucune modification n'a été ajoutée à la validation (utilisez "git add" ou "git commit -a")

git remote -v:

origin	https://github.com/FreshRSS/FreshRSS.git (fetch)
origin	https://github.com/FreshRSS/FreshRSS.git (push)

git branch -v:

* master 3aa66f31 Merge pull request #2599 from FreshRSS/dev

@Alkarex
Copy link
Member

Alkarex commented Nov 3, 2019

@Reman38 Ok, it looks like several files have been modified locally. In particular a bunch of .gitignore files, which have been deleted.

@RemiCab
Copy link
Author

RemiCab commented Nov 8, 2019

After resetting the .gitignore deletions, I have updated FreshRSS to 1.15.1 without problems. Thank you for your help!

javerous pushed a commit to javerous/FreshRSS that referenced this issue Jan 20, 2020
Fix FreshRSS#2619
Avoid potentially dangerous git clean, and use more robust fetch + reset
strategy instead
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants