Robustness Big Issue 1: cron shouldn't be allowed to run as root

[pgorod]

This could also be titled "Solve permissions problems once and for all" :-)

After recent tests reliably reproducing permissions degradation problems in a controlled testing environment, and finally finding a proper explanation for them, I venture saying that we are POSSIBLY looking at something that will solve ALL permissions problems, or at least we're SURELY looking at something that will solve a LOT of permissions problems that lots of people experience.

We're talking about 5 or 6 cases of this problem every week on the forums, not counting all the other instances where people don't ask, because they already know how to do it. I know I used to fix permissions before and after every upgrade, and some people run cron scripts to fix them every hour... let's put an end to that, shall we?

You can read about the tests here: #1688

What is permissions degradation in SuiteCRM?

It's when file permissions are set correctly, but at a later moment are changed in a way that causes problems. To cause problems, normally it's because files get harder permissions (either ownership by a more secured user like root, or losing some mode of access like writeable) and then when SuiteCRM needs to read or write those same files, it gets access denied errors. These will break some functionality, probably at an unexpected point, resulting in poor logging/messages to the user and admins.

Why does it happen?

It certainly doesn't happen just "because", for no reason at all. It happens because some process running on the server does the changes; it could be a third party process messing with the files, but that's really strange. Why would any process just randomly mess with another app's files?

It could be (and it is) the SuiteCRM application itself. If it's not the web application, then it would have to be the Scheduler jobs (and it is).

How should Scheduler jobs be configured?

They should run under the same user that is used for the Web Server. For example, on a typical Ubuntu install, this would be www-data. On Bitnami installs it would be bitnami, etc.

Does anybody configure Schedulers to run as root? Why would they?

I did : - )

I didn't know I could edit the crontab of a user that does not allow logins, like www-data (with crontab -e -u www-data). Initially I didn't even consider that there was a separate cron for each user. So I just edited crontab as root and made cron.php always run as root. And the jobs worked, so I just forgot about it... Not good practice, I understand now. But Linux actually induces ignorant admins to do just that.

Also, the fact is that there is nothing in the wiki documentation about which user to use. The advice commonly given on the forums doesn't mention it. 'SuiteCRM for Developers' doesn't mention it. Finally, the line suggested by the installer doesn't mention it.

From asking in the forums in these past weeks, I know other people did it (just one example: https://suitecrm.com/forum/installation-upgrade-help/12155-import-data-from-a-sugarcrm-6-5-20-db-to-suitecrm-7-7-8).

I suspect LOTS of other people did it.

So, how exactly do Schedulers break permissions when running as root?

They write files, and these will get ownership by root. Then other processes like the SuiteCRM app running from the web server won't be able to access them. The most relevant case has to do with caching: just by accessing certain common data structures, vardefs and other elements, the system will generate cached files. The odds of this happening increase during installations, upgrades, and Quick Repair and Rebuilds, when caches are flushed and get rebuilt from scratch, so whatever the cron jobs touch first, before the rest of the web app, will be owned by root.

How to fix it?

For individual installs, configure cron correctly.

For the project, to really increase the real-world stability and robustness of the app, and to remove the bad fame of being an app that breaks inexplicably, a combination of these things:

1. Specify in the wiki installation instructions that the crontab must be configured to run under the same user as the web server, recommending usage of crontab -e -u TheWebServerUserName.

2. Inside the app, make the same message appear in the installer scripts (install/confirmSettings.php, install/ready.php). Tell the user specifically the user name to use.

3. Make the same message appear in the Admin-->Schedulers page (modules/Schedulers/Scheduler.php)

4. Make cron.php check under which user it is running. If running as root, log a FATAL error cron.php is running as root. This will cause serious errors and must be corrected.. Then die();.

5. Have the SuiteCRM installer save both user and group in config.php after install. These might be valuable clues for cron.php to use to compare to its current user.

Other users?

As a final note, strictly speaking, these problems could arise with users different from root, basically with any user that is not the web server user. But I expect root is the culprit 99% of the time.

I've thought about this in the past weeks and it's not easy to get a more robust solution. When running from within cron.php (from CLI) there's no easy way to find the web server user. Maybe solution number 5 above would be a good place to start. Possibly in the future we should be moving to a situation where config.php includes a list of possible users under which SuiteCRM and cron will run. All others would cause the app to exit. This would bring the issue of consistent user configuration under a wiser attention from admins.

But for now, let's start with just checking for root, to keep things simple.

High Priority?

I know everybody thinks 'their' bugs are High Priority. But this on definitely is (and it's not 'my' bug, my system is fixed now and stays fixed!).

This is High Priority on all counts: number of people affected, long-standing bug, critically breaks affected systems, very hard to diagnose, affects SuiteCRM public fame, cripples adoption by new users, consumes forum resources greatly and needlessly, etc.

[shogunpol]

@pgorod - Thank you for you question. However, We use github primarily for reporting of Core SuiteCRM Issues and we feel your question would be more suited to our Community Forum. We will be able to help you with your issue there.

[pgorod]

@shogunpol you have got to be kidding me! I know it's too long, but did you even read it? It's not a question, it is a core SuiteCRM issue, it's a code problem, it's a High Priority bug, and it is actually good news for the project that we have this diagnosis.

It also represents a lot of hours on my behalf studying the problem, debugging and testing, and it surely deserves more than 8 minutes of life on Github.

[horus68]

Now lets make this possible and solve the permissions problems!

[daniel-samson]

A problem with this solution is that it assumes that everyone has the same deployment strategy and configuration.

So for example on windows the permissions may work but the read only flag might be set on the files, which can also cause issues. On some distributions of Linux the user may well actually be root. Perhaps we could check who is the user when an entrypoint (eg index.php) has been accessed. Then when cron.php is called we could compare it with the last user we recorded. If they are different then create a log entry.

An other solution would to use the same checks that are in the installer/upgrade. Checking every file would be too time consuming. So maybe we could just check the www root directory(itself) of the instance or the entrypoint file that is being accessed. To see if the there are any issues. Then add them to the log.

[pgorod]

Thanks for joining the discussion!

About Windows, I'm leaving this aside for the moment. If there are problems there, I'd say they are different.

About Linux, you are right that there might be special cases. I was assuming that using root could be considered a bad-practice in all of these cases, so that we could be justified in denying it every time. But maybe there are cases where it's justified, I don't know, can you think of one?

Other than that, the approach you suggest of recording last user going through index.php is a possibility. It's similar to recording the user at the moment of installation, but more dynamic.

What do you think of the idea of having a list of allowed users in config.php? We would start that list during installation, simply filling it with whatever user was used for install.php. Then admins could add to that if they had any special requirements of using multiple users. This way things would be simple for the simple installations (no work to be done), and versatile for people with more advanced needs.

[gunnicom]

Maybe saving the uid of the www-data user in the config, and then using setuid ( http://php.net/manual/en/function.posix-setuid.php ) could be a solution. You can throw a warning, but at least cron would still work.
Additionally you have to take into consideration, that there are many different configurations including windows, suPHP and more.

[daniel-samson]

@gunnicom I think the setuid is the better solution for cron.php instead of logging. However this will only work on Unix Machines.

@pgorod I think having allowed users is taking it a step too far. As an admin, you should be able to run a database dump and move an instance to an other machine, with minimal configuration. The real issue is mostly an education issue with users who do not necessarily have the experience of correctly deploying suitecrm. That isn't to say that there isn't a problem with permissions. It would be good If SuiteCRM could prevent these issues from occurring.

My concern here is performance. We should try to limit system calls as much as possible. Otherwise it will add additional latency to each request a user makes.

We should not log such issues as fatal. Perhaps it should be a debug/info instead. So that users don't end up with a massive log file.

We need to ensure we have a windows solution as well. There are plenty of users who run SuiteCRM on IIS/Apache on Windows. I don't think we should simply ignore them. I think IIS is covered. As the install informs the user if there are issues. However, checking that the user is ISR or ISR_User could be useful when setting up the cron.php on windows. I am not sure how other servers would work on windows like apache, nginx, lighthttp etc. it would be good if a community member could test that.

This is a perfect opportunity for a community member to come up with a solution.

[pgorod]

The only reason I'm not looking for a Windows solution is because we don't have a Windows problem :-) Not here, on this issue. I would have to look into a case of Windows permissions degradation, try to reproduce it, debug it, understand it. I'm fine with that, I was much more familiar with Windows security than Linux before all of this - but it's not what I have achieved here.

So I think it is more precise to say here we need a solution that doesn't negatively impact Windows - that's for sure. But we're working on a solution to a documented Linux problem.

I think it's fair to try to induce some degree of best-practices, lots of mature programs and systems do it. If running as root is always bad in Linux, then it wouldn't be a stretch to make SuiteCRM a system that refuses to run on such bad practices. But as I said, that's an "if" and I'm not sure to what extent it is true. If anybody can give specific counter-example, please do.

About the logging and "dying", A FATAL error and a "die" at installation or upgrade time is not necessarily a bad thing. You have the admin's attention, he scheduled time, he can quickly Google things up and discover he needs a simple tweak to config.php. Believe me, getting a phone call later to learn that the system crumbled inexplicably at a random time, with nothing consistent in the logs is a much worse experience, and a frequent one we're trying to solve here precisely.

Apart from the setuid solution that seems problematic and too Linux specific, I wouldn't worry about performance or system calls, since the only call would be once per minute on cron.php (an offline, asynchronous process), checking which user it is running under.

About the scenario of dumping the DB and moving elsewhere, isn't writing a user name in config.php (only if you changed it) a "minimal configuration"?

Slightly new proposal

So, after reading your suggestions and thinking about this better, my preferred solution now would be to forget about actions targeting specifically the "root" user and instead just do this:

1. At installation time: record user name of web server in config.php, adding an entry to an array of allowed_php_users.

2. At upgrade time: do the same. This ensures that the array is there for everybody post v.7.7.8 (or whatever it is when we add this code)

3. When cron runs: check if the current user is in that array, if not, log FATAL error ("for cron to run as 'someuser' you need to allow that in config.php!") and die. All the problems this might create would be at installation time, upgrade time, or serious-system-reconfiguration time. The admin would just need to Google a quick solution and fix it, while thanking SuiteCRM for helping him keep things secure and consistent.

Impact:

• I don't see any negative impact on Windows, is there any?
• no negative impact on people who have things configured correctly
• lots of people with things configured incorrectly will be nudged into a better situation
• project's fame improves much faster and permanently than if you try any kind of re-education effort
• one additional, easy, logical step added for migrations and serious system reconfigurations like changing which user your php runs under. Not a very high cost at all. It's as easy as reconfiguring the database credentials if you change them.

What do you think about this revised proposal?

[gunnicom]

I am not so optimistic on this two:

• lots of people with things configured incorrectly will be nudged into a better situation

Or will think "Oh that crap does not work, i switch to something different"

• project's fame improves much faster and permanently than if you try any kind of re-education effort

Or people turn away from the project, because there are other projects that "just work" and they dont have to deal with this.

[pgorod]

@everyone Maybe the words "fatal" and "die" make the proposal seem menacing, but remember that's just the cron jobs; not the main app. Some people go without cron jobs for months without even noticing it; it depends on your use of indexes, notifications, workflows, and a few other things.

@gunnicom I don't think people drop projects because something breaks when they configure it wrong. Remember my proposal includes updating the wiki, and the screens giving instructions, and the release notes. There's also a specific log message providing a solution. So they will just come across an issue, learn that it is justified (works in their favour, in the long run), and be instructed how to fix it easily.

And I'm not sure everybody here is aware of the degree of chaos that this issue causes. This is the stumbling block for SuiteCRM. I spend a lot of time in the forums. The hassle of the current situation is 10x, maybe 100x, greater than the hassle of this proposal.

So we need to take our pick. I think it's pretty obvious, it just takes some time getting our heads around it. Things have been this way for so long we've gotten used to them.

[chris001]

• When I first solved this issue for SugarCRM 6.5 CE in spring 2012, my "quick and easy solution" was to remove the lines from cron.php where it exits if it detects it's being called from the http, and change the cron job so that it would run cron.php via http context instead of calling php to run cron.php locally on the command line environment. This was accepted by SugarCRM bug tracker staff, with the additional precaution of requiring I add a secret pre-shared key to the URL, in order to prevent Denial of Service attacks on your CRM server, from the global internet. This solution put a slight additional load on the server every minute due to the cron causing a new web server process usage, not very noticeable, yet technically it is real. All the same, this solved the "permissions/ownership ruined up by cron" bug.

*/5 * * * * curl --silent --compressed "http://www.company.com/crm/cron.php" >/dev/null 2>&1

• Better solution is to run cron.php from the command line, as designed, exit immediately when called from the internet as the original behavior, change the user and group cron is running as, to match the php process user and group detected during installation. Any files and directories created or modified would then automatically receive the proper user/group ownership. The bug would then be solved.

I have the code I just have to add it into the fork and do the pull request.

[pgorod]

@chris001 your first solution is interesting, I hadn't thought of that.

About the second bullet, the"better solution", I don't understand what you mean when you say change the user and group cron is running as, to match the php process user and group detected during installation, is that using a setuid technique like @gunnicom suggested?

The two difficulties I see with that approach are a) what happens in Windows? and b) you might have edge cases where setuid will fail because you're actually trying to elevate privileges instead of lowering them. You need to be root to use setuid, which might not be the case, but still be problematic for file/folder ownership. On the other hand, on a weird configuration the web app could be running as root while cron is running with lower privileges, and setuid would fail...

If what you meant was setuid, please explain how that could work since I don't know much about it... thanks.

[gunnicom]

If someone wants to test this. On Ubuntu you can edit the crontab like:
sudo crontab -u www-data -e

If you ran the cron as root before dont forget to correct your permissions, otherwise the www-data cron wont run:
sudo chmod -R 775 cache custom data modules && sudo chmod 664 config.php && sudo chown www-data:www-data * -R

[chris001]

Over the weekend I've written and tested the PHP code to fix this bug or enhance the application, depending on how you look at it. Debating whether to add it into an existing file such as cron.php, or add it as a 3rd party library, and call it from one or two lines in cron.php and from elsewhere as needed, such as from the application installer code.

[pgorod]

I'm curious about your approach, and I'd love to see that code. Even if it's not finished yet you could just paste it here so we can take a peek. : - )

Thanks for your help.

[anothermouse]

Lets start with the most blatant perceived problem....people running the cron.php as root. It's easy to deal with:

$user=posix_getuid();
echo $user."\n";

if ($user==0) //then root
{
posix_setuid(1000);
posix_setgid(1000);

}

However, how do we find the correct user that the webserver is running as? How about:

file_put_contents("tmpFile", "123");
$user = fileowner("tmpFile");
echo $user."\n";
$group = filegroup("tmpFile");
echo $group."\n";
unlink("tmpFile");

run during initial installation and configuration and value stored somewhere?....perhaps in the database?

That stops people inadvertently running as root, as root can be made to run as the correct web server user. trying to come up with a suitable strategy for a non-root user is actually more difficult. Under these circumstances running as root is better as the script can be written to give up root permissions early on. This is the approach used by many system daemons and works for *nix systems. I'm just wondering whether this might break on a windows system? Requirement for php-system?

Unfortunately on a shared box, the user may not have root/admin access available to them, and under these circumstances, chris001's appoach to trigger the task via a curl cron task is actually becoming more appealing.

[pgorod]

My heart is still with the "Slightly New Proposal" I gave above. It's straightforward, makes sense, and works everywhere.

And it's about a few hours of development time, really ridiculous when compared with the benefits it would bring. I hate myself for having the life of a fireman, running from one place to another, and not yet having had the time to put aside an afternoon to develop this myself...

[anothermouse]

Code snippet above allows point 1.
For point 3, code snippet allows switching to that user if root. Doesn't fully work for work for users on shared platforms as they are only part of the www-data group rather than running as that user, unless they access it via the curl approach (with authentication secret - unique key?)
Point 2 requires time to implement.
Therefore step one is to integrate the web user into the install code so that it appears in the config.php

[pgorod]

@samus-aran thank you for coming in on a Saturday to do some merging. We were needing some of that to get our spirits up : - )

If you care to take a look at this issue here I believe it would be in the best interests of this project. Never mind the fact that it's labelled "Suggestion", it's really a bug and a very old and nasty one.

If this gets merged I promise I'll put in a couple more permissions/robustness fixes. I have this all figured out and it's perfectly fixable. SuiteCRM does not need to fail as much as it does on account of permissions. We can fix this. I just need to see that your interest in this matches mine...

[samus-aran]

@pgorod Haha, don't expect any more Saturdays runs tho! But seriously, had a quick review of the PR supplied and it looks sound probably only needs tiding up. I'll test it but I will not merge this weekend as I will discuss the solution internally next week. 👍 Thanks again for all your hard work!

[chris001]

Hi @samus-aran Very busy, yet, look for a PR from me coming any day now, to solve this category of issues thoroughly.

[samus-aran]

Was going to close this as moving it to the Suggestion Box, but instead I'll keep it open but add it also so you can see the progression.

[samus-aran]

Robustness Big Issue 1: cron shouldn't be allowed to run as root

[samus-aran]

Ok @chris001 and @pgorod

Are you two working together so you both aren't making each other work redundant? I will mark this on the Suggestion Box a Community roadmap item and set to In Progress as you two have already establish your work on it.

[pgorod]

Hi @samus-aran. Chris first promised the code back in December, but after insisting with him a few times I just went ahead and coded it myself. I'm not complaining, I see he's been quite busy with numerous other valuable contributions.

I am happy with the current proposal and I think it is ready to be merged today. I don't see a point in waiting for a second version of the same thing. However, if @chris001 explains to us that his proposal is different, and better than this one, that's cool with me. Normally when we are both talking to each other here in Github he's teaching me, and I'm just learning : - )

I would hate to see this delayed, though, so let's all try to keep things moving...

[samus-aran]

No Problem @pgorod !
@chris001 any immediate conflicts on your side or can we (i.s. SA) try and progress this ASAP?

[chris001]

@samus-aran @pgorod No conflicts, the code doesn't overlap, you can try and progress this ASAP.

[ghost]

I didn't read every line in this thread, but just want to back the general intention of this thread. I've been administering Sugar CE / Suite for 6 years. The re-occurring permissions problem is the most wacko thing about the project as a whole. Everybody just knows something's broken, but since it can (almost) always be fixed with a global resetting of permissions, it's never been fixed. I sincerely hope you guys are coming up with a good solution.
I've since installed another cron-job (as root) that sets the permissions via a script every night... That fixed it for me. But things can go wrong there and this is not a solution for everybody.

Again, I think a solid fix and/or consistent information about this would add alot to the overall feel and appeal of SuiteCRM.

[pgorod]

@EZS-JD I am pretty confident I have the problem diagnosed, basically the permissions problems come from this issue (fixed since 7.8.3, but you can fix it in any version by configuring cron correctly), and from the issue described here #2810 (the initial issue, plus other issues described later in that thread). This is still waiting to be fixed, I hope to do it soon. But it has a lot less influence than the cron jobs issue.

But I would say if you configured your cron jobs correctly, then your permissions wouldn't degrade. You don't need a script to reset permissions, what you really need is for them not to degrade.

If you want you can read about the whole thing in this blog post, and then if you have any questions start a thread in the forums and I'll help you do it specifically for your system.

[ghost]

Thanks for the excellent blog post. Should've had it a few years ago :-)

I have our systems set up with www-data running cron. But I guess I had no way of knowing whether that would really fix all the permissions issues I had been seeing over the years (and even experts seemed just used to having occasionally), so I got that daily permission settings script running... Now that it's running I see no reason to turn it off :-)

But I love that you guys are digging deep and trying to clean up the mess. Thanks for your efforts! It's all a little over my head I'm afraid.

[pgorod]

The one reason for getting rid of you script would be this: if permissions degrade, bugs happen; sometimes these bugs can really break some part of your data, or even of your code files, since SuiteCRM is an app that writes itself (Studio etc).

Of course if you reset your permissions nightly, these things are mitigated. But they are not fixed. You are 95%, or 99% covered for problems, not 100%. Your degraded permissions can damage your system between the time when they degrade, and the night-time script.

If your cron jobs run under user www-data, you would just need to check if your web-server is also running under www-data. If you're using 7.8.3 or newer, you can just check your config.php for value inside cron_allowed_users array.

[ghost]

I understand and agree. our apache is running as www-data and suitecrm-scheduler-cron is too.

[pgorod]

So, if you're feeling courageous take down that nightly script, just fix your permissions one final time running these commands from your SuiteCRM directory:

    sudo chown -R www-data:www-data .
    sudo chmod -R 755 .
    sudo chmod -R 775 cache custom modules themes data upload config_override.php 

Also make sure in config.php you have something like this:

  array (
    'dir_mode' => 1517,
    'file_mode' => 420,
    'user' => 'www-data',
    'group' => 'www-data',
  ),

Until you gain confidence in this, you can regularly watch for any degraded permissions with this command on the same directory:

tree -iupf . | grep -v www-data

It will say something like 3329 directories, 14955 files, that's ok, but if it lists files or directories owned by any user different from www-data, watch out.

Of course only you can make the decision whether this is too risky for you or not (there are also risks to not doing it!).

If you want a really prudent way, just run that tree command automatically inside your script, before resetting permissions, and append the output to a log file. After a few weeks you can have some notion of how much your permissions are degrading in real life, now that your cron jobs are well configured.

[ghost]

Thanks for the detailed info. I don't however feel that courageous right now (if it ain't broke, ....)
But I'll try to remember that this issue may be soon fixed in future versions and will revisit when we do a next big upgrade.

[SuiteBot]

This issue has been mentioned on SuiteCRM. There might be relevant details there:

https://community.suitecrm.com/t/unable-to-progress-past-the-templates-page-when-creating-an-email-campaign/88671/11