Errors on non-pi systems

[borpin]

Hi @TrystanLea Just installed on an Ubuntu system. had an error because of hardcoded user name so tried again with the user pi created.

1. No log window

2. Demandshaper not in menu

Just realised it has moved 🤦‍♂

3. Dashboards not imported

4. Cannot create new dashboards

[andyjbm]

Hi,
Also tried to install on a non-pi system (debian based).
Exec summary of issues: mainly hard coded user pi and hardcoded /var/www/emoncms.
Also updater not honouring choices made in install/config.ini at install time.

So this is a quick keyboard bash just to list them before I forget and have to rediscover them next time i do this!

It appears that convention defines webroot in $emoncms_www from the config.ini file but there are places where this is not honoured.

My emoncms webroot is not at /var/www (because this serves another application) its at /srv/dev-disk-by-label-SSD/webroot/emoncms. Now I could create a symlink /var/www/emoncms to point to /srv...etc but...

Seeing the option to define it in install/config.ini in $emoncms_www I thought this was the way to go rather than mucking about with symlinks but there are places where this is not honoured eg the service unit files that point back to the .php and .py files that are the services.

In emoncms/scripts/services/feedwriter/feedwriter.service:

• Hardcoded user pi
• Environment='USER=pi' single quotes should not be there. ie should be Environment=USER=pi
• chown statement failing. Did it manually then commented it out.
• Hardcoded webroot as /var/www/emoncms and not referencing $emoncms_www
• Hardcoded Log path as /var/log/emoncms

feedwriter service fails to start unless feeds redis low write mode is enabled. So is this only for SDcard based installs? Do i not need it then? Do I need to install it?

In emoncms/scripts/services/service-runner/service-runner.service

• User hardcoded as pi
• webroot hard coded as /var/www/emoncms and not referencing $emoncms_www

In emoncms_core.sh updater:

• Only checks for settings.php user changes - not settings.ini. Does this matter? I used settings.ini thinking that was the new way - is this not ready yet? Should I stick with settings.php?

• Reinstall of feedwriter and service runner services will break them because of hardcoded /var/www/emoncms path in the service units.

• The updater will install the mqtt service even if it wasn't installed by the installer (set false in install/config.ini). (Same for feedwriter and service-runner I guess)

All for now..

Thanks, Andy

[borpin]

@andyjbm - it should be better than this.

Just to confirm, you edited the config.ini to set the www path and set the emonSD_pi_env = 0

EmonScripts/install/emonsd.config.ini

Line 6 in 3359f9b

emonSD_pi_env=1

[borpin]

• Hardcoded user pi

This is solved by the use of a drop-in for non-pi users.

EmonScripts/install/emoncms_core.sh

Line 87 in d03d8e1

if [ "$user" != "pi" ]; then

• chown statement failing. Did it manually then commented it out.

It will if the USER is wrong.

feedwriter service fails to start unless feeds redis low write mode is enabled.

Yes Feedwriter, service-runner and emoncms_mqtt all need REDIS - it is a wants in the service file.

I think this is the source of your issues as these services are not installed by the installer unless REDIS is installed.

EmonScripts/install/emoncms_core.sh

Line 78 in d03d8e1

if [ "$install_redis" = true ]; then

• Environment='USER=pi' single quotes should not be there. ie should be Environment=USER=pi

Not according to the docs https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Environment=

[borpin]

• Only checks for settings.php user changes - not settings.ini.

Because the newer 'ini' mechanism is hierarchical, it doesn't need to. In the 'old days' if a new setting was added, the user had to regenerate the settings.php file to include the new setting.

• Reinstall of feedwriter and service runner services will break them because of hardcoded /var/www/emoncms path in the service units.

Depends how you reinstall them. Again, if the emonscripts config is correct, I think it is fine as it passes a $servicepath variable to the service installer.

• The updater will install the mqtt service even if it wasn't installed by the installer (set false in install/config.ini). (Same for feedwriter and service-runner I guess)

That could be a bug. Can you point me to where this may be happening? (I think you are right #96 )

[andyjbm]

Just to confirm, you edited the config.ini to set the www path and set the emonSD_pi_env = 0

@borpin Yes I did all of that. I more or less single stepped through the installer to see what was happening.

[andyjbm]

@borpin

• chown statement failing. Did it manually then commented it out.

It will if the USER is wrong.

I modified the service-runner.service file with USER=root and it still failed: eg:

Environment=USER=root
Environment=LOG_PATH=/var/log/emoncms
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p ${LOG_PATH}
# ExecStartPre=/bin/chown ${USER} ${LOG_PATH}
ExecStartPre=/bin/chmod 666 ${LOG_PATH}/emoncms.log

• Environment='USER=pi' single quotes should not be there. ie should be Environment=USER=pi

Not according to the docs https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Environment=

It didn't appear to make any difference actually. I came across more examples without quotes than with.
I tried something like:

touch /var/log/emoncms/$(USER).txt

to see what was happening and the file created was always wrapped in single quotes thus 'root'.txt

At that point life became too short and I just wanted to get the install working.

As an aside I would suggest migrating to a user eg emoncms for the services and for emoncms in general which would make it pi agnostic. In hindsight I wish I'd done that rather than setting them up as root.

Andy.

[andyjbm]

@borpin

feedwriter service fails to start unless feeds redis low write mode is enabled.

Yes Feedwriter, service-runner and emoncms_mqtt all need REDIS - it is a wants in the service file.
Yes the redis service was up and running, in settings.ini redis=true was set but low write was false. Isn't there a distinction between using feedwriter for low write mode feed updates and using redis to communicate to service-runner to execute backups and postprocess?

So I don't actually need feedwriter if I don't want low write functionallity but service-runner will still work?
I'm really wanting backups to work from the webgui which needs redis & service-runner to work. I've sussed that much but not sure on the settings.ini that is required.

I think this is the source of your issues as these services are not installed by the installer unless REDIS is installed.

This was actually an upgrade from V9.8 (2017). So nginx, php, Redis were all installed and up.

I basically renamed my old "/var/www/emoncms" and started again, also moving feed data files from /var/lib to /var/opt/emoncms etc as I could see this is the direction you were headed.
In the new settings.ini I had:

[redis]
enabled = true
...
[feed]
redisbuffer[enabled] = false

And saw that /var/www/emoncms/scripts/feedwriter.php also needed redisbuffer[enabled] = true to work.

Yes Feedwriter, service-runner and emoncms_mqtt all need REDIS - it is a wants in the service file.

It wasn't the service unit. See here, this is what stopped me:

https://github.com/emoncms/emoncms/blob/28fb790b1aba5445f0b6c984e555e5502e491af2/scripts/feedwriter.php#L19

I think I misunderstood whether I needed feedwriter. I'm guessing I don't now.

Andy

[andyjbm]

@borpin

• Only checks for settings.php user changes - not settings.ini.

Because the newer 'ini' mechanism is hierarchical, it doesn't need to. In the 'old days' if a new setting was added, the user had to regenerate the settings.php file to include the new setting.

Ok. So I needn't worry about this.

[andyjbm]

@borpin

• Reinstall of feedwriter and service runner services will break them because of hardcoded /var/www/emoncms path in the service units.

Depends how you reinstall them. Again, if the emonscripts config is correct, I think it is fine as it passes a $servicepath variable to the service installer.

I was thinking I saw a git pull over the top of the emoncms folder in the update script which will overwrite the now modified service-runner.service file in emoncms/scripts/services/service-runner?
As this is symlinked to from /lib/system/systemd the service would not start as the path to the service executable in the .service file is now pointing to a hardcoded base of /var/www/emoncms... etc. (My webroot is not /var/www...)

[andyjbm]

@borpin

• The updater will install the mqtt service even if it wasn't installed by the installer (set false in install/config.ini). (Same for feedwriter and service-runner I guess)

That could be a bug. Can you point me to where this may be happening? (I think you are right #96 )

hardcoded at Lines 90 and 103 in /opt/openenergymonitor/EmonScripts/update/emoncms_core.sh:

for service in "feedwriter" "emoncms_mqtt" "emonhub"; do

EmonScripts/update/emoncms_core.sh

Line 90 in 3359f9b

for service in "emoncms_mqtt" "feedwriter" "service-runner"; do

EmonScripts/update/emoncms_core.sh

Line 103 in 3359f9b

for service in "feedwriter" "emoncms_mqtt" "emonhub"; do

So as they will be present in the /var/www/emoncms folder but not necessarily required or installed as services they will get installed and started when the update is run. The inistall/config.ini settings are not honoured.

Thanks for your help in understanding.
Andy.

[borpin]

As an aside I would suggest migrating to a user eg emoncms for the services and for emoncms in general which would make it pi agnostic.

Happy to accept the PR 😁

I was thinking I saw a git pull over the top of the emoncms folder in the update script which will overwrite the now modified service-runner.service file in emoncms/scripts/services/service-runner?

Yes and that is deliberate. In general it is a bad idea to edit any service file as there is always a risk a package update will overwrite it. Either create your on drop-in or use sudo systemctl edit to create the drop-in then any changes you make survive a service file update.

Feedwriter is a buffering system that reduces disk writes. It costs nothing to run in reality and probably de-stresses any system.

I modified the service-runner.service file with USER=root and it still failed

With what error?

I've successfully installed on Debian with a different user but not with a modified www location.

[borpin]

I'm really wanting backups to work from the webgui which needs redis & service-runner to work.

Ah, did you run the install script with REDIS enabled? It does a number of different things as there are 3 separate bits of REDIS (server, PHP & Python).

[andyjbm]

As an aside I would suggest migrating to a user eg emoncms for the services and for emoncms in general which would make it pi agnostic.

Happy to accept the PR

We'll see how much is left of lockdown when I've got this working... You never know! My weakness is github - I don't use it enough. Lots of one off projects. Now how the heck did you post that smiley emoji? ;-)

I was thinking I saw a git pull over the top of the emoncms folder in the update script which will overwrite the now modified service-runner.service file in emoncms/scripts/services/service-runner?

Yes and that is deliberate. In general it is a bad idea to edit any service file as there is always a risk a package update will overwrite it. Either create your on drop-in or use sudo systemctl edit to create the drop-in then any changes you make survive a service file update.

Indeed. Not come across this situation before. /var/www... seems an unusual place for a service file?? (yeah yeah all PRs appreciated!) And it has hardcoded references. systemctl edit - ok will do it that way.

Feedwriter is a buffering system that reduces disk writes. It costs nothing to run in reality and probably de-stresses any system.

Ok so worth it then. But just to be clear this is just for feeds and not for service-runner messages? I'm seeing that now.

I modified the service-runner.service file with USER=root and it still failed

With what error?

I will try and reproduce and get back to you.

I've successfully installed on Debian with a different user but not with a modified www location.

TBF the different /var/www was the real headache. and moving from /home/pi/data to /var/opt/emoncms was a good step.