500 error after apt update

[koopa]

After running an apt update of the onlyoffice-* pkgs on ubuntu I got an 500 Internal Server Error

I found that the mail aggregator service failed to start. Is that causing the 500?
systemctl shows:

onlyofficeMailAggregator.service - ONLYOFFICE MailAggregator Service
   Loaded: loaded (/lib/systemd/system/onlyofficeMailAggregator.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2022-06-09 12:49:33 CEST; 1s ago
  Process: 22599 ExecStartPre=/bin/bash -c rm -f /tmp/onlyofficeMailAggregator (code=exited, status=0/SUCCESS)
  Process: 22600 ExecStart=/usr/bin/mono-service -d:/var/www/onlyoffice/Services/MailAggregator -l:/tmp/onlyofficeMailAggregator /var/www/onlyoffice/Services/MailAggregator/ASC.Mail.Aggregator.CollectionService.exe (code=exited, status=0/SUCCESS)
 Main PID: 22601 (code=exited, status=1/FAILURE)

Jun 09 12:49:31 oocs systemd[1]: Starting ONLYOFFICE MailAggregator Service...
Jun 09 12:49:31 oocs systemd[1]: Started ONLYOFFICE MailAggregator Service.
Jun 09 12:49:33 oocs mono[22601]: /var/www/onlyoffice/Services/MailAggregator/ASC.Mail.Aggregator.CollectionService.exe: The given key '0' was not present in the dictionary.
Jun 09 12:49:33 oocs systemd[1]: onlyofficeMailAggregator.service: Main process exited, code=exited, status=1/FAILURE
Jun 09 12:49:33 oocs systemd[1]: onlyofficeMailAggregator.service: Failed with result 'exit-code'.

Any suggestions?

[koopa]

Found more errors in web.log:

2022-06-09 11:37:10,033 ERROR [5] ASC.Resources - Can not get resource from TalkResource.resx for en: GetString(ProductName), System.Collections.Generic.KeyNotFoundException: The given key '0' was not present in the dictionary.
2022-06-09 11:37:10,145 ERROR [5] ASC.Resources - Can not get resource from TalkResource.resx for Neutral: GetString(ProductName), System.Collections.Generic.KeyNotFoundException: The given key '0' was not present in the dictionary.
2022-06-09 11:37:10,502 ERROR [5] ASC.Resources - ReplaceLogo System.Collections.Generic.KeyNotFoundException: The given key '0' was not present in the dictionary.
2022-06-09 11:37:10,796 ERROR [5] ASC.Web - Warmup error System.Collections.Generic.KeyNotFoundException: The given key '0' was not present in the dictionary.
2022-06-09 11:37:10,801 ERROR [5] ASC - Start Warmup System.NullReferenceException: Object reference not set to an instance of an object
2022-06-09 11:37:45,851 ERROR [5] ASC.Resources - Can not get resource from CRMCommonResource.resx for en-US: GetString(ProductName), System.Collections.Generic.KeyNotFoundException: The given key '0' was not present in the dictionary.
2022-06-09 11:37:45,970 ERROR [5] ASC.Resources - Can not get resource from CRMCommonResource.resx for en: GetString(ProductName), System.Collections.Generic.KeyNotFoundException: The given key '0' was not present in the dictionary.
2022-06-09 11:37:46,224 ERROR [5] ASC.Resources - Can not get resource from CRMCommonResource.resx for Neutral: GetString(ProductName), System.Collections.Generic.KeyNotFoundException: The given key '0' was not present in the dictionary.
2022-06-09 11:37:46,601 ERROR [5] ASC.Resources - ReplaceLogo System.Collections.Generic.KeyNotFoundException: The given key '0' was not present in the dictionary.
2022-06-09 11:37:46,788 ERROR [5] ASC.Resources - Can not get resource from CRMVoipResource.resx for en-US: GetString(VoipModuleTitle), System.Collections.Generic.KeyNotFoundException: The given key '0' was not present in the dictionary.
2022-06-09 11:37:46,873 ERROR [5] ASC.Resources - Can not get resource from CRMVoipResource.resx for en: GetString(VoipModuleTitle), System.Collections.Generic.KeyNotFoundException: The given key '0' was not present in the dictionary.
2022-06-09 11:37:46,965 ERROR [5] ASC.Resources - Can not get resource from CRMVoipResource.resx for Neutral: GetString(VoipModuleTitle), System.Collections.Generic.KeyNotFoundException: The given key '0' was not present in the dictionary.
2022-06-09 11:37:47,216 ERROR [5] ASC.Resources - ReplaceLogo System.Collections.Generic.KeyNotFoundException: The given key '0' was not present in the dictionary.
[more results like that cut off...]

It seems like it's looking for some key "0" but can't find it?

[Carazyda]

Hello @koopa, please download our new script you used to install and run with -u true parameter. New components that are needed for the new version will be installed.

[koopa]

@Carazyda Thanks for the info! I'm not sure which script you mean exactly, this: https://download.onlyoffice.com/install/install-Debian.sh ?
But if I remember correctly I only added some apt-get deb sources and then ran apt-get install. Will this still work? Also do I need to stop some services first or just run it? (with -u)

Edit: Sorry, could have just looked at the script! It does what i did manually back then.
Now I ran it and it removed onlyoffice-documentserver to replace it with onlyoffice-documentserver-ee and it installed some ASP and .NET stuff. Fingers crossed, I'll update this when I know more.

[Carazyda]

Yes, if you installed using this script, then download the new version and run it with the sudo bash install-Debian.sh -u true. You don't need to stop anything.

We recommend updating through our scripts, because new dependencies may appear in the new version and they are installed by the script.

[koopa]

Thanks so much for the quick help, you rock!
The script was running and warnings like that showed up - they can probably be ignored though?

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: plugin [ingest-attachment] was built for Elasticsearch version 7.4.0 but version 7.10.0 is required

It ended with

Thank you for installing ONLYOFFICE.
You can now configure your portal using the Control Panel
In case you have any questions contact us via http://support.onlyoffice.com or visit our forum at http://dev.onlyoffice.org

So looks everything went fine! However, the server doesn't seem to be up yet. Instead of the 500 now the connection is refused as if the server isn't up. Going to investigate.

[koopa]

Had to change the server_name back (was set to _) in the nginx conf (HTTPS proxy).
Now I get a connection, but it's looking like the CSS is not being loaded, it showing a very bare white page with a couple of list items and links.

[Carazyda]

Sorry, we had problems with scripts. Now everything should be ok. Try to download the script again and run the update.

Are you using nginx proxy? in which config did you change it? maybe the necessary headers have also flown there?

[koopa]

Alright, thanks for the update and no worries! Going to give it a try in a sec.
Good question, where can I find the required header configs again? So far I only adjusted the server_name.

[koopa]

Ok, ran the script again.

Warnings gone except one from apt:

W: Conflicting distribution: https://download.mono-project.com/repo/debian stable-buster/snapshots/6.8.0.123 InRelease (expected stable-buster/snapshots/6.8.0.123 but got buster)

However, the page is still the same. when I try to load the CSS directly with wget it doesn't return any data it seems:

--2022-06-09 18:30:26--  (try: 2)  https://groups.local/discbundle/common/css/_Bf3BZrh8JqnMWNZxxCxig2.css?ver=12.0.0.1726
Connecting to groups.local (groups.local)|10.1.1.10|:443... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

Any ideas?

BTW: I also noticed there are empty lines at the top of the page source, before the DOCTYPE:

Or is that normal?

[Carazyda]

It looks like the bundle is broken. Try to recreate it. To do this, delete the folder rm -rf /var/www/onlyoffice/Data/bundle and restart the site systemctl restart monoserve.

[koopa]

Deleted and recreated the "bundle" dir but it can still not download the files. The 3 newlines at the top are also still there.

Index page looks like this:

No response for the css and js files, example:

--2022-06-09 20:59:05--  (try: 2)  https://groups.local/discbundle/common/javascript/first-ueNbtxPzcji_ZoVXSk3i5Q2.js?ver=12.0.0.1726
Connecting to groups.local (groups.local)|10.1.1.10|:443... connected.
HTTP request sent, awaiting response... No data received.

But file is definitely there on the filesystem:

~# find /var/www/onlyoffice/Data/bundle/ -name first-ueNbtxPzcji_ZoVXSk3i5Q2.js
/var/www/onlyoffice/Data/bundle/common/javascript/first-ueNbtxPzcji_ZoVXSk3i5Q2.js

There seems to be something wrong with my nginx setup maybe?
When you said the headers for nginx, did you mean those proxy_header_set directives here, right?
https://github.com/ONLYOFFICE/document-server-proxy/blob/master/nginx/minimal.conf

[koopa]

Ok, I also ran rm -rf /var/cache/nginx/tmp/* to remove some nginx caches, then the css and js files downloaded properly and now everything works and looks good.
Without your help this would have taken me much longer to track down. Thanks again @Carazyda!

[Carazyda]

Glad to hear it! We will write a separate instruction for setting up a nginx proxy for a community server.