[Bug]: Very slow access to apps.nextcloud.com/api/v1/apps.json for checking app updates

[hammeractual]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

Often to always failed when checking for Apps Update caused by very slow access to https://apps.nextcloud.com/api/v1/apps.json
Connection speed on the Client and on the Server seems normal, but when accessing https://apps.nextcloud.com/api/v1/apps.json either from the server or from the client is pretty slow (mostly failed to load) for json file to be loaded completely.

Steps to reproduce

1. Login
2. Go to Admin Page
3. Overview Menu
4. Never finish "Checking apps for compatible versions"

Expected behavior

Checking update for apps completed successfully.

Installation method

Community Manual installation with Archive

Nextcloud Server version

28

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Updated from a MINOR version (ex. 22.1 to 22.2)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

No response

List of activated Apps

No response

Nextcloud Signing status

No response

Nextcloud Logs

[appstoreFetcher] Warning: Could not connect to appstore: cURL error 28: Operation timed out after 60000 milliseconds with 903254 out of 6408674 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json
	GET /ocs/v2.php/apps/updatenotification/api/v1/applist/28.0.5.1
	from x.x.x.x by e2cd37b6-6ebf-103d-84c2-addc165358a8 at 3 Mei 2024, 09.09.50

Additional info

No response

[bigcat88]

This issue is related more to the appstore and related to infrastructure, imho.

Sometimes getting of json from "api/v1/apps.json" can fail(when appstore is under high load), but in 98-99% cases it should be fine.

[hammeractual]

This issue is related more to the appstore and related to infrastructure, imho.

Sometimes getting of json from "api/v1/apps.json" can fail(when appstore is under high load), but in 98-99% cases it should be fine.

Could you please tell the infrastructure division/department to check/troubleshoot the issue?

Since it can't load the apps.json file, I can't update the apps and also can't accessing the Appstore too to install apps. :(

[bigcat88]

Since it can't load the apps.json file, I can't update the apps and also can't accessing the Appstore too to install apps. :(

It works for me now and has worked since the very beginning of today.
there was no peak load in the appstore today, there shouldn't have been any slowdowns

[hammeractual]

Since it can't load the apps.json file, I can't update the apps and also can't accessing the Appstore too to install apps. :(

It works for me now and has worked since the very beginning of today. there was no peak load in the appstore today, there shouldn't have been any slowdowns

How to increase the curl timeout in the nextcloud to fetch apps.json file completely? Do you have any clue?

[joshtrichards]

@hammeractual What does curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test from your Nextcloud Server's command line look like? Here's how it looks for me from a test instance:

www-data@ef86c134f021:/tmp$ curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.9M  100 13.9M    0     0  6545k      0  0:00:02  0:00:02 --:--:-- 6548k
www-data@ef86c134f021:/tmp$ 

[appstoreFetcher] Warning: Could not connect to appstore: cURL error 28: Operation timed out after 60000 milliseconds with 903254 out of 6408674 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json

How to increase the curl timeout in the nextcloud to fetch apps.json file completely? Do you have any clue?

#40082 has a manual approach for doing so. Also it's tracking the feature request.

[hammeractual]

@hammeractual What does curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test from your Nextcloud Server's command line look like? Here's how it looks for me from a test instance:

www-data@ef86c134f021:/tmp$ curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.9M  100 13.9M    0     0  6545k      0  0:00:02  0:00:02 --:--:-- 6548k
www-data@ef86c134f021:/tmp$ 

[appstoreFetcher] Warning: Could not connect to appstore: cURL error 28: Operation timed out after 60000 milliseconds with 903254 out of 6408674 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json

How to increase the curl timeout in the nextcloud to fetch apps.json file completely? Do you have any clue?

#40082 has a manual approach for doing so. Also it's tracking the feature request.

Thanks for the info

Here is mine.

Meanwhile, the ping result is pretty good to that address.

[bigcat88]

Here is mine.

I was wrong.
It seems that this is not a problem with the appstore code itself, since during high loads the appstore simply does not serve the file at all, but if it has already started serving, it does it quickly, because the file itself is given by NGINX(reverse proxy).

[joshtrichards]

Meanwhile, the ping result is pretty good to that address.

Something does not add up. Your throughput is incredibly slow relative to your latency. You're getting like 6558 B/s (!).

Can you sanity check that curl performance by downloading something from elsewhere:

Maybe curl https://github.com/nextcloud/server/archive/refs/tags/v28.0.5.zip -o /dev/null

P.S. Is your Server @ Hetzner?

[hammeractual]

Meanwhile, the ping result is pretty good to that address.

Something does not add up. Your throughput is incredibly slow relative to your latency. You're getting like 6558 B/s (!).

Can you sanity check that curl performance by downloading something from elsewhere:

Maybe curl https://github.com/nextcloud/server/archive/refs/tags/v28.0.5.zip -o /dev/null

P.S. Is your Server @ Hetzner?

Here is mine

Quite fast while curl to another server/address.

[hammeractual]

Any updates guys? :'(

I've tried to increase timeout value in Fetcher.php to 1800 seconds, but no success.

[appstoreFetcher] Warning: Could not connect to appstore: cURL error 28: Operation timed out after 1800000 milliseconds with 3661824 out of 6421307 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json GET /settings/apps from x.x.x.x by e2cd37b6-6ebf-103d-84c2-addc165358a8 at 7 Mei 2024, 17.14.29

[joshtrichards]

Any updates guys? :'(

I think the challenge is we're having a hard time reproducing this and there doesn't seem be an explanation for why your environment is experiencing this.

It's tough to fix without a clear (or even suspected possible) cause. :-)

[joshtrichards]

P.S. Is your Server @ Hetzner?

You may have missed this question. I asked because your ping times are ~1-2 ms.

At the time I was taking a shot in the dark that maybe there is something unusual about Hetzner's internal client-to-client connectivity. Today I ran across a note in the Portainer Knowledge Base while trying to help out a user on the forums that suggests Hetzner's private networking uses an MTU of 1450. Got me thinking about this again.