Demo: Server is slow

[benmccann]

It's taking me 25s just to load the JS on https://demo.photoprism.org/browse. Certainly some of that is that the file is a bit large at 759 kB and can be improved by things like #841, but even still it seems quite surprising that it'd take that long since it's just a static file and there's nothing being computed.

I can't really imagine what would make it take this long except maybe the demo server is overloaded? I'm curious what hosting service is used. I wonder if there's anything we can do to expose some metrics (possibly publicly) to make it clear if CPU, disk, or network is experiencing high utilization. Maybe something like Prometheus Node Exporter?

[lastzero]

Maybe there's an issue with the server / provider. Our upload to Docker Hub was also super slow, had to get a new server for this. Going to check this after our vacation. In general, PhotoPrism seems fast enough for most users and it's really super fast if the server is within the same country, even on a mobile network. I'm using it every day. No performance issues whatsoever.

[lastzero]

System load wasn't high at all. Rebooted the server. Better now?

[lastzero]

Slow routing to the US in particular seems possible too.

[benmccann]

It took me 16s this time, so it doesn't look like the reboot helped much. I agree it's not likely to be an issue with the PhotoPrism code.

Slow network does seem like a leading contender. I'm not quite sure how to confirm that diagnosis. ping doesn't take terribly long. I think the default TCP window size on Ubuntu is large enough that there shouldn't be many round trips needed.

PING demo.photoprism.org (45.132.246.223) 56(84) bytes of data.
64 bytes from v2202007125883123470.happysrv.de (45.132.246.223): icmp_seq=1 ttl=46 time=163 ms
64 bytes from v2202007125883123470.happysrv.de (45.132.246.223): icmp_seq=2 ttl=46 time=165 ms
64 bytes from v2202007125883123470.happysrv.de (45.132.246.223): icmp_seq=3 ttl=46 time=227 ms
64 bytes from v2202007125883123470.happysrv.de (45.132.246.223): icmp_seq=4 ttl=46 time=165 ms
64 bytes from v2202007125883123470.happysrv.de (45.132.246.223): icmp_seq=5 ttl=46 time=272 ms
64 bytes from v2202007125883123470.happysrv.de (45.132.246.223): icmp_seq=6 ttl=46 time=164 ms
64 bytes from v2202007125883123470.happysrv.de (45.132.246.223): icmp_seq=7 ttl=46 time=164 ms
64 bytes from v2202007125883123470.happysrv.de (45.132.246.223): icmp_seq=8 ttl=46 time=239 ms
64 bytes from v2202007125883123470.happysrv.de (45.132.246.223): icmp_seq=9 ttl=46 time=262 ms
64 bytes from v2202007125883123470.happysrv.de (45.132.246.223): icmp_seq=10 ttl=46 time=164 ms
^C
--- demo.photoprism.org ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9012ms
rtt min/avg/max/mdev = 162.847/198.392/271.869/43.510 ms

Netcup claims 2.5 Gbps on https://www.netcup.eu/vserver/ so it's not clear that it's a throughput issue either

I'm tempted to try adding Server-Timing headers (#945) to see if it gives any more info.

[lastzero]

Guess it's an issue specifically with Netcup or this server as we've probably caused above average traffic, so they may have reduced our speed. Going to move to a different hoster as soon as possible.

[lastzero]

Request times from our office in Berlin, Germany:

Looks ok, doesn't it? So maybe in fact routing to the US is the issue.

[alexislefebvre]

Demo has seen from France:

Reloading the pages show images randomly.

Maybe disabling that setting could help?

[lastzero]

No, thumbs are prerendered. Must be the routing or connection. Strange that it's fast within Germany.

[lastzero]

Performed a KVM upgrade and tested the connection from LA via VPN:

Looks reasonable.

@benmccann Has the performance improved for you?

[benmccann]

It's definitely improved though still a bit on the high side for me. It's now taking 4s for app.js to download and 52s for the page to finish loading.

[lastzero]

Interesting, the VPN connection to LA and back should have been slower as you're based in California...

[lastzero]

New server at a different provider is ordered. Won't be ready before Monday though. May also try using a CDN.

[lastzero]

Disabled prometheus. Had been testing it together with Traefik to see server load. Some users report better performance now, see community chat.

[lastzero]

Google PageSpeed looks OK for the first load of a web app (obviously slow compared to a static site they're comparing it with):

https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fdemo.photoprism.org%2F

[lastzero]

If it's still slow, maybe you reduced the bandwidth in the developer tools for testing? We had that before.

Time to Interactive 2.4 s for Desktop. That's reasonable for a larger web app.

[benmccann]

It is surprising that you're not seeing this on the LA VPN. I'm on Gigabit fiber and don't see slowness on other sites, so it shouldn't be my connection. I did disable caching for the test, but made sure that the bandwidth was not being throttled.

[lastzero]

I reloaded with a cleared cache, but didn't disable caching permanently. Maybe slower without any caching at all, but that's not how users see our demo by default. Did you also try other sites hosted in Germany (that are not using a US CDN)?

[lastzero]

[benmccann]

There's no difference between cleared and disabled cache unless viewing multiple pages is there? Both should be equivalent to a first visit to the site

Do you know of other German hosted sites without US CDN that I can test against? I'm not sure how I'd be able to determine if a site meets that description

[lastzero]

Do you know of other German hosted sites without US CDN that I can test against? I'm not sure how I'd be able to determine if a site meets that description

Our public site and CI are hosted in Frankfurt at DigitalOcean:

• https://drone.photoprism.app/
• https://photoprism.app/

[lastzero]

There's no difference between cleared and disabled cache unless viewing multiple pages is there? Both should be equivalent to a first visit to the site

For a small, static site that is probably true.

Without going through the Chrome source on a Saturday evening, I suspect there are small differences in execution. Even just havening the dev tools open makes a slight difference.

For example, our service worker, which runs in a different "thread", might download assets again that have been loaded in the main "thread" already as it caches certain URLs so that they are available offline.

[G2G2G2G]

browser request wise for cleared/disabled cache only different is a dns lookup internally in the browser and an SSL cert lookup as well

[benmccann]

photoprism.app loads much more reasonably for me. No asset takes more than 1s to load. I didn't try drone.photoprism.app because it was wanting me to log in

[lastzero]

Moved demo.photoprism.org to a new server, seems much faster now. Might take half an hour until the IP change propagated.

[lastzero]

@benmccann Do you notice a difference?

[benmccann]

No major change yet. Not sure if I'm getting the old IP or new IP, but I've included the one I get below. I'd love to get #1365 in for some more info

Request URL: https://demo.photoprism.org/static/build/app.js?650bbfa7
Remote Address: 88.198.141.220:443
Time: 3.93 s

[lastzero]

We've added CDN support to our demo today. Is loading JS from this URL still slow for you?

https://demo-cdn.photoprism.org/static/build/app.js?96ba506e

[benmccann]

The site is ridiculously faster now. Literally about 20x faster

The tradeoff is that it does make the demo site a bit less representative of how people would typically be running Photoprism. E.g. if there's some regression that arises that causes the tiles to be loaded slowly we might not notice since the CDN would be caching them.

Anyhow, thanks for all your investigation into this issue!

[lastzero]

Excellent, thanks for testing! We're using it ourselves on a small home server, also remotely through a firewall on our mobile phones. Major performance issues won't go unnoticed.

With the latest improvements, everyone can use a CDN with PhotoPrism if needed. Just need to create an account and configure the URL. Makes sense that long distance connections may lead to a bad user experience otherwise. Didn't think it's that much of a difference.

[lastzero]

As these questions came up in the chat:

• We're using https://www.keycdn.com/ in pulling / caching mode for our demo. No need to configure special keys or anything. Your CDN provider only needs your PhotoPrism site URL configured as source, which must be visible on the internet as a CDN provider typically can't pull from private networks.
• If you don't want to publicly share with others or/and run a demo like we do, using a CDN usually doesn't make much sense.
  To be faster, a CDN needs enough repeated requests for cashing and edge servers that are closer to your users than your PhotoPrism instance.

[benmccann]

Also, as far as I'm aware, CDNs only cache public files and don't operate on private files not necessarily because of repeated use but because of the caching headers used on the responses. So if you're using it for your own private photos it won't end up caching anything beyond maybe the JS/CSS

[fensminger]

For me the server respond now fast. I am French but when I go on the demo, the language was chinese or other language that I don't speak ?

[fensminger]

[alexislefebvre]

For me the server respond now fast. I am French but when I go on the demo, the language was chinese or other language that I don't speak ?

Anyone can change it in the settings, so you'll see the same language that the last user used.