Very slow after the latest 2023.05 update #13373

tio-trom · 2023-08-22T13:59:57Z

This is something so weird and I still do not understand it.

Context.

This is our instance https://social.trom.tf . Our database is around 30GB in size. We use a daemon for Friendica with these configs:

/etc/mysql/mariadb.conf.d/50-server.cnf

[mariadb]
key_buffer_size = 16K
max_allowed_packet = 16M
table_open_cache = 4
sort_buffer_size = 4M
read_buffer_size = 256K
read_rnd_buffer_size = 256K
net_buffer_length = 2K
thread_stack = 128K
table_definition_cache = 500
innodb_buffer_pool_size = 30G

/etc/php/8.2/fpm/pool.d/friendica.conf

[friendica]

user = friendica
group = friendica

chdir = /var/www/friendica

listen = /var/run/php/php8.2-fpm-friendica.sock
listen.owner = www-data
listen.group = www-data

pm = static
pm.max_children = 100
pm.max_requests = 500
pm.start_servers = 70
pm.min_spare_servers = 70
pm.max_spare_servers = 80
request_terminate_timeout = 1d


pm.process_idle_timeout = 10s

; Additional php.ini defines, specific to this pool of workers.

php_admin_value[upload_max_filesize] = 50M
php_admin_value[post_max_size] = 500M
php_admin_value[memory_limit] = 512M

/etc/nginx/conf.d/social.trom.tf.d/friendica.conf

#sub_path_only rewrite ^/$ / permanent;
location /  {

    alias /var/www/friendica/;

    if (!-e $request_filename) {
    	rewrite ^(.*)$ /index.php?pagename=$1;
    }

    #allow uploads up to 20MB in size
    client_max_body_size 50M;
    client_body_buffer_size 128k;

    #Default indexes and catch-all
    index  index.php;

  # make sure webfinger and other well known services aren't blocked
  # by denying dot files and rewrite request to the front controller
  location ^~ /.well-known/ {
    allow all;
   if (!-e $request_filename) {
     rewrite ^(.*)$ /index.php?pagename=$1;
   }
  }

  include mime.types;

  # block these file types
  location ~* \.(tpl|md|tgz|log|out)$ {
    deny all;
  }

    #Execute and serve PHP files
    location ~* \.php$ {
fastcgi_buffers 8 16k;
        fastcgi_buffer_size 32k;
        fastcgi_connect_timeout 60;
        fastcgi_send_timeout 300;
        fastcgi_read_timeout 300;
        fastcgi_split_path_info ^(.+?\.php)(/.*)$;
        try_files $uri =404;
        fastcgi_pass unix:/var/run/php/php8.2-fpm-friendica.sock;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    }

  # deny access to all dot files
  location ~ /\. {
    deny all;
  }

  #Include SSOWAT user panel.
  include conf.d/yunohost_panel.conf.inc;
}

Basically I tried increasing the values for these. At times I got these logs for PHP FPM:

[22-Aug-2023 14:28:54] WARNING: [pool friendica] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 5 idle, and 19 total children
[22-Aug-2023 14:33:01] WARNING: [pool friendica] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 37 total children
[22-Aug-2023 14:33:02] WARNING: [pool friendica] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 45 total children

I increased these values now I do not seem to get these errors, for now, but nothing else improved.

Server stats:

Message queues
44199 - 654

The weird part is this: it is ONLY slow (takes minutes to load) if I try to access my own https://social.trom.tf/network - regardless what sub-sections I choose like (last posts, starred, etc.). The rest of the Friendica loads very fast. If I use other profiles where I have a lot less stuff in my newsfeed it loads faster, but still much slower than before.

I am baffled I do not know what more to try. My server should be capable o handling all of this, has plenty of resources.

I tried to optimize the database with mysqloptimize -p friendica and I get this:

note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.2fa_recovery_codes
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.2fa_trusted_browser
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.account-suggestion
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.account-user
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.addon
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.advancedcontentfilter_rules
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.apcontact
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.application
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.application-marker
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.application-token
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.arrived-activity
note     : The storage engine for the table doesn't support optimize
friendica.attach
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.auth_codes
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.cache
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.challenge
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.clients
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
friendica.config
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK

It gets stuck for a long time at friendica.config and seems to not continue.... And I get 504 Gateway Time-out nginx for the main url...

Even more weird, before the update everything worked well for me and other people but for one friend it was again super slow when accessing the Network page. We also had a similar issue before, see #10916 - but now none of the fixes there seem to work.

Looking at the Network with an Inspector does not seem to point to anything suspicious:

I am baffled....

The text was updated successfully, but these errors were encountered:

tio-trom · 2023-08-22T14:16:08Z

Checking the processes:

 	2023-08-22 	ClearCache
9 	2023-08-22 	ContactDiscovery
11 	2023-08-22 	Contact\Remove
1 	2023-08-22 	Cron
2 	2023-08-22 	Notifier
12 	2023-08-22 	OnePoll
1 	2023-08-22 	PostUpdate
1 	2023-08-22 	PullDirectory
1 	2023-08-22 	UpdateBlockedServers
286 	2023-08-22 	UpdateContact
1 	2023-08-22 	UpdateContacts
2 	2023-08-22 	UpdateGServer
1 	2023-08-22 	UpdateGServers
1 	2023-08-22 	UpdateServerDirectories
5 	2023-08-22 	UpdateServerDirectory
5 	2023-08-22 	UpdateServerPeers
2 	2023-08-22 	UpdateSuggestions

Nothing extreme

Raroun · 2023-08-22T14:20:09Z

Do you have any php fatals in your php error log?

tio-trom · 2023-08-22T14:29:13Z

Do you have any php fatals in your php error log?

This is the log for the PHP https://paste.yunohost.org/febabileto I do not see any.

tio-trom · 2023-08-22T14:40:49Z

Looking at the CPU it seems to spike up a lot. Close to 98% and we have tens of php-fpm: pool friendica processes running. Mind you it worked perfectly fine and no major CPU impact before the update.

tio-trom · 2023-08-22T15:29:43Z

I will have to postpone this for a week or so. I need to do a presentation in 2 days and Friendica is one of the things I talk about, so I need to have a properly working Friendica. Thus I reverted back to an old friendica that works. As soon as I can I will do the update again and update this post. If there is anyone who can give me more tips that'd be lovely. THanks!

Raroun · 2023-08-22T15:38:13Z

I probably can´t help you :(

But i experienced exactly the same behavior some Months ago when my Instance ran into a Post with some thousand activities.
Everythig worked, but the Network Page took like 3 Minutes to load - on a very big server.

Actually i have some endless loops on my instance - but that shouldn´t be you problem because i don´t see php process reach max_memory and terminatet in your logs.

Probably it is some Post your instance catched thats weird. I hope dev´s can help :)

tio-trom · 2023-08-22T15:48:21Z

This is very interesting indeed....now that I know it happened to others too, it is something that perhaps we should detect as a bug somewhere. I will test it again in detail in a week or two. Thanks a lot for the comments!

Raroun · 2023-08-22T15:58:01Z

When you check in a week or so, please look at your ram - in both of my cases the ram got exhaustet very fast.
Good luck with your presentation!

tio-trom · 2023-08-22T16:06:09Z

When you check in a week or so, please look at your ram - in both of my cases the ram got exhaustet very fast. Good luck with your presentation!

Thanks! My RAM is doing fine, maybe the RAM that Friendica uses is limited? idk but we will check.

tio-trom · 2023-09-06T23:55:11Z

I have installed it again. Same issues. Even if I should have a different feed of posts in my Network tab. For other accounts of mine that only follow a few people, the Network tab is fast. But for my account takes around 30 seconds. If I filter the feeds by groups, the "friends" group where I have most of my contact, takes many minutes to load....

I tested it with Tokodon, so basically an app, and everything loads super fast. So, same server, but different "interfaces". What does this say?

I wonder if there are ways to try and improve the loading speeds for Friendica in general.

MrPetovan · 2023-09-07T03:08:26Z

It's tricky because highly customized queries (like on circles) are inherently more costly. I'm not sure why your network tab is slower, the number of follow shouldn't be an issue. Are you following accounts that mainly have long threads with dozens of replies? This would slow down the network tab significantly, as the page limit is only for top-level posts, after that we have to independently fetch the individual conversations' children.

Try reducing the number of posts per page in your settings, it should make the network tab significantly faster. I know it did for me.

annando · 2023-09-07T05:42:32Z

Have you activated the slow query log for your SQL server?

tio-trom · 2023-09-07T12:33:06Z

Try reducing the number of posts per page in your settings, it should make the network tab significantly faster. I know it did for me.

Oh yeah I had 40. Now 10 and it is a lot faster. Thank you. I wonder tho what is different now since before the update all loaded fine....

Are you following accounts that mainly have long threads with dozens of replies?

Not really...

Have you activated the slow query log for your SQL server?

No. But I am looking into it now. Thank you!

tio-trom · 2023-09-07T12:40:10Z

Have you activated the slow query log for your SQL server?

This is what I am seeing:


Tcp port: 3306  Unix socket: /run/mysqld/mysqld.sock
Time		    Id Command	Argument
# Time: 230907 14:36:55
# User@Host: friendica[friendica] @ localhost []
# Thread_id: 151599  Schema: friendica  QC_hit: No
# Query_time: 15.242331  Lock_time: 0.001896  Rows_sent: 1  Rows_examined: 3690065
# Rows_affected: 0  Bytes_sent: 11544
use friendica;
SET timestamp=1694090215;
SELECT `uid`, `id`, `parent`, `guid`, `network`, `gravity`, `uri-id`, `uri`, `thr-parent-id`, `thr-parent`, `parent-uri-id`, `parent-uri`, `conversation`, `commented`, `created`, `edited`, `received`, `verb`, `object-type`, `postopts`, `plink`, `wall`, `private`, `starred`, `origin`, `parent-origin`, `title`, `body`, `language`, `content-warning`, `location`, `coord`, `app`, `rendered-hash`, `rendered-html`, `object`, `quote-uri`, `quote-uri-id`, `allow_cid`, `allow_gid`, `deny_cid`, `deny_gid`, `mention`, `global`, `author-id`, `author-link`, `author-name`, `author-avatar`, `author-network`, `author-updated`, `author-gsid`, `author-addr`, `author-uri-id`, `owner-id`, `owner-link`, `owner-name`, `owner-avatar`, `owner-network`, `owner-contact-type`, `owner-updated`, `causer-id`, `causer-link`, `causer-name`, `causer-avatar`, `causer-contact-type`, `causer-network`, `contact-id`, `contact-uid`, `contact-link`, `contact-name`, `contact-avatar`, `writable`, `self`, `cid`, `alias`, `event-created`, `event-edited`, `event-start`, `event-finish`, `event-summary`, `event-desc`, `event-location`, `event-type`, `event-nofinish`, `event-ignore`, `event-id`, `question-id`, `question-multiple`, `question-voters`, `question-end-time`, `has-categories`, `has-media`, `delivery_queue_count`, `delivery_queue_done`, `delivery_queue_failed`, `featured`, `post-type`, `post-reason` FROM `post-user-view` WHERE (`quote-uri-id` IN (16452364, 16452361, 16452358, 16452351, 16452350, 16452348, 16452345, 16452338, 16452336, 16452306, 16452302, 16452241, 16452215, 16452213, 16452212, 16452208, 16452199, 16452194, 16452182, 16452179, 16452177, 16452169, 16452165, 16452164, 16452163, 16452160, 16452159, 16452151, 16452129, 16452128, 16452126, 16452081, 16452059, 16451988, 16451970, 16451962, 16451939, 16451905, 16451904, 16451853, 16451804, 16451757, 16451651, 16451645, 16451621, 16451604, 16451569, 16451553, 16451404, 16450584, 16450299, 16449623, 16448778, 16448689, 16448633, 16448474, 16448461, 16448197, 16447886, 16447844, 16447833, 16447821, 16446907, 16446868, 16446822, 16446526, 16446525, 16446523, 16445543, 16444845, 16444833, 16444831, 16444830, 16444829, 16444274, 16444100, 16443860, 16443517, 16443266, 16443183, 16442970, 16442941, 16442881, 16442032, 16441787, 16441642, 16440954, 16440698, 16440616, 16440588, 16439926, 16439706, 16439705, 16439611, 16439275, 16438983, 16438858, 16438847, 16438586, 16438581, 16438546, 16438527, 16438302, 16438301, 16438003, 16437981, 16437979, 16437930, 16437848, 16437722, 16437685, 16437635, 16436373, 16436310, 16436007, 16435864, 16435675, 16435631, 16435614, 16435611, 16435583, 16435572, 16435546, 16435511, 16435268, 16434865, 16434792, 16434689, 16434631, 16434628, 16434587, 16434556, 16434309, 16434285, 16434214, 16433960, 16433951, 16433468, 16433350, 16433111, 16433061, 16432673, 16432610, 16432606, 16432438, 16432392, 16432374, 16432334, 16432330, 16432288, 16432118, 16431933, 16431919, 16431661, 16431648, 16431424, 16431368, 16431362, 16431311, 16431266, 16431243, 16431105, 16431099, 16431082, 16430908, 16430857, 16430662, 16430580, 16430504, 16430473, 16430378, 16430302, 16430126, 16430076, 16430066, 16430032, 16430015, 16429854, 16429836, 16429807, 16429790, 16429788, 16429785, 16429696, 16429652, 16429618, 16429615, 16429605, 16429566, 16429561, 16429509, 16429430, 16429411, 16429314, 16429299, 16429295, 16429281, 16429277, 16429275, 16429271, 16429241, 16429131, 16428996, 16428990, 16428811, 16428799, 16428718, 16428668, 16428611, 16428595, 16428551, 16428392, 16428382, 16428310, 16428228, 16428197, 16428164, 16428103, 16428059, 16427916, 16427776, 16427589, 16427583, 16427562, 16427557, 16427503, 16427482, 16427480, 16427403, 16427373, 16427350, 16427288, 16427266, 16427219, 16427155, 16427056, 16427039, 16426925, 16426909, 16426888, 16426869, 16426791, 16426773, 16426688, 16426673, 16426487, 16425533, 16425174, 16425127, 16424138, 16423982, 16423905, 16423094, 16421421, 16420799, 16420680, 16420631, 16419453, 16419269, 16418488, 16418048, 16417683, 16417483, 16417248, 16416690, 16416557, 16416535, 16416164, 16414923, 16414697, 16414273, 16414040, 16413702, 16413533, 16413361, 16412066, 16411854, 16411625, 16411275, 16410632, 16410441, 16410422, 16410237, 16408936, 16408492, 16407851, 16406455, 16405841, 16405676, 16405520, 16405515, 16405501, 16405283, 16404951, 16404944, 16404904, 16404485, 16404080, 16401158, 16400974, 16400619, 16399942, 16399485, 16397213, 16396410, 16396295, 16395862, 16395539, 16395367, 16394324, 16392732, 16392327, 16392117, 16391684, 16390647, 16390234, 16390229, 16390046, 16389877, 16389233, 16389212, 16388739, 16388443, 16143708, 16142901, 16137397, 16135651, 16129763, 16127101, 16124532, 16123360, 16123316, 16122559, 16121387, 16119299, 16119076, 16118990, 16118753, 16118650, 16118623, 16118521, 16118338, 16118003, 16116626, 15928063) AND `body` = '' AND `uid` = 0);
# Time: 230907 14:37:29
# User@Host: friendica[friendica] @ localhost []
# Thread_id: 151735  Schema: friendica  QC_hit: No
# Query_time: 12.430650  Lock_time: 0.000722  Rows_sent: 40  Rows_examined: 31347511
# Rows_affected: 0  Bytes_sent: 5069
SET timestamp=1694090249;
SELECT * FROM `network-thread-view` WHERE (`uid` = 2) ORDER BY `received` DESC LIMIT 40;
# Time: 230907 14:38:10
# User@Host: friendica[friendica] @ localhost []
# Thread_id: 151897  Schema: friendica  QC_hit: No
# Query_time: 11.867627  Lock_time: 0.001918  Rows_sent: 1  Rows_examined: 3690082
# Rows_affected: 0  Bytes_sent: 11544
SET timestamp=1694090290;
SELECT `uid`, `id`, `parent`, `guid`, `network`, `gravity`, `uri-id`, `uri`, `thr-parent-id`, `thr-parent`, `parent-uri-id`, `parent-uri`, `conversation`, `commented`, `created`, `edited`, `received`, `verb`, `object-type`, `postopts`, `plink`, `wall`, `private`, `starred`, `origin`, `parent-origin`, `title`, `body`, `language`, `content-warning`, `location`, `coord`, `app`, `rendered-hash`, `rendered-html`, `object`, `quote-uri`, `quote-uri-id`, `allow_cid`, `allow_gid`, `deny_cid`, `deny_gid`, `mention`, `global`, `author-id`, `author-link`, `author-name`, `author-avatar`, `author-network`, `author-updated`, `author-gsid`, `author-addr`, `author-uri-id`, `owner-id`, `owner-link`, `owner-name`, `owner-avatar`, `owner-network`, `owner-contact-type`, `owner-updated`, `causer-id`, `causer-link`, `causer-name`, `causer-avatar`, `causer-contact-type`, `causer-network`, `contact-id`, `contact-uid`, `contact-link`, `contact-name`, `contact-avatar`, `writable`, `self`, `cid`, `alias`, `event-created`, `event-edited`, `event-start`, `event-finish`, `event-summary`, `event-desc`, `event-location`, `event-type`, `event-nofinish`, `event-ignore`, `event-id`, `question-id`, `question-multiple`, `question-voters`, `question-end-time`, `has-categories`, `has-media`, `delivery_queue_count`, `delivery_queue_done`, `delivery_queue_failed`, `featured`, `post-type`, `post-reason` FROM `post-user-view` WHERE (`quote-uri-id` IN (16452412, 16452405, 16452404, 16452397, 16452396, 16452395, 16452390, 16452388, 16452382, 16452380, 16452376, 16452374, 16452367, 16452364, 16452361, 16452358, 16452351, 16452350, 16452348, 16452345, 16452338, 16452336, 16452306, 16452302, 16452241, 16452215, 16452213, 16452212, 16452208, 16452199, 16452194, 16452179, 16452169, 16452149, 16452129, 16452106, 16452059, 16452054, 16451988, 16451979, 16451970, 16451962, 16451651, 16451621, 16451604, 16451569, 16451553, 16451404, 16451402, 16451237, 16450379, 16450299, 16449623, 16448778, 16448689, 16448633, 16448474, 16448461, 16448197, 16447886, 16447844, 16447833, 16447821, 16446907, 16446868, 16446822, 16446737, 16444845, 16444833, 16444831, 16444830, 16444829, 16444274, 16444100, 16443860, 16443517, 16443183, 16442970, 16442941, 16442881, 16442032, 16440698, 16440616, 16440588, 16439926, 16439706, 16439705, 16439611, 16438983, 16438847, 16438581, 16438302, 16438301, 16437981, 16437979, 16437930, 16437848, 16437722, 16437685, 16437635, 16436007, 16435864, 16435675, 16435631, 16435614, 16435583, 16435572, 16435546, 16434792, 16434689, 16434631, 16434628, 16434587, 16434556, 16434309, 16434285, 16434214, 16433960, 16433951, 16433468, 16433350, 16433111, 16432610, 16432606, 16432374, 16432334, 16432330, 16432288, 16432118, 16431933, 16431919, 16431661, 16431648, 16431424, 16431368, 16431362, 16431311, 16431266, 16431243, 16431105, 16431099, 16430857, 16430662, 16430580, 16430504, 16430473, 16430378, 16430302, 16430126, 16430032, 16430015, 16429854, 16429836, 16429807, 16429790, 16429788, 16429785, 16429696, 16429652, 16429618, 16429615, 16429605, 16429566, 16429561, 16429509, 16429430, 16429411, 16429314, 16429299, 16429295, 16429281, 16429277, 16429275, 16429271, 16429241, 16428996, 16428990, 16428811, 16428799, 16428718, 16428668, 16428611, 16428595, 16428551, 16428392, 16428310, 16428228, 16428197, 16428059, 16427916, 16427776, 16427589, 16427562, 16427557, 16427503, 16427482, 16427480, 16427403, 16427373, 16427350, 16427288, 16427266, 16427219, 16427155, 16427056, 16427039, 16426925, 16426909, 16426888, 16426869, 16426791, 16426773, 16426688, 16426673, 16426487, 16423982, 16423094, 16421421, 16420799, 16420680, 16420631, 16419453, 16419269, 16418488, 16418048, 16417683, 16417483, 16417248, 16416690, 16416557, 16416535, 16416164, 16414923, 16414697, 16414273, 16414040, 16413702, 16413533, 16413361, 16412066, 16411854, 16411625, 16411275, 16410632, 16410441, 16410422, 16410237, 16408936, 16408492, 16407851, 16406455, 16405841, 16405676, 16405520, 16405515, 16405501, 16405283, 16404951, 16404944, 16404904, 16404485, 16404080, 16401158, 16400974, 16400619, 16399942, 16399485, 16397213, 16396410, 16396295, 16395862, 16395539, 16395367, 16394324, 16393080, 16392732, 16392327, 16392117, 16391684, 16390647, 16390234, 16390229, 16390046, 16389877, 16389233, 16389212, 16388739, 16388443, 16343217) AND `body` = '' AND `uid` = 0);
# Time: 230907 14:38:16
# User@Host: friendica[friendica] @ localhost []
# Thread_id: 151896  Schema: friendica  QC_hit: No
# Query_time: 20.358720  Lock_time: 0.000656  Rows_sent: 100  Rows_examined: 28411096
# Rows_affected: 0  Bytes_sent: 11486
SET timestamp=1694090296;
SELECT * FROM `network-thread-view` WHERE (`uid` = 171) ORDER BY `commented` DESC LIMIT 100;
# Time: 230907 14:38:28
# User@Host: friendica[friendica] @ localhost []
# Thread_id: 151896  Schema: friendica  QC_hit: No
# Query_time: 10.842191  Lock_time: 0.001112  Rows_sent: 0  Rows_examined: 3690073
# Rows_affected: 0  Bytes_sent: 8043
SET timestamp=1694090308;
SELECT `uid`, `id`, `parent`, `guid`, `network`, `gravity`, `uri-id`, `uri`, `thr-parent-id`, `thr-parent`, `parent-uri-id`, `parent-uri`, `conversation`, `commented`, `created`, `edited`, `received`, `verb`, `object-type`, `postopts`, `plink`, `wall`, `private`, `starred`, `origin`, `parent-origin`, `title`, `body`, `language`, `content-warning`, `location`, `coord`, `app`, `rendered-hash`, `rendered-html`, `object`, `quote-uri`, `quote-uri-id`, `allow_cid`, `allow_gid`, `deny_cid`, `deny_gid`, `mention`, `global`, `author-id`, `author-link`, `author-name`, `author-avatar`, `author-network`, `author-updated`, `author-gsid`, `author-addr`, `author-uri-id`, `owner-id`, `owner-link`, `owner-name`, `owner-avatar`, `owner-network`, `owner-contact-type`, `owner-updated`, `causer-id`, `causer-link`, `causer-name`, `causer-avatar`, `causer-contact-type`, `causer-network`, `contact-id`, `contact-uid`, `contact-link`, `contact-name`, `contact-avatar`, `writable`, `self`, `cid`, `alias`, `event-created`, `event-edited`, `event-start`, `event-finish`, `event-summary`, `event-desc`, `event-location`, `event-type`, `event-nofinish`, `event-ignore`, `event-id`, `question-id`, `question-multiple`, `question-voters`, `question-end-time`, `has-categories`, `has-media`, `delivery_queue_count`, `delivery_queue_done`, `delivery_queue_failed`, `featured`, `post-type`, `post-reason` FROM `post-user-view` WHERE (`quote-uri-id` IN (16452358, 16452061, 16451929, 16451903, 16451628, 16450655, 16449547, 16449521, 16449104, 16448931, 16448822, 16448803, 16448780, 16448778, 16448740, 16448627, 16448130, 16448057, 16447696, 16446976, 16446907, 16446602, 16446578, 16446559, 16446544, 16446537, 16446517, 16446490, 16446485, 16446182, 16445544, 16445168, 16445139, 16444959, 16444724, 16444148, 16443999, 16443629, 16443266, 16442949, 16442558, 16442481, 16442446, 16442443, 16442434, 16442424, 16442392, 16442362, 16442218, 16441941, 16441833, 16441787, 16441779, 16441774, 16441748, 16441726, 16441672, 16441642, 16441624, 16441569, 16441524, 16441465, 16440954, 16440832, 16440746, 16440733, 16440575, 16440538, 16440466, 16440448, 16440428, 16440336, 16440321, 16440297, 16440270, 16440268, 16440263, 16440249, 16440201, 16440188, 16440177, 16440174, 16440154, 16440139, 16439984, 16439973, 16439824, 16439811, 16439779, 16439701, 16439693, 16439681, 16439666, 16439637, 16439634, 16439623, 16439619, 16439612, 16439508, 16439499, 16439279, 16439275, 16439154, 16439073, 16438992, 16438956, 16438881, 16438858, 16438854, 16438758, 16438748, 16438646, 16438586, 16438547, 16438546, 16438527, 16438505, 16438494, 16438469, 16438447, 16438374, 16438353, 16438332, 16438284, 16438048, 16438012, 16438003, 16437890, 16437870, 16437866, 16437860, 16437835, 16437823, 16437803, 16437694, 16437689, 16437687, 16437670, 16437668, 16437649, 16437644, 16437636, 16437617, 16437616, 16437600, 16437594, 16437586, 16437511, 16437491, 16437488, 16437484, 16437467, 16437355, 16437309, 16437300, 16437279, 16437269, 16437268, 16437267, 16437256, 16437253, 16437235, 16437232, 16437231, 16437166, 16437110, 16436710, 16436596, 16436588, 16436477, 16436373, 16436343, 16436336, 16436310, 16436291, 16436272, 16436123, 16436095, 16436066, 16436050, 16436031, 16436018, 16435970, 16435722, 16435693, 16435680, 16435655, 16435642, 16435626, 16435616, 16435611, 16435597, 16435592, 16435578, 16435544, 16435535, 16435511, 16435491, 16435488, 16435407, 16435369, 16435268, 16435175, 16434877, 16434865, 16434722, 16434682, 16434479, 16434373, 16434361, 16434318, 16434147, 16434134, 16434081, 16434078, 16433980, 16433976, 16433914, 16433893, 16433845, 16433768, 16433753, 16433465, 16433427, 16433408, 16433366, 16433349, 16433290, 16433251, 16433234, 16433140, 16433115, 16433109, 16433061, 16432990, 16432955, 16432822, 16432779, 16432759, 16432737, 16432726, 16432701, 16432680, 16432677, 16432673, 16432643, 16432601, 16432562, 16432447, 16432438, 16432423, 16432414, 16432407, 16432392, 16432350, 16432342, 16432303, 16432285, 16432215, 16432142, 16432089, 16432088, 16432016, 16431979, 16431975, 16431929, 16431874, 16431768, 16431760, 16431708, 16431702, 16431646, 16431589, 16431410, 16431404, 16431397, 16431360, 16431338, 16431329, 16431304, 16431292, 16431128, 16431082, 16431070, 16431063, 16431002, 16431000, 16430999, 16430990, 16430982, 16430973, 16430968, 16430926, 16430908, 16430849, 16430829, 16430799, 16430722, 16430719, 16430703, 16430690, 16430683, 16430672, 16430671, 16430658, 16430649, 16430645, 16430636, 16430618, 16430608, 16430598, 16430590, 16430578, 16430556, 16430533, 16430519, 16430515, 16430418, 16430388, 16430346, 16430298, 16430241, 16430237, 16430235, 16430208, 16430202, 16430192, 16430190, 16430185, 16430182, 16430173, 16430162, 16430161, 16430159, 16430145, 16430143, 16430138, 16430130, 16430121, 16430106, 16430096, 16430093, 16430092, 16430090, 16430089, 16430083, 16430076, 16430074, 16430066, 16430061, 16430056, 16430049, 16430042, 16430031, 16430028, 16430022, 16430021, 16430019, 16430014, 16429951, 16429940, 16429866, 16429513, 16429501, 16429403, 16429401, 16429329, 16429306, 16429179, 16429131, 16429119, 16428982, 16428855, 16428770, 16428731, 16428658, 16428648, 16428519, 16428471, 16428412, 16428382, 16428297, 16428283, 16428209, 16428164, 16428103, 16427966, 16427923, 16427769, 16427745, 16427583, 16427448, 16426857, 16426813, 16426678, 16426644, 16426617, 16426499, 16426442, 16426408, 16426390, 16426389, 16426351, 16426337, 16426197, 16426193, 16426185, 16426115, 16426073, 16425930, 16425862, 16425666, 16425614, 16425587, 16425533, 16425364, 16425260, 16425174, 16425127, 16424686, 16424605, 16424313, 16424138, 16424136, 16423931, 16423928, 16423909, 16423905, 16423716, 16423537, 16423162, 16422983, 16422890, 16422507, 16422293, 16422275, 16422220, 16422058, 16421722, 16421077, 16420848, 16420770, 16420541, 16420056, 16419667, 16418745, 16418469, 16414303, 16413235, 16412744, 16411566, 16411411, 16410622, 16410103, 16409875, 16409748, 16408893, 16408676, 16408664, 16408636, 16408322, 16407762, 16407398, 16406591, 16405912, 16405511, 16405083, 16403620, 16403614, 16403567, 16403382, 16403341, 16403170, 16402984, 16402943, 16402694, 16402658, 16402481, 16402319, 16402318, 16402305, 16402193, 16402007, 16401925, 16401823, 16401404, 16401273, 16401243, 16401098, 16401093, 16400991, 16400946, 16400667, 16400545, 16400284, 16400202, 16400199, 16400181, 16400046, 16400001, 16399950, 16399932, 16399884, 16399870, 16399768, 16399740, 16399654, 16399528, 16399431, 16399407, 16399316, 16399216, 16399127, 16399117, 16399103, 16398396, 16397822, 16397589, 16397526, 16397476, 16397398, 16397139, 16397136, 16397044, 16396866, 16396477, 16396154, 16395711, 16395694, 16395478, 16395366, 16395361, 16395338, 16395291, 16395236, 16395046, 16394689, 16394370, 16394342, 16394224, 16394128, 16394007, 16393873, 16393695, 16393643, 16393567, 16393483, 16393406, 16393092, 16393031, 16392922, 16392894, 16392882, 16392766, 16392754, 16392723, 16392716, 16391776, 16391767, 16391717, 16391705, 16391696, 16391674, 16391655, 16391629, 16391621, 16391618, 16391590, 16391512, 16391489, 16391478, 16391454, 16391433, 16391404, 16391262, 16391197, 16391165, 16391070, 16391060, 16391052, 16391046, 16391033, 16391024, 16391014, 16391004, 16390990, 16390987, 16390964, 16390958, 16390953, 16390951, 16390943, 16390932, 16390926, 16390906, 16390903, 16390896, 16390892, 16390878, 16390835, 16390818, 16390811, 16390787, 16390626, 16390248, 16382623, 16374454, 16372371, 16366134, 16365627, 16365159, 16361055, 16344004, 16315386, 16309928, 16309771, 16306526, 16305904, 16305880, 16305827, 16305771, 16304791, 16304785, 16304649, 16304642, 16304168, 16302374, 16302339, 16302282, 16302227, 16302169, 16301938, 16301928, 16301789, 16301577, 16301564, 16301167, 16300731, 16300669, 16300624, 16299923, 16299805, 16299696, 16298541, 16298504, 16298216, 16298125, 16297884, 16297360, 16296729, 16296250, 16295888, 16295737, 16295102, 16294705, 16294409, 16294286, 16294079, 16293939, 16278365, 16274063, 16272312, 16271735, 16270138, 16270091, 16250123, 16245858, 16245857, 16245555, 16244924, 16244876, 16244752, 16244628, 16244418, 16242611, 16242460, 16242263, 16241990, 16241814, 16241802, 16241632, 16241433, 16241392, 16241377, 16241303, 16241202, 16231872, 16200899, 16199415, 16192514, 16191479, 16191203, 16191118, 16173999, 16173845, 16172431, 16170354, 16169588, 16168917, 16167994, 16167023, 16166799, 16166754, 16166687, 16166676, 16166524, 16166506, 16166410, 16166243, 16063280, 15989194, 15937517, 15889899, 15887253, 15884833, 15884821, 15883302, 15881983, 15876805, 15873695, 15872765, 15872703, 15872281, 15871480, 15868036, 15867554, 15867450, 15866575, 15866072, 15864940, 15864550, 15860170, 15860118, 15859740, 15859575, 15858979, 15858715, 15858632, 15858582, 15858125, 15858046, 15857911, 15857832, 15856943, 15856685, 15856359, 15855642, 15855345, 15854914, 15854875, 15854713, 15853396, 15853086, 15853064, 15852925, 15852852, 15852822, 15852782, 15852743, 15852456, 15852362, 15852272, 15852098, 15852014, 15851942, 15851854, 15851639, 15851590, 15851409, 15851252, 15851143, 15851115, 15851016, 15850961, 15850908, 15850614, 15850548, 15850524, 15850349, 15849603, 15849433, 15849303, 15849161, 15848733, 15848668, 15848369, 15848327, 15848244, 15848128, 15848097, 15847925, 15847817, 15847677, 15847658, 15847440, 15847368, 15847252, 15847239, 15847015, 15846350, 15846278, 15846193, 15846135, 15846075, 15846006, 15845918, 15845732, 15845475, 15843168, 15248601, 15247186, 15180030, 15177814, 15174833, 15174652, 15174428, 15173871, 15173704, 15173548, 15170078, 15168535, 15165742, 15163898, 15163393, 15162436, 15162132, 15161732, 15159853, 15159403, 15159171, 15158983, 15158959) AND `body` = '' AND `uid` = 0);
# Time: 230907 14:39:13
# User@Host: friendica[friendica] @ localhost []
# Thread_id: 152245  Schema: friendica  QC_hit: No
# Query_time: 33.778032  Lock_time: 0.000587  Rows_sent: 40  Rows_examined: 31347549
# Rows_affected: 0  Bytes_sent: 5069
SET timestamp=1694090353;
SELECT * FROM `network-thread-view` WHERE (`uid` = 2) ORDER BY `received` DESC LIMIT 40;
# Time: 230907 14:39:17
# User@Host: friendica[friendica] @ localhost []
# Thread_id: 152768  Schema: friendica  QC_hit: No
# Query_time: 17.548982  Lock_time: 0.000382  Rows_sent: 40  Rows_examined: 31347587
# Rows_affected: 0  Bytes_sent: 5069
SET timestamp=1694090357;
SELECT * FROM `network-thread-view` WHERE (`uid` = 2) ORDER BY `received` DESC LIMIT 40;

More here https://paste.trom.tf/timezineza.yaml

MrPetovan · 2023-09-07T13:03:34Z

# Time: 230907 14:39:17
# User@Host: friendica[friendica] @ localhost []
# Thread_id: 152768  Schema: friendica  QC_hit: No
# Query_time: 17.548982  Lock_time: 0.000382  Rows_sent: 40  Rows_examined: 31347587
# Rows_affected: 0  Bytes_sent: 5069
SET timestamp=1694090357;
SELECT * FROM `network-thread-view` WHERE (`uid` = 2) ORDER BY `received` DESC LIMIT 40;

There it is, it shouldn't be that slow, but you can see it had to examine 31 million rows. @annando Index issue?

annando · 2023-09-07T13:52:46Z

It can't be an index issue, since it is running on other machines really fast. It would be an idea to have a look into the query using explain and then to possibly adjust the configuration.

tio-trom · 2023-09-07T14:06:57Z

It can't be an index issue, since it is running on other machines really fast. It would be an idea to have a look into the query using explain and then to possibly adjust the configuration.

Would you mind telling me how to look into that?

Also why would this only happen for the Network tab? I noticed it is even worse for the admin/federation. The rest is super fast...

MrPetovan · 2023-09-07T14:12:40Z

Please perform the following query manually using a MySQL client (either command line or GUI):

EXPLAIN SELECT * FROM `network-thread-view` WHERE (`uid` = 2) ORDER BY `received` DESC LIMIT 40;

And report the full result.

tio-trom · 2023-09-07T14:15:24Z

Thanks. Here:

id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
1	SIMPLE	post-thread-user	ref	PRIMARY, uid,contact-id,post-user-id, uid_received, uid_commented, uid_starred, uid_mention, uid_wall_received	uid_received	3	const	344858	Using where
1	SIMPLE	author	eq_ref	PRIMARY, cid	PRIMARY	7	const, friendica.post-thread-user.author-id	1	Using where
1	SIMPLE	owner	eq_ref	PRIMARY, cid	PRIMARY	7	const, friendica.post-thread-user.owner-id	1	Using where
1	SIMPLE	post-user	eq_ref	PRIMARY	PRIMARY	4	friendica.post-thread-user.post-user-id	1	Using where
1	SIMPLE	contact	eq_ref	PRIMARY, pending_uid, blocked_uid	PRIMARY	4	friendica.post-thread-user.contact-id	1	Using where
1	SIMPLE	ownercontact	eq_ref	PRIMARY	PRIMARY	4	friendica.post-thread-user.owner-id	1
1	SIMPLE	parent-post	ref	uid_uri-id, uri-id, uid_contactid, uid_unseen_contactid, uid_unseen, uid_hidden_uri-id, uid_wall_received	uid_hidden_uri-id	3	friendica.post-user.uid	586	Using where; Using index

MrPetovan · 2023-09-07T14:27:39Z

Thanks, nothing looks wrong in this execution plan, it's using indices (or "keys") at every step, but maybe @annando has better insight.

annando · 2023-09-07T14:38:05Z

I can't have a deeper look, I'm still at work. I know that MariaDB has got different optimizers. Somewhere here in the issues someone tested and found out that using the old optimizer does help. I can't search for that issue in the moment.

tio-trom · 2023-09-07T14:43:54Z

I can't have a deeper look, I'm still at work. I know that MariaDB has got different optimizers. Somewhere here in the issues someone tested and found out that using the old optimizer does help. I can't search for that issue in the moment.

Perhaps this is the one you are referring to #11119 - I am doing mysqloptimize friendica - if it improves I'll update this issue.

Possibly this #11465

Trying stuff.

tio-trom · 2023-09-07T16:28:56Z

I got a lot of

note     : Table does not support optimize, doing recreate + analyze instead
status   : OK

The process finished but no improvement. I will try to improve with the old mariadb optimize method if I managed to figure out how :D

annando · 2023-09-07T17:20:08Z

No, I meant the setting for the query optimizer. The query optimizer is the part of the SQL engine that tries to improve the query and decides which index to use: For an explanation see here: https://use-the-index-luke.com/sql/glossary/query-optimizer-query-planner

There are different versions of the optimizer out there. I know that there is an issue, where someone found a switch that helped. But I don't find the issue at the moment.

tio-trom · 2023-09-07T21:20:55Z

I think it is fixed now. Based on this comment #11465 (comment)

Basically I edited the file /etc/mysal/mariadb.conf.d/50-server.cnf and added this:


[mariadb]

# speed up network transactions (name resolution not needed when DB runs on localhost)
skip_name_resolve       = 1

# thread-cache set equal to max connections
thread_cache_size       = 500
max_connections         = 500

#enable the old query optimizer setting
optimizer_use_condition_selectivity = 1
optimizer_switch = 'rowid_filter=off'

# speed up joins w/o indexes 
join_buffer_size        = 16M

# table cache settings for Friendica
table_open_cache        = 4200
table_definition_cache  = 2500

# InnoDB basic settings
innodb_file_per_table = ON
innodb_default_row_format = dynamic

# use large buffer pool
innodb_buffer_pool_size = 48G

# increased values to avoid too frequent checkpointing
innodb_log_file_size = 2G
innodb_log_buffer_size = 128M

Now it seems to load in less than 4 seconds. Which is fantastic. I wonder if I need all of those variables in the database conf file, but so far so good. I will wait a bit then I can close this or maybe you can recommend me (us) better values for the conf file. I see others complain about the same issue - https://social.trom.tf/display/b42a482c-1164-fa3b-b83d-e9b342023432

tio-trom · 2023-09-07T22:18:48Z

Remove

# speed up network transactions (name resolution not needed when DB runs on localhost)
skip_name_resolve       = 1

In case you have services like Gitea or Facilmaps. It will kill them. So far Friendica is so fast it never has been so fast :D

oe4dns · 2023-09-07T22:47:22Z

I am experiencing the same issue and this seems to fix it for me too. Still I'd like to learn the reason for the slowdown and how/what exactly fixes it.

tio-trom · 2023-09-07T22:49:55Z

I am also curious how these configs will affect other services since these are the settings for the entire mariadb....

annando · 2023-09-08T04:22:39Z

I guess you can boil down the changes to these two lines:

#enable the old query optimizer setting
optimizer_use_condition_selectivity = 1
optimizer_switch = 'rowid_filter=off'

See here: https://jira.mariadb.org/browse/MDEV-25830

oe4dns · 2023-09-08T06:37:17Z

I have tested this, and it is not sufficient!

It could be that there are two separate issues though, let me explain:

When I first experienced the slowdown, I checked Firefoxes Web Dev Console to discover that a lot of images (jpegs, pngs) took about 1000-2000ms to load each. I attributed this to the avatar images and enabled the file based avatar cache.

Long image load times where now gone, but I now had extremely long timeouts from JQuery trying to load something local (see attached image)

As stated above, applying the mariadb config from #13373 (comment)
fixes the issue completely. If I only switch to the old query optimizer, Friendica remains slow and the long image fetches are back. So this does something to the avatar cache settings.

Related question - where is the avatar cache supposed to be. I could not find the files and am therefore not sure it is really working. How to check? This might help to uncover the root of the issue because a non-working avatar cache would likely add to the long load times.

annando · 2023-09-08T07:20:54Z

@oe4dns whatever causes your problems, I guess that (like you already said) this is a separate issue, so I suggest to split it into separate issues. Putting too much into a single issues often causes long and clogged issues where at the end no one really knows if all issues have been solved.

oe4dns · 2023-09-08T07:52:39Z

This is exactly what I am trying to do because I am definitely also having the very issue discussed here. So I have to somehow exclude the other one from interfering with my tests.

tio-trom · 2023-09-08T10:45:01Z

I guess we can trial and error and see what values fix the issue. Since my instance has over a thousand users and I use it daily I can only do it late at night. Maybe @oe4dns you can test it?

oe4dns · 2023-09-08T11:30:39Z

Well yes, I already did.

#13373 (comment)

And discovered the fact that I might have two separate issues. So I am currently trying to isolate them before I can do more testing

tio-trom · 2023-09-08T11:32:49Z

Well yes, I alreafy did.

#13373 (comment)

And discovered the fact that I might have two separate issues. So I am currently trying to isolate them before I can do more testing

Ah sorry ok I see. I will look further into this.

oe4dns · 2023-09-08T11:36:11Z

I will also do so, in the evening

oe4dns · 2023-09-09T06:24:39Z

I have now separated the two issues (mainly by disabling the file-based avatar cache, but that is another story)

I can already report that only switching to the old optimizer does not help.

Applying all other configuration changes without the optimizer settings also does not work. So the optimizer is indeed part of the issue.

More testing will follow.

AlfredSK · 2023-09-09T09:24:27Z

I guess you can boil down the changes to these two lines:
#enable the old query optimizer setting
optimizer_use_condition_selectivity = 1
optimizer_switch = 'rowid_filter=off'

I had a lot of trouble with the default settings for the query optimizer in the past. And those settings helped a lot to get better query plans with much faster execution.
But lately, with current stable version of MariaDB 10.6, I can use the default settings again. The problem with very, very slow queries seems to be gone.

oe4dns · 2023-09-09T20:25:54Z

After after more testing I think I have found the culprit - or better the config option that speeds up reloads.

Contrary to my former reports, switching to the old optimizer does not help! This has been a wrong observation caused by the fact that even if the correct config settings are made, the first reload is still slow. Subsequent reloads are much snappier.

This is what I have now added to my /etc/mysal/mariadb.conf.d/50-server.cnf file.

[mariadb]

# use large buffer pool
innodb_buffer_pool_size = 16G

As you can see I reduced the buffer pool from 48G to 16G which still seems to be enough. I did not test other values. No other changes were necessary on my server to speed up the reloads. But only after the first reload which is still slow - I think it is filling the buffer at first load but this is just a guess.

Commenting out this line leads to slow page loads again - this is reproducible.

Please test. Maybe there can be a conclusion/solution for future friendica releases too.

tio-trom · 2023-09-09T20:43:52Z

Well now let me ruin the party. For me only having

optimizer_use_condition_selectivity = 1
optimizer_switch = 'rowid_filter=off'

Does the job...everything is so fast is unbelievable. No other settings in the config does the job but this....

innodb_buffer_pool_size = 16G makes things faster for sure, but nowhere near the above one.

So yah....there we go. For me is different :D

oe4dns · 2023-09-09T20:57:29Z

What is your mariadb version? I am running version 10.11 on Debian bookworm.

In #13373 (comment) it was stated that there were performance issues with the query optimzier up to version 10.5 of mariadb - so maybe there's a reason.

And regarding the innodb_buffer_pool_size - I am running a pretty low end system here - If you're on bigger hardware, maybe the cache is not an issue for you. These could be two different reasons with the same outcome - slow network page loads

tio-trom · 2023-09-09T21:00:04Z

What is your mariadb version? I am running version 10.11 on Debian bookworm.

MariaDB version 10.5.19

oe4dns · 2023-09-09T21:03:44Z

Looks suspicious ;-)

tio-trom · 2023-09-09T21:06:13Z

Yeah idk but the difference in speed is unbelievable with just those settings. It is almost instant....

Our server has an AMD EPYC 7282 16-Core Processor and 64GB of RAM.

oe4dns · 2023-09-09T21:13:24Z

In my case setting the buffer size does not make it instant - reloads take a few seconds. But nowhere close to minutes as before. It is a dualcore vserver with 8G RAM.

Switching to the old query optimiser settings does not improve it further. I missed this in my first tests, because the first reload still was slow.

I'm pretty convinced that we have two issues here

mariadb < 10.6 seems to require the "old" query optimisation
low spec systems need the innodb_buffer_pool_size set to a big enough value (16G in my case)

Maybe this helps some of the devs to figure out what changed in 2023.5 that lead to these issues. There could even be a common cause.

tio-trom · 2023-09-15T17:08:38Z

For me this is fixed. I will close. Please open a new issue if you need so. I think it makes sense. If not let me know and I will reopen.

tio-trom added the Bug label Aug 22, 2023

MrPetovan added the Performance label Aug 22, 2023

tio-trom closed this as completed Sep 15, 2023

MrPetovan closed this as not planned Won't fix, can't repro, duplicate, stale Dec 19, 2023

Very slow after the latest 2023.05 update #13373

Very slow after the latest 2023.05 update #13373

Comments

tio-trom commented Aug 22, 2023 • edited

tio-trom commented Aug 22, 2023

Raroun commented Aug 22, 2023

tio-trom commented Aug 22, 2023

tio-trom commented Aug 22, 2023

tio-trom commented Aug 22, 2023

Raroun commented Aug 22, 2023

tio-trom commented Aug 22, 2023

Raroun commented Aug 22, 2023

tio-trom commented Aug 22, 2023

tio-trom commented Sep 6, 2023

MrPetovan commented Sep 7, 2023

annando commented Sep 7, 2023

tio-trom commented Sep 7, 2023 • edited

tio-trom commented Sep 7, 2023 • edited

MrPetovan commented Sep 7, 2023

annando commented Sep 7, 2023

tio-trom commented Sep 7, 2023

MrPetovan commented Sep 7, 2023

tio-trom commented Sep 7, 2023 • edited by MrPetovan

MrPetovan commented Sep 7, 2023

annando commented Sep 7, 2023

tio-trom commented Sep 7, 2023 • edited

tio-trom commented Sep 7, 2023

annando commented Sep 7, 2023

tio-trom commented Sep 7, 2023 • edited

tio-trom commented Sep 7, 2023

oe4dns commented Sep 7, 2023

tio-trom commented Sep 7, 2023

annando commented Sep 8, 2023

oe4dns commented Sep 8, 2023

annando commented Sep 8, 2023

oe4dns commented Sep 8, 2023 • edited

tio-trom commented Sep 8, 2023

oe4dns commented Sep 8, 2023 • edited

tio-trom commented Sep 8, 2023

oe4dns commented Sep 8, 2023

oe4dns commented Sep 9, 2023

AlfredSK commented Sep 9, 2023

oe4dns commented Sep 9, 2023

tio-trom commented Sep 9, 2023

oe4dns commented Sep 9, 2023

tio-trom commented Sep 9, 2023

oe4dns commented Sep 9, 2023

tio-trom commented Sep 9, 2023

oe4dns commented Sep 9, 2023 • edited

tio-trom commented Sep 15, 2023

tio-trom commented Aug 22, 2023 •

edited

tio-trom commented Sep 7, 2023 •

edited

tio-trom commented Sep 7, 2023 •

edited

tio-trom commented Sep 7, 2023 •

edited by MrPetovan

tio-trom commented Sep 7, 2023 •

edited

tio-trom commented Sep 7, 2023 •

edited

oe4dns commented Sep 8, 2023 •

edited

oe4dns commented Sep 8, 2023 •

edited

oe4dns commented Sep 9, 2023 •

edited