[1.7.8] No activities after small load (~80000 messages in 20hours) #6052
Assignees
Labels
cat/bug
A bug which needs fixing
closed/migrated
prio/p0
The priority of a bug. p0 means blocking
source/qe
Raised by QE
zenhub/done
Milestone
Comments
mmelko
added
cat/bug
pure-bot
bot
added
the
notif/triage
|
@zregvart Do you want to take stab at this? |
|
First thing to notice is that PostgreSQL is using all available memory: |
|
There are some activities in the database: |
|
I've increased the memory limit to double the available memory to 512MB. After redeploy Postgresql is using 17MB and the activities can be seen in the UI. |
|
After few minutes the memory usage is 467MB. |
heiko-braun
removed
the
notif/triage
|
@mmelko how do I access -dev cluster ? Or can you provide the integration project for me to run locally would be good too. |
|
@claudio4j you can access the test project that @mmelko has setup here: |
|
Today activities load, they even load surprisingly fast (110-120msec). There are two possibilities that I'm considering:
We can increase the memory in the template. I'm worried about the quota limits we have on the evaluation cluster, and if I'm not mistaking the quota is 4GiB. Right now with todo and komodo enabled the maximum memory is 4382MiB. Without todo and komodo we're at 3102MiB. Each integration pod launched has a maximum of 512MiB by default. |
|
I spoke to @zregvart and maybe it is a good idea to repeat the test setup with 256MB for the syndesis-db once more forcing the error to occur another time. Once the error is there we should have a look at the logs and postgres state first. Then just restart the DB (without increasing the memory) to see if this restart only solved the problem (when #6054 is happening because of the restart we are doomed). Then increase the memory to 512MB and restart the DB to see if this really solves the issue for now. |
|
Related to #5989 ? |
|
We have this reproduced currently on rh-idev in the Here's what can be observed: From The two connections in This shows waiting=f (not locked) for all of those connections:
(16 rows) Thread dumps from the server pod: |
|
This bit of code is issuing the And the |
(6 rows) This shows that two pids ( My current thinking is that we could add @mmelko I can do an experiment in building a custom version of Syndesis |
|
Actually because of the |
|
This caught my eye, from the database log: I've tried vacuuming the I think that high number of non-removable rows (574779) is causing queries to run slow, so the inner |
|
According to: SELECT schemaname, relname, n_live_tup, n_dead_tup, last_autovacuum
FROM pg_stat_all_tables
WHERE relname = 'jsondb'
ORDER BY n_dead_tup
/ (n_live_tup
* current_setting('autovacuum_vacuum_scale_factor')::float8
+ current_setting('autovacuum_vacuum_threshold')::float8)
(1 row) The autovacuum is getting triggered but there are still a great number of dead tuples. |
|
Seems that the Without it: syndesis=> explain analyse select path from jsondb where path like '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/%';
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan using jsondb_pkey on jsondb (cost=0.55..8.57 rows=5847 width=119) (actual time=97334.348..97920.892 rows=758 loops=1)
Index Cond: ((path >= '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/'::text) AND (path < '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz0'::text))
Filter: ((path)::text ~~ '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/%'::text)
Heap Fetches: 112569
Planning time: 739.563 ms
Execution time: 97920.976 ms
(6 rows)With it: syndesis=> explain analyse select path from jsondb where path like '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/%' ORDER BY path DESC;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan Backward using jsondb_pkey on jsondb (cost=0.55..8.57 rows=5843 width=119) (actual time=529.474..612900.143 rows=758 loops=1)
Index Cond: ((path >= '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/'::text) AND (path < '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz0'::text))
Filter: ((path)::text ~~ '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/%'::text)
Heap Fetches: 112569
Planning time: 259.770 ms
Execution time: 612900.246 ms
(6 rows)I'm considering adding a sorted index based on a substring of the path column just for activity entry id, if such a thing is possible in PostgreSQL. |
|
What's the |
|
I've created an index: CREATE INDEX CONCURRENTLY jsondb_activity_idx ON jsondb (path DESC) WHERE path LIKE '/activity/exchanges/%';I'm not 100% on the I've scaled
(1 row) With this I can see the activities in the UI, I think we need to let it cook a bit and check back in 12 or so hours to see if the number of dead tuples keeps increasing or the queries take long time. |
That query deletes all but the last 50 activity entries for a integration. The |
|
I'm actually quite certain that the newly created index won't help. I'll try to redefine the index without the Seems that the planner is using the syndesis=> explain analyse select path from jsondb where path like '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/%' ORDER BY path DESC;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan Backward using jsondb_pkey on jsondb (cost=0.54..8.56 rows=3 width=121) (actual time=0.063..0.100 rows=50 loops=1)
Index Cond: ((path >= '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/'::text) AND (path < '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz0'::text))
Filter: ((path)::text ~~ '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/%'::text)
Heap Fetches: 0
Planning time: 1.186 ms
Execution time: 0.140 ms
(6 rows)
syndesis=> explain analyse select path from jsondb where path like '/activity/exchanges/%' ORDER BY path DESC;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan Backward using jsondb_pkey on jsondb (cost=0.54..8.56 rows=308 width=121) (actual time=0.043..1.043 rows=1146 loops=1)
Index Cond: ((path >= '/activity/exchanges/'::text) AND (path < '/activity/exchanges0'::text))
Filter: ((path)::text ~~ '/activity/exchanges/%'::text)
Heap Fetches: 863
Planning time: 0.382 ms
Execution time: 1.102 ms
(6 rows)
syndesis=> \d jsondb
Table "public.jsondb"
Column | Type | Modifiers
--------+-------------------+--------------------
path | character varying | collate C not null
value | character varying |
ovalue | character varying |
idx | character varying | collate C
Indexes:
"jsondb_pkey" PRIMARY KEY, btree (path)
"jsondb_activity_idx" btree (path DESC) WHERE path::text ~~ '/activity/exchanges/%'::text
"jsondb_idx" btree (idx, value) WHERE idx IS NOT NULL
syndesis=> explain analyse select path from jsondb where path like '/activity/exchanges/%' ORDER BY path DESC;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan using jsondb_activity_idx on jsondb (cost=0.40..8.42 rows=950 width=119) (actual time=0.026..0.645 rows=1283 loops=1)
Index Cond: ((path >= '/activity/exchanges/'::text) AND (path < '/activity/exchanges0'::text))
Heap Fetches: 1000
Planning time: 0.422 ms
Execution time: 0.696 ms
(5 rows)
syndesis=> explain analyse select path from jsondb where path like '/activity/exchanges/%' ORDER BY path DESC;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan using jsondb_activity_idx on jsondb (cost=0.40..8.42 rows=950 width=119) (actual time=0.013..0.482 rows=1314 loops=1)
Index Cond: ((path >= '/activity/exchanges/'::text) AND (path < '/activity/exchanges0'::text))
Heap Fetches: 1031
Planning time: 0.361 ms
Execution time: 0.542 ms
(5 rows)
syndesis=> explain analyse select path from jsondb where path like '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/%' ORDER BY path DESC;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan Backward using jsondb_pkey on jsondb (cost=0.54..8.56 rows=3 width=119) (actual time=0.020..0.041 rows=50 loops=1)
Index Cond: ((path >= '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/'::text) AND (path < '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz0'::text))
Filter: ((path)::text ~~ '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/%'::text)
Heap Fetches: 0
Planning time: 0.220 ms
Execution time: 0.053 ms
(6 rows)
syndesis=> explain analyse select path from jsondb where path like '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/%' ORDER BY path DESC;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan Backward using jsondb_pkey on jsondb (cost=0.54..8.56 rows=3 width=119) (actual time=0.022..0.034 rows=50 loops=1)
Index Cond: ((path >= '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/'::text) AND (path < '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz0'::text))
Filter: ((path)::text ~~ '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/%'::text)
Heap Fetches: 0
Planning time: 0.238 ms
Execution time: 0.050 ms
(6 rows)
syndesis=> explain analyse select path from jsondb where path like '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/%' ORDER BY path DESC;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan Backward using jsondb_pkey on jsondb (cost=0.54..8.56 rows=3 width=119) (actual time=0.020..0.041 rows=50 loops=1)
Index Cond: ((path >= '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/'::text) AND (path < '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz0'::text))
Filter: ((path)::text ~~ '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz/%'::text)
Heap Fetches: 0
Planning time: 0.231 ms
Execution time: 0.056 ms
(6 rows)
syndesis=> explain analyse select path from jsondb where path like '/activity/exchanges/%' ORDER BY path DESC;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan using jsondb_activity_idx on jsondb (cost=0.40..8.42 rows=950 width=119) (actual time=0.019..0.799 rows=1378 loops=1)
Index Cond: ((path >= '/activity/exchanges/'::text) AND (path < '/activity/exchanges0'::text))
Heap Fetches: 1095
Planning time: 0.515 ms
Execution time: 0.882 ms
(5 rows) |
|
The new index is: CREATE INDEX CONCURRENTLY jsondb_activity_idx ON jsondb (path DESC);And it results in this execution plan: syndesis=> explain analyse DELETE FROM jsondb WHERE path LIKE '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz0/%' AND path NOT IN (SELECT path FROM jsondb WHERE path LIKE '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz0/%' ORDER BY path DESC FETCH FIRST (50) ROWS ONLY);
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Delete on jsondb (cost=8.85..16.88 rows=2 width=6) (actual time=0.013..0.013 rows=0 loops=1)
-> Index Scan using jsondb_activity_idx on jsondb (cost=8.85..16.88 rows=2 width=6) (actual time=0.009..0.009 rows=0 loops=1)
Index Cond: (((path)::text >= '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz0/'::text) AND ((path)::text < '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz00'::text))
Filter: (((path)::text ~~ '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz0/%'::text) AND (NOT (hashed SubPlan 1)))
SubPlan 1
-> Limit (cost=0.41..8.43 rows=3 width=118) (never executed)
-> Index Only Scan using jsondb_activity_idx on jsondb jsondb_1 (cost=0.41..8.43 rows=3 width=118) (never executed)
Index Cond: ((path >= '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz0/'::text) AND (path < '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz00'::text))
Filter: ((path)::text ~~ '/activity/exchanges/i-LjGC-0XNPX6fkTQoWYDz0/%'::text)
Heap Fetches: 0
Planning time: 0.543 ms
Execution time: 0.042 ms
(12 rows)The index is now used. I think to be certain that there are no deadlocks we need to restructure the |
|
I've considered using OIDs as criteria instead of the path column: syndesis=> ALTER TABLE jsondb SET WITH OIDS;
ALTER TABLEThis performs worse: syndesis=> explain analyse DELETE FROM jsondb WHERE path LIKE '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/%' AND oid NOT IN (SELECT oid FROM jsondb WHERE path LIKE '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/%' ORDER BY path DESC FETCH FIRST (50) ROWS ONLY);
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Delete on jsondb (cost=2.07..10.10 rows=179 width=6) (actual time=1.192..1.192 rows=0 loops=1)
-> Index Scan using jsondb_activity_idx on jsondb (cost=2.07..10.10 rows=179 width=6) (actual time=0.157..0.835 rows=452 loops=1)
Index Cond: (((path)::text >= '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/'::text) AND ((path)::text < '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz0'::text))
Filter: (((path)::text ~~ '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/%'::text) AND (NOT (hashed SubPlan 1)))
Rows Removed by Filter: 50
SubPlan 1
-> Limit (cost=0.41..1.53 rows=50 width=117) (actual time=0.012..0.074 rows=50 loops=1)
-> Index Scan using jsondb_activity_idx on jsondb jsondb_1 (cost=0.41..8.44 rows=359 width=117) (actual time=0.012..0.067 rows=50 loops=1)
Index Cond: (((path)::text >= '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/'::text) AND ((path)::text < '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz0'::text))
Filter: ((path)::text ~~ '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/%'::text)
Planning time: 0.868 ms
Execution time: 1.234 ms
(12 rows)
syndesis=> explain analyse DELETE FROM jsondb WHERE path LIKE '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/%' AND path NOT IN (SELECT path FROM jsondb WHERE path LIKE '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/%' ORDER BY path DESC FETCH FIRST (50) ROWS ONLY);
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Delete on jsondb (cost=2.07..10.10 rows=179 width=6) (actual time=0.764..0.764 rows=0 loops=1)
-> Index Scan using jsondb_activity_idx on jsondb (cost=2.07..10.10 rows=179 width=6) (actual time=0.148..0.719 rows=63 loops=1)
Index Cond: (((path)::text >= '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/'::text) AND ((path)::text < '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz0'::text))
Filter: (((path)::text ~~ '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/%'::text) AND (NOT (hashed SubPlan 1)))
Rows Removed by Filter: 50
SubPlan 1
-> Limit (cost=0.41..1.53 rows=50 width=113) (actual time=0.038..0.084 rows=50 loops=1)
-> Index Only Scan using jsondb_activity_idx on jsondb jsondb_1 (cost=0.41..8.44 rows=359 width=113) (actual time=0.037..0.078 rows=50 loops=1)
Index Cond: ((path >= '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/'::text) AND (path < '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz0'::text))
Filter: ((path)::text ~~ '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/%'::text)
Heap Fetches: 50
Planning time: 0.657 ms
Execution time: 0.805 ms
(13 rows)WITH variant: syndesis=> explain analyse WITH d AS (SELECT oid FROM jsondb WHERE path LIKE '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/%' ORDER BY path DESC OFFSET 50 ROWS) DELETE FROM jsondb WHERE oid IN (SELECT oid FROM d);
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Delete on jsondb (cost=162.38..2984.76 rows=20704 width=34) (actual time=43.823..43.823 rows=0 loops=1)
CTE d
-> Limit (cost=0.47..8.44 rows=6642 width=111) (actual time=0.068..7.874 rows=6941 loops=1)
-> Index Scan using jsondb_activity_idx on jsondb jsondb_1 (cost=0.41..8.44 rows=6692 width=111) (actual time=0.021..7.384 rows=6991 loops=1)
Index Cond: (((path)::text >= '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/'::text) AND ((path)::text < '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz0'::text))
Filter: ((path)::text ~~ '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/%'::text)
-> Hash Join (cost=153.94..2976.32 rows=20704 width=34) (actual time=18.690..31.731 rows=6941 loops=1)
Hash Cond: (jsondb.oid = d.oid)
-> Seq Scan on jsondb (cost=0.00..2665.09 rows=41409 width=10) (actual time=0.038..11.274 rows=41378 loops=1)
-> Hash (cost=151.44..151.44 rows=200 width=32) (actual time=14.740..14.740 rows=6941 loops=1)
Buckets: 8192 (originally 1024) Batches: 1 (originally 1) Memory Usage: 553kB
-> HashAggregate (cost=149.44..151.44 rows=200 width=32) (actual time=12.677..13.631 rows=6941 loops=1)
Group Key: d.oid
-> CTE Scan on d (cost=0.00..132.84 rows=6642 width=32) (actual time=0.076..10.760 rows=6941 loops=1)
Planning time: 0.568 ms
Execution time: 44.029 ms
(16 rows)
syndesis=> explain analyse WITH d AS (SELECT path FROM jsondb WHERE path LIKE '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/%' ORDER BY path DESC OFFSET 50 ROWS) DELETE FROM jsondb WHERE path IN (SELECT path FROM d);
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------
Delete on jsondb (cost=8.87..16.91 rows=1 width=62) (actual time=0.816..0.816 rows=0 loops=1)
CTE d
-> Limit (cost=8.44..8.44 rows=1 width=115) (actual time=0.094..0.205 rows=98 loops=1)
-> Index Only Scan using jsondb_activity_idx on jsondb jsondb_1 (cost=0.41..8.44 rows=3 width=115) (actual time=0.032..0.198 rows=148 loops=1)
Index Cond: ((path >= '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/'::text) AND (path < '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz0'::text))
Filter: ((path)::text ~~ '/activity/exchanges/i-LjGDJG5NPX6fkTQoWYHz/%'::text)
Heap Fetches: 99
-> Nested Loop (cost=0.44..8.47 rows=1 width=62) (actual time=0.324..0.686 rows=98 loops=1)
-> HashAggregate (cost=0.02..0.03 rows=1 width=88) (actual time=0.313..0.329 rows=98 loops=1)
Group Key: (d.path)::text
-> CTE Scan on d (cost=0.00..0.02 rows=1 width=88) (actual time=0.099..0.259 rows=98 loops=1)
-> Index Scan using jsondb_activity_idx on jsondb (cost=0.41..8.43 rows=1 width=121) (actual time=0.003..0.003 rows=1 loops=98)
Index Cond: ((path)::text = (d.path)::text)
Planning time: 0.754 ms
Execution time: 0.869 ms
(15 rows) |
|
I think I'm settling on two chages:
|
|
Thanks for the thorough investigation @zregvart. Changing the sql statements and redoing the load tests seems to safe, right? |
|
We had a test running over night just with the index added and the I'm inclined to just add the index as the only code change and leave the queries as is. |
zregvart
added a commit
to zregvart/syndesis
that referenced
this issue
Jul 10, 2019
...vements Tries to improve performance of activity tracking persistence by: - adding a ordered index on the JSONDB table, in order to help with purging of old activity tracking entries during which we order the rows by the PATH column - explicitly trying to avoid locking when fetching the rows that need to be deleted from the database by adding `FOR KEY SHARE SKIP LOCKED` to the SELECT statement that's fetching the rows to be deleted Fixes syndesisio#6052
backporting bot
pushed a commit
that referenced
this issue
Jul 10, 2019
...vements Tries to improve performance of activity tracking persistence by: - adding a ordered index on the JSONDB table, in order to help with purging of old activity tracking entries during which we order the rows by the PATH column - explicitly trying to avoid locking when fetching the rows that need to be deleted from the database by adding `FOR KEY SHARE SKIP LOCKED` to the SELECT statement that's fetching the rows to be deleted Fixes #6052
|
@mmelko Did you get a chance to verify of the improvement help (at least to a degree that we can lower the priority of this ticket?) |
|
Last test @mmelko performed was on I've performed this maintenance on the installation: # check to see if there are dead tuples (not vacuumed)
$ oc exec -c postgresql $(oc get pod -l 'syndesis.io/component=syndesis-db' --no-headers=true -o=custom-columns=x:.metadata.name) -- bash -c "echo SELECT schemaname, relname, n_live_tup, n_dead_tup, last_autovacuum FROM pg_stat_all_tables WHERE relname = \'jsondb\'|psql -U syndesis"
schemaname | relname | n_live_tup | n_dead_tup | last_autovacuum
------------+---------+------------+------------+-------------------------------
public | jsondb | 26893 | 491210 | 2019-07-17 09:26:51.264029+00
(1 row)
# since there are 491210 dead tuples, performed the following
# scaled down the server
$ oc scale --replicas=0 dc syndesis-server
# terminated all running connections
$ oc exec -c postgresql $(oc get pod -l 'syndesis.io/component=syndesis-db' --no-headers=true -o=custom-columns=x:.metadata.name) -- bash -c "echo SELECT pg_terminate_backend\(a.pid\) FROM pg_locks l join pg_stat_activity a ON a.pid = l.pid WHERE l.mode = \'ExclusiveLock\' AND a.usename = \'syndesis\'|psql -U syndesis"
FATAL: terminating connection due to administrator command
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
connection to server was lost
command terminated with exit code 2
# the above FATAL output is expected the statement also closes the connection psql is using
# executed `VACUUM FULL ANALYSE`
$ oc exec -c postgresql $(oc get pod -l 'syndesis.io/component=syndesis-db' --no-headers=true -o=custom-columns=x:.metadata.name) -- bash -c "echo VACUUM FULL ANALYSE jsondb|psql -U syndesis"
VACUUM
# scaled up server
$ oc scale --replicas=1 dc syndesis-serverWith this the activities are being fetched again. We can wait another day and see if this preventative maintenance is needed again tomorrow or it somehow fixes the issue, as it has, perplexingly/seemingly , fixed on this issue on an instance running If this indeed seems to help, perhaps we can lower the priority of this as there is a possible workaround: performing the above maintenance. |
|
@mmelko do you think we can lower the priority with the maintenance steps outlined in #6052 (comment)? My argument is that if this can be accepted as a workaround and we document it as a known limitation then we can address this issue in the activity tracking redesign planned for the next release. |
|
@zregvart I'd rather keep this as p0. I'd still to document it as know limitation. But compared to other p1's this is still more serious and will have bigger impact on some potential enterprise deployments. |
|
@zregvart Can you work with @TovaCohen on the release notes? |
|
Tracked downstream: https://issues.jboss.org/browse/FUSEDOC-3376 |
|
Migrated to https://issues.jboss.org/browse/ENTESB-11407 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
See also https://issues.jboss.org/browse/ENTESB-11407
This is a...
Description
After putting some small load (~3 messages /s ) for couple of hours no activities are listed in integrations. I hit this also on i-dev cluster and openshift pro.
GET https://syndesis-test.b6ff.rh-idev.openshiftapps.com/api/v1/activity/integrations/i-LilthRWl7ZNNJJhTBQNzresults in504 Gateway timeout. Looks like connection to DB is really slow.There is
syndesis-testproject on i-dev cluster that can be used for investigation.The text was updated successfully, but these errors were encountered: