New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove nuke-room-from-db.sh script #3888

Merged
merged 2 commits into from Sep 17, 2018

Conversation

Projects
None yet
4 participants
@richvdh
Member

richvdh commented Sep 17, 2018

The problem with this script is that it is largely untested, entirely
unmaintained, and running it is likely to make your synapse blow up in
exciting ways.

For example, it leaves a bunch of tables with dead values in it, like
event_to_state_groups.

Having it here sends a message that it is a supported part of
synapse, which is absolutely not the case.

richvdh added some commits Sep 17, 2018

Remove nuke-room-from-db.sh script
The problem with this script is that it is largely untested, entirely
unmaintained, and running it is likely to make your synapse blow up in
exciting ways.

For example, it leaves a bunch of tables with dead values in it, like
event_to_state_groups.

Having it here sends a message that it is a supported part of
synapse, which is absolutely not the case.
@jcgruenhage

This comment has been minimized.

Member

jcgruenhage commented Sep 17, 2018

I probably would not have run this in the past, if I had known how unsupported it is. On the other hand, in the cases when I used it, it got my synapse back up and running after rooms got messed up somehow.

@richvdh richvdh requested a review from matrix-org/synapse-core Sep 17, 2018

@hawkowl hawkowl merged commit c6363f7 into develop Sep 17, 2018

14 checks passed

Synapse Sytest Postgres (Commit) Build #7050 origin/rav/nuke_nuke_rooms succeeded in 6 min 31 sec
Details
Synapse Sytest Postgres (Merged PR) Build finished.
Details
Synapse Sytest SQLite (Commit) Build #7263 origin/rav/nuke_nuke_rooms succeeded in 3 min 41 sec
Details
Synapse Sytest SQLite (Merged PR) Build finished.
Details
ci/circleci: sytestpy2 Your tests passed on CircleCI!
Details
ci/circleci: sytestpy2merged Your tests passed on CircleCI!
Details
ci/circleci: sytestpy2postgres Your tests passed on CircleCI!
Details
ci/circleci: sytestpy2postgresmerged Your tests passed on CircleCI!
Details
ci/circleci: sytestpy3 Your tests passed on CircleCI!
Details
ci/circleci: sytestpy3merged Your tests passed on CircleCI!
Details
ci/circleci: sytestpy3postgres Your tests passed on CircleCI!
Details
ci/circleci: sytestpy3postgresmerged Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@hawkowl hawkowl deleted the rav/nuke_nuke_rooms branch Sep 20, 2018

hawkowl added a commit that referenced this pull request Sep 24, 2018

Merge tag 'v0.33.5'
Features
--------

- Python 3.5 and 3.6 support is now in beta.
([\#3576](#3576))
- Implement `event_format` filter param in `/sync`
([\#3790](#3790))
- Add synapse_admin_mau:registered_reserved_users metric to expose
number of real reaserved users
([\#3846](#3846))

Bugfixes
--------

- Remove connection ID for replication prometheus metrics, as it creates
a large number of new series.
([\#3788](#3788))
- guest users should not be part of mau total
([\#3800](#3800))
- Bump dependency on pyopenssl 16.x, to avoid incompatibility with
recent Twisted.
([\#3804](#3804))
- Fix existing room tags not coming down sync when joining a room
([\#3810](#3810))
- Fix jwt import check
([\#3824](#3824))
- fix VOIP crashes under Python 3 (#3821)
([\#3835](#3835))
- Fix manhole so that it works with latest openssh clients
([\#3841](#3841))
- Fix outbound requests occasionally wedging, which can result in
federation breaking between servers.
([\#3845](#3845))
- Show heroes if room name/canonical alias has been deleted
([\#3851](#3851))
- Fix handling of redacted events from federation
([\#3859](#3859))
-  ([\#3874](#3874))
- Mitigate outbound federation randomly becoming wedged
([\#3875](#3875))

Internal Changes
----------------

- CircleCI tests now run on the potential merge of a PR.
([\#3704](#3704))
- http/ is now ported to Python 3.
([\#3771](#3771))
- Improve human readable error messages for threepid
registration/account update
([\#3789](#3789))
- Make /sync slightly faster by avoiding needless copies
([\#3795](#3795))
- handlers/ is now ported to Python 3.
([\#3803](#3803))
- Limit the number of PDUs/EDUs per federation transaction
([\#3805](#3805))
- Only start postgres instance for postgres tests on Travis CI
([\#3806](#3806))
- tests/ is now ported to Python 3.
([\#3808](#3808))
- crypto/ is now ported to Python 3.
([\#3822](#3822))
- rest/ is now ported to Python 3.
([\#3823](#3823))
- add some logging for the keyring queue
([\#3826](#3826))
- speed up lazy loading by 2-3x
([\#3827](#3827))
- Improved Dockerfile to remove build requirements after building
reducing the image size.
([\#3834](#3834))
- Disable lazy loading for incremental syncs for now
([\#3840](#3840))
- federation/ is now ported to Python 3.
([\#3847](#3847))
- Log when we retry outbound requests
([\#3853](#3853))
- Removed some excess logging messages.
([\#3855](#3855))
- Speed up purge history for rooms that have been previously purged
([\#3856](#3856))
- Refactor some HTTP timeout code.
([\#3857](#3857))
- Fix running merged builds on CircleCI
([\#3858](#3858))
- Fix typo in replication stream exception.
([\#3860](#3860))
- Add in flight real time metrics for Measure blocks
([\#3871](#3871))
- Disable buffering and automatic retrying in treq requests to prevent
timeouts. ([\#3872](#3872))
- mention jemalloc in the README
([\#3877](#3877))
- Remove unmaintained "nuke-room-from-db.sh" script
([\#3888](#3888))
@rubo77

This comment has been minimized.

Contributor

rubo77 commented on e7b3b4d Oct 1, 2018

It would be easy to add event_to_state_groups to the list of tables there. I think event_id is the foreign key to another table, which?

could you not provide the script somewhere else then?

@rubo77

This comment has been minimized.

Contributor

rubo77 commented Oct 1, 2018

If i look at the database structure: #934

I think it is easy:

  1. delete all entries in event_to_state_groups that match entries.entry_id with the key event_to_state_groups.entry_id

  2. delete entries in event_to_state_groups

      SELECT * FROM event_to_state_groups s, events e WHERE s.event_id=e.event_id AND e.room_id like '%jGUBNvBzjGyQyKKMJV:matrix.org'
    

how do I delete only those entries now?

@richvdh

This comment has been minimized.

Member

richvdh commented Oct 1, 2018

none of that addresses the fact that we're not able to support and test it. I suggest you maintain your own sql script.

@rubo77

This comment has been minimized.

Contributor

rubo77 commented Oct 1, 2018

I will :)

Delete data concerning only the room would be

 # (untested!) 
 DELETE 
    FROM event_to_state_groups USING events
    WHERE event_to_state_groups.event_id = events.event_id 
    AND room_id = '$ROOMID'

I think it is much easyer and also deletes orphaned entries like this:

DELETE FROM event_to_state_groups WHERE event_id NOT IN (SELECT event_id FROM events)

am I right?

@richvdh what other tables have orphaned entries after using the old nuke script?

@rubo77

This comment has been minimized.

Contributor

rubo77 commented Oct 13, 2018

A more secure way to clear event_to_state_groups will be these commands:

-- check how many rows in event_to_state_groups

SELECT count(*) from event_to_state_groups;

-- create temporary TABLE event_to_state_groups_to_delete with event_ids that are missing in events TABLE

BEGIN;
SET enable_seqscan=off;
SELECT s.event_id INTO event_to_state_groups_to_delete FROM event_to_state_groups AS s 
LEFT JOIN events
ON s.event_id = events.event_id
WHERE events.event_id IS NULL;
COMMIT;

-- check how many rows will be deleted

SELECT count(*) from event_to_state_groups_to_delete;

(this can take some minutes if you have millions of events already)

shut synapse down to delete the entries

(not sure if shutdown is needed)

-- delete obsolete entries

DELETE FROM event_to_state_groups WHERE event_id IN (SELECT event_id FROM event_to_state_groups_to_delete);

start it up again

-- delete temporary TABLE event_to_state_groups_to_delete;

DROP TABLE event_to_state_groups_to_delete;

Do I understand it right, that event_to_state_groups only connects events with state_group on the same serer?
so If there are event_ids in event_to_state_groups that are not in events, they can be deleted?
And the same with the state_groups table: if there are event_ids in state_groups that are not in events, they can be deleted?

If so, then we would have to do the same deletion script with state_groups too?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment