New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add purge_history.sh and purge_remote_media.sh scripts #4155

Merged
merged 1 commit into from Nov 7, 2018

Conversation

Projects
None yet
3 participants
@rubo77
Contributor

rubo77 commented Nov 6, 2018

add purge_history.sh and purge_remote_media.sh scripts to contrib/scripts/ and link them in purge_history_api.rst and purge_remote_media_api.rst

Signed-off-by: Ruben Barkow github@r.z11.de

replaces #1034

@rubo77 rubo77 force-pushed the rubo77:purge-api branch from 43a78de to a8916ff Nov 6, 2018

@rubo77 rubo77 force-pushed the rubo77:purge-api branch 2 times, most recently from 1460796 to 0aacc72 Nov 6, 2018

@richvdh

please can you (a) remove the changes to the docs (b) put the scripts in their own directory with a readme

@rubo77 rubo77 force-pushed the rubo77:purge-api branch 4 times, most recently from c0eb08b to 699bf4e Nov 6, 2018

@4nd3r

This comment has been minimized.

Contributor

4nd3r commented Nov 6, 2018

can you please make the scripts pass shellcheck?

$ shellcheck purge_history.sh 

In purge_history.sh line 91:
POSTDATA='{"delete_local_events":"true"}' # this will really delete local events, so the messages in the room really disappear unless they are restored by remote federation
         ^-- SC2089: Quotes/backslashes will be treated literally. Use an array.


In purge_history.sh line 94:
    echo "########################################### "$(date)" ################# "
                                                       ^-- SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                       ^-- SC2046: Quote this to prevent word splitting.


In purge_history.sh line 108:
      OUT=$(curl -s -d $POSTDATA -X POST $DOMAIN':8008/_matrix/client/r0/admin/purge_history/'$ROOM'/'$EVENT_ID'/?access_token='$TOKEN)
                       ^-- SC2090: Quotes/backslashes in this variable will not be respected.
                                                                                              ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                                      ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                                                                ^-- SC2086: Double quote to prevent globbing and word splitting.


In purge_history.sh line 109:
      PURGE_ID=$(echo $OUT |grep purge_id|cut -d'"' -f4 )
                      ^-- SC2086: Double quote to prevent globbing and word splitting.


In purge_history.sh line 116:
          STATUS=$(curl -s GET $DOMAIN':8008/_matrix/client/r0/admin/purge_history_status/'$PURGE_ID'?access_token='$TOKEN |grep status|cut -d'"' -f4)
                                                                                           ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                                                    ^-- SC2086: Double quote to prevent globbing and word splitting.


In purge_history.sh line 119:
          SLEEP=$(($SLEEP + 1))
                   ^-- SC2004: $/${} is unnecessary on arithmetic variables.
$ shellcheck purge_remote_media.sh 

In purge_remote_media.sh line 50:
curl -v POST '$DOMAIN:8008/_matrix/client/r0/admin/purge_media_cache/?before_ts='$UNIX_TIMESTAMP'&access_token='$TOKEN
             ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that.
                                                                                 ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                                                ^-- SC2086: Double quote to prevent globbing and word splitting.
Show resolved Hide resolved contrib/API/README.md Outdated
@rubo77

This comment has been minimized.

Contributor

rubo77 commented Nov 6, 2018

SC2089: Quotes/backslashes will be treated literally. Use an array.

using arrays everywhere would make the code reeaaally cluttered

@rubo77 rubo77 force-pushed the rubo77:purge-api branch 3 times, most recently from 35ee422 to d5d6877 Nov 6, 2018

@rubo77

This comment has been minimized.

Contributor

rubo77 commented Nov 6, 2018

where do I find a documentation about how to send an Authorization header? So I can change that in my scripts?

@richvdh

This comment has been minimized.

Member

richvdh commented Nov 7, 2018

where do I find a documentation about how to send an Authorization header?

https://matrix.org/docs/spec/client_server/r0.4.0.html#using-access-tokens

@richvdh richvdh dismissed their stale review Nov 7, 2018

fixed

add purge_history.sh and purge_remote_media.sh scripts to contrib/pur…
…ge_api/

Signed-off-by: Ruben Barkow <github@r.z11.de>

@rubo77 rubo77 force-pushed the rubo77:purge-api branch from d5d6877 to 2904d13 Nov 7, 2018

@rubo77

This comment has been minimized.

Contributor

rubo77 commented Nov 7, 2018

I refactored the authentification and added a database function

There are only two shellcheck warnings left:

$ shellcheck purge_history.sh 

In purge_history.sh line 87:
POSTDATA='{"delete_local_events":"true"}' # this will really delete local events, so the messages in the room really disappear unless they are restored by remote federation
         ^-- SC2089: Quotes/backslashes will be treated literally. Use an array.


In purge_history.sh line 107:
      OUT=$(curl --header "$AUTH" -s -d $POSTDATA POST "$API_URL/admin/purge_history/$ROOM/$EVENT_ID")
                                        ^-- SC2090: Quotes/backslashes in this variable will not be respected.

Those are irrelevant to security, because there are no external variables used.

@richvdh

This comment has been minimized.

Member

richvdh commented Nov 7, 2018

thanks!

@richvdh richvdh merged commit c8ba793 into matrix-org:develop Nov 7, 2018

5 checks passed

ci/circleci: sytestpy2merged Your tests passed on CircleCI!
Details
ci/circleci: sytestpy2postgresmerged Your tests passed on CircleCI!
Details
ci/circleci: sytestpy3merged Your tests passed on CircleCI!
Details
ci/circleci: sytestpy3postgresmerged Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

hawkowl added a commit that referenced this pull request Nov 19, 2018

Merge tag 'v0.33.9'
Features
--------

- Include flags to optionally add `m.login.terms` to the registration flow when consent tracking is enabled.
([\#4004](#4004), [\#4133](#4133),
[\#4142](#4142), [\#4184](#4184))
- Support for replacing rooms with new ones ([\#4091](#4091), [\#4099](#4099),
[\#4100](#4100), [\#4101](#4101))

Bugfixes
--------

- Fix exceptions when using the email mailer on Python 3. ([\#4095](#4095))
- Fix e2e key backup with more than 9 backup versions ([\#4113](#4113))
- Searches that request profile info now no longer fail with a 500. ([\#4122](#4122))
- fix return code of empty key backups ([\#4123](#4123))
- If the typing stream ID goes backwards (as on a worker when the master restarts), the worker's typing handler will no longer erroneously report rooms containing new
typing events. ([\#4127](#4127))
- Fix table lock of device_lists_remote_cache which could freeze the application ([\#4132](#4132))
- Fix exception when using state res v2 algorithm ([\#4135](#4135))
- Generating the user consent URI no longer fails on Python 3. ([\#4140](#4140),
[\#4163](#4163))
- Loading URL previews from the DB cache on Postgres will no longer cause Unicode type errors when responding to the request, and URL previews will no longer fail if
the remote server returns a Content-Type header with the chartype in quotes. ([\#4157](#4157))
- The hash_password script now works on Python 3. ([\#4161](#4161))
- Fix noop checks when updating device keys, reducing spurious device list update notifications. ([\#4164](#4164))

Deprecations and Removals
-------------------------

- The disused and un-specced identicon generator has been removed. ([\#4106](#4106))
- The obsolete and non-functional /pull federation endpoint has been removed. ([\#4118](#4118))
- The deprecated v1 key exchange endpoints have been removed. ([\#4119](#4119))
- Synapse will no longer fetch keys using the fallback deprecated v1 key exchange method and will now always use v2.
([\#4120](#4120))

Internal Changes
----------------

- Fix build of Docker image with docker-compose ([\#3778](#3778))
- Delete unreferenced state groups during history purge ([\#4006](#4006))
- The "Received rdata" log messages on workers is now logged at DEBUG, not INFO. ([\#4108](#4108))
- Reduce replication traffic for device lists ([\#4109](#4109))
- Fix `synapse_replication_tcp_protocol_*_commands` metric label to be full command name, rather than just the first character
([\#4110](#4110))
- Log some bits about room creation ([\#4121](#4121))
- Fix `tox` failure on old systems ([\#4124](#4124))
- Add STATE_V2_TEST room version ([\#4128](#4128))
- Clean up event accesses and tests ([\#4137](#4137))
- The default logging config will now set an explicit log file encoding of UTF-8. ([\#4138](#4138))
- Add helpers functions for getting prev and auth events of an event ([\#4139](#4139))
- Add some tests for the HTTP pusher. ([\#4149](#4149))
- add purge_history.sh and purge_remote_media.sh scripts to contrib/ ([\#4155](#4155))
- HTTP tests have been refactored to contain less boilerplate. ([\#4156](#4156))
- Drop incoming events from federation for unknown rooms ([\#4165](#4165))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment