Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Twitter refactor data #3348

Merged
merged 14 commits into from Oct 24, 2018
Merged

Twitter refactor data #3348

merged 14 commits into from Oct 24, 2018

Conversation

rowasc
Copy link
Contributor

@rowasc rowasc commented Oct 22, 2018

This pull request makes the following changes:

Attempts to solve Twitter compliance issues

  • Tweet content and related data that we had from before we fixed this issue should be removed

    • For posts where the posts.content field matched a messages.message that was created from a tweet, replaced the message with the tweet's URL.
    • For posts where the posts.title field matched a messages.message that was created from a tweet, replaced the title with 'From Twitter on {date}'`
    • For posts that had location data extracted from Twitter, removed the location data
    • Removing tweet's contents from the messages database after the previous steps are done.
  • Twitter Usernames should not be saved

    • Removing contact usernames that we stored from twitter
    • For future tweets, saving the tweet user ID, instead of tweet username
  • New tweets that we extract with php artisan datasource:incoming

    • Do not save the tweet's content. Instead, use 'From Twitter on {date}'` as the title of the post created and the Tweet URL as the content.
    • Do not save location data from Twitter
  • Exports:

    • Add the data_source_message_id field to the CSV exports to be able to extract the tweet id
    • Do not export content for unstructured posts from Twitter.

Test checklist:

  • [ ]

  • I certify that I ran my checklist

Fixes ushahidi/platform# .

Ping @ushahidi/platform

@coveralls
Copy link

coveralls commented Oct 23, 2018

Coverage Status

Coverage decreased (-0.1%) to 26.087% when pulling 517fabc on twitter-refactor-data into c94854b on develop.

public function change()
{
$this->table('contacts')
->changeColumn('contact', 'string', ['null' => true])
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ugh no this should just be the opposite, null out the contact in message , and keep contact null constraint .

@rowasc rowasc requested a review from willdoran October 23, 2018 20:55
@willdoran willdoran merged commit 7c58400 into develop Oct 24, 2018
@rowasc rowasc deleted the twitter-refactor-data branch September 28, 2019 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants