New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Normalize Events and Histories #213

Closed
mfenner opened this Issue Jan 2, 2015 · 0 comments

Comments

Projects
None yet
1 participant
@mfenner
Member

mfenner commented Jan 2, 2015

This will make it easier to properly display events associated with an article, but also to slice and dice our data as needed for reporting. Normalizing events and histories also makes the push API easier to use, as there is a clear data model behind it. Another important consequence is that we no longer need CouchDB and can store all data in MySQL.

  • store events as individual database rows
  • using the works table, and the events table (see above) to describe the relationship (e.g. Cites, Reviews)
  • normalize events using CSL (Citation Style Language) for easier display in aggregate form and easier import/export
  • store raw json response in separate responses table with 1-1 relationship to works table (for faster queries of the works table)
  • store counts over time in two tables: months and days. These tables are used both for sources that have events (e.g. twitter), and sources that don't (e.g. counter).
  • event counts are aggregated in the retrieval_statuses table
  • categories (e.g. html, pdf, comments, likes) are stored in separate fields in the retrieval_statuses, months and days tables to allow sorting and filtering by category

@mfenner mfenner added the roadmap label Jan 2, 2015

@mfenner mfenner self-assigned this Jan 2, 2015

mfenner pushed a commit that referenced this issue Jan 16, 2015

mfenner pushed a commit that referenced this issue Jan 16, 2015

mfenner pushed a commit that referenced this issue Jan 16, 2015

mfenner pushed a commit that referenced this issue Jan 16, 2015

mfenner pushed a commit that referenced this issue Jan 16, 2015

mfenner pushed a commit that referenced this issue Jan 16, 2015

mfenner pushed a commit that referenced this issue Jan 16, 2015

mfenner pushed a commit that referenced this issue Jan 16, 2015

mfenner pushed a commit that referenced this issue Jan 16, 2015

mfenner pushed a commit that referenced this issue Jan 16, 2015

mfenner pushed a commit that referenced this issue Jan 16, 2015

mfenner pushed a commit that referenced this issue Feb 5, 2015

mfenner pushed a commit that referenced this issue Mar 23, 2015

mfenner pushed a commit that referenced this issue Mar 23, 2015

mfenner pushed a commit that referenced this issue Mar 23, 2015

mfenner pushed a commit that referenced this issue Mar 23, 2015

mfenner pushed a commit that referenced this issue Mar 23, 2015

mfenner pushed a commit that referenced this issue Mar 23, 2015

mfenner pushed a commit that referenced this issue Mar 24, 2015

mfenner pushed a commit that referenced this issue Mar 24, 2015

mfenner pushed a commit that referenced this issue Mar 24, 2015

mfenner pushed a commit that referenced this issue Mar 24, 2015

mfenner pushed a commit that referenced this issue Mar 24, 2015

mfenner pushed a commit that referenced this issue Mar 24, 2015

mfenner pushed a commit that referenced this issue Mar 24, 2015

mfenner pushed a commit that referenced this issue Mar 24, 2015

mfenner pushed a commit that referenced this issue Apr 17, 2015

mfenner pushed a commit that referenced this issue Apr 17, 2015

mfenner pushed a commit that referenced this issue Apr 17, 2015

mfenner pushed a commit that referenced this issue Apr 17, 2015

mfenner pushed a commit that referenced this issue Apr 18, 2015

mfenner pushed a commit that referenced this issue Apr 19, 2015

mfenner pushed a commit that referenced this issue Apr 30, 2015

mfenner pushed a commit that referenced this issue May 2, 2015

mfenner pushed a commit that referenced this issue May 3, 2015

mfenner pushed a commit that referenced this issue May 4, 2015

mfenner pushed a commit that referenced this issue May 4, 2015

mfenner pushed a commit that referenced this issue May 4, 2015

mfenner pushed a commit that referenced this issue May 4, 2015

mfenner pushed a commit that referenced this issue May 4, 2015

mfenner pushed a commit that referenced this issue May 4, 2015

mfenner pushed a commit that referenced this issue May 5, 2015

mfenner pushed a commit that referenced this issue May 5, 2015

mfenner pushed a commit that referenced this issue May 5, 2015

mfenner pushed a commit that referenced this issue May 14, 2015

mfenner pushed a commit that referenced this issue May 15, 2015

mfenner pushed a commit that referenced this issue May 15, 2015

@mfenner mfenner closed this Jun 14, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment