Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create "Core" Table Once and Then Never Again #15

Closed
alexdanilowicz opened this issue Nov 26, 2020 · 3 comments
Closed

Create "Core" Table Once and Then Never Again #15

alexdanilowicz opened this issue Nov 26, 2020 · 3 comments
Assignees

Comments

@alexdanilowicz
Copy link
Collaborator

alexdanilowicz commented Nov 26, 2020

We only need to create the SPLIT_TEXT_TABLE once and then never again. This should be a "Core" table that is created on initialize. This will greatly improve performance because then the WordCount and EmojiCount (and I imagine a word by day chart) tables can just read from this core table, which is created only once.

The core table could also already exclude stopWords, unicode characters, etc, so we don't have to run those WHERE clauses multiple times.

@alexdanilowicz
Copy link
Collaborator Author

As part of this, we should also have a REGEX on the core table to get rid of punctuation and contractions.

@alexdanilowicz
Copy link
Collaborator Author

Also, do we even need a top_friends_table core table? Could we just query directly every time? Querying directly might make filtering easier. Filters we probably want: is_from_me, group_message, person, date.

@alexdanilowicz
Copy link
Collaborator Author

As part of this ticket, I think I'll change it so that there is no concept of a Emojis, Word, or Friends table. Instead there are just core tables and we query off of those core tables every time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant