Tweak Data Collection

[maxachis]

There are several possible ways we could tweak our data collection efforts to potentially be more efficacious for machine learning components. For example:

Add Bag-Of-Words data to Div Text Collection

A Bag-Of-Words model disregards word order and focuses solely on how many of which words are present. It would look something like this:

{"John":1,"likes":3,"to":2,"watch":2,"movies":2,"Mary":2,"too":1,"also":1,"football":1,"games":1};

This would enable us to balance data collection (especially for pages with more div content) by focusing on keywords rather than accounting for every single word sequence in the text. Presumably certain web pages for certain record types will be more likely to have certain keywords than others.

[maxachis]

I'm debating between whether to keep that data within this repository or within a separate repository. Here's my thinking:

This Repository

• Centralizes logic
• We can incorporate it into a lot of our existing logic, such as the tasks
• Synergizes well with HTML data
• We can host it easily within the same database
• Could potentially incorporate it into annotation

Separate Repository

• This data may not be useful for annotation
• Less overhead and easier to iterate on in a separate repository
• Easier to take into account our various sources that already exist
• Allows us to utilize data sources URLs without being too redundant.

At the moment, I'm leaning towards a separate repository, since I think it's primarily to be used for training at the moment, and we may not incorporate it into the larger workflow.

[maxachis]

I've begun working on this in https://github.com/Police-Data-Accessibility-Project/huggingface-trainer. To wit, I'm working on exploratory data analysis and data scraping, with rapid iterations.

[maxachis]

So quite a bit has happened on this since we last spoke, my dear issue.

The above repository huggingface-trainer has been rebranded to data-analysis, where I pull data and pull it apart and inspect the gooey insides to see what is best for running machine learning models.

Then I've created machine-learning-lab, which is where I've been experimenting with different machine learning models before determining which ones should be implemented in Source Collector -- and how.

One of the first things I've done is create a prototype bag-of-words Logistic Regression model for relevancy assessment, which has worked decently, producing 82% accuracy (albeit with precision of 98% on Relevant and 40% on Not-Relevant, meaning it's very good at identifying when things are relevant but sub-par at identifying when things are not relevant). That's good enough to start playing around with incorporating things into our workflow.

But it does mean we need to start thinking carefully about how we organize our logic now, for a few reasons:

• Machine Learning is considerably more resource-intensive than serving API requests and scraping HTML data. If we host that on the same location as the Source Collector, we may run into some issues in terms of timely responses for SC.
• Organizationally, the ML components are quite distinct from the SC components -- The Source Collector can collect sources and their relevant data and facilitate annotations, but the ML components extract information from that relevant data and put it into hungry models and spit out labels.

That leaves me leaning more towards having a separate application for ML components, with SC serving requests for machine-generated annotations to ML, and ML doing the grunt work and spitting back answers. ML could also be the location where we train the models, or we could train them on our own in local environments. At least right now, our models are not super computationally intensive (the current Logistic Regression model trained itself in less than a second), so hosting them in either location could be fine, but something to bear in mind in case we get more avaricious models. They could still be connected to the same database as SC (there's not necessarily a reason to have another separate database, at least not yet), but performing different functions and communicating with each other primarily through the API.

The details of all of this are subject to change by whim and fancy, so nothing's set in stone, but that's where I'm starting to think.

What do you think, @josh-chamberlain?

[josh-chamberlain]

@maxachis sorry about the delay. I don't believe we need an application for ML yet:

• Local training is indeed OK, but we have long wanted to train on DO, because it can be a lot for a local machine depending on the model: Deploy hugging face trainer #41
  • This may not be worth setting up until we get into some heavier models.
• I think the Inference API takes care of where to use the models

[maxachis]

@josh-chamberlain What do you think is remaining for this one?

[josh-chamberlain]

@maxachis IMO, unless we have a specific reason for tweaking the data collection further, we should close this. We can reopen it with a more specific goal in the future.