Throughout this project, I’ve seen that data and the ways we can deal with it are unpredictable. I began this project assuming that my main page would show a full dataset, similar to some news applications that we have looked at and thinking that I would use a lot more Artificial Intelligence than I ended up working with in my final project.
One of the things that I wish I had more time and capacity (mental, emotional, etc) to do is make the doctor pages more detailed. In the end, I have a page that shows the significant attributes, including the case numbers, alerts, type of doctor and license number. However, I couldn’t show the status of the licenses as they weren’t clear. Additionally, the best I could do with the alerts was redirect them over to the PDF that contained more information. If I had more time, I definitely would have made summaries for each alert that details the accusations and resulting actions taken.
On the positive side, I was able to create functional searches from the main page for doctors and a page for the full list of alerts that would allow people to see all of the doctors and click to redirect to their pages. Additionally, I made the text searchable for the documents, meaning that a person could look up terms like “assaulted” or “circuit court” and see what reports show up. Unfortunately, the OCR is a little bit unreliable and the text search isn’t the most refined. However, there is a lot of important information that can be gleaned from these searches. I also made sure that the searches would be clear about which search was happening (doctor or text based on the URL) and would display a message if there were no alerts found.
Another thing that I opted out of doing was creating graphics to go on the front page. I think I wanted to do too much with this news application within the time allotted and found that displaying important information was more important than making it pretty or making it look like something that would be published by ProPublica. I was able to make it a little bit pretty and engaging to look at while keeping it relatively simple. This was similar for all the pages I created, not just the index page. I tried to maintain a similar style throughout the pages, which made my life easier and made the project look more cohesive.
The biggest issues I ran into were with the database querying and inserting new information and tables into the database. It took me some time to figure out how to clean and insert new data without messing up the old data. I ran into an issue where I would add on data instead of overwriting it, which I solved by deleting the table before rebuilding the dataset. I also ran into issues with the connections between my tables, where the primary key on the alerts table was being set to the filename instead of the ID. I was able to work through this by using ChatGPT and sqlite-utils documentation. It allowed me to create a connection with the cases table, which was integral to running the peewee queries.
The entire project can be deployed within codespaces and could probably be frozen and then redeployed or updated. I don't have strong opinions on how this needs to be maintained, but would recommend doing something that will either check when new records come in and update the system based on that or run periodically. All the data is stored in codespaces, which may become a problem. I think the best way to deal with this is to preserve the text documents in the codespace and delete the intermediary image files after they are processed. The PDFs are good to keep because we don’t know if they will be removed by the state or not. Users can see a snippet of the full dataset and have the option to contact me or access the github repository if they want to access the full database.
There isn’t a significant point where I would say this app has to die. I think it all depends on how the state develops their system and what kinds of stories come out of this data (if any). The data should outlive the app, likely stored in a datasette or github repository that continually pulls through GitHub Actions. Overall, I’m happy with the product that I have created. It does what I want it to and, more importantly, what I need it to. There is a lot I could have added, but I’ve done enough to create a useful product.