Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Reading Sheets Import and Process prototype #2523
Who is this PR for?
What problem does this PR fix?
Reading data from schools is kept in spreadsheets that must be manually moved for the data to reach Insights
What does this PR do?
Introduces Importer and Processor classes that will fetch sheets from a google drive folder and process the data so it can be added to Insights.
This assumes the sheets within a grade follow a consistent template, but the specific template can be flexible between grades.
Does this PR use tests to help verify we can deploy these changes quickly and confidently?
@kevinrobinson I just realized I wasn't rerunning the tests when I reopened this, so I don't think these failures are from this branch.
That aside, when you have a chance, would you take a look and see if this looks suitable for the reading sheets import?
@edavidsonsawyer This is awesome progress!
I'm assuming this is built towards the format in
First, let's update the format to use a new sheet called
Second, let's remove any computed values (eg, how many levels of growth). The new format should discourage this more explicitly, but either way if there is anything that's computed we don't need to import it as we can just re-compute ourselves. There are sometimes subtle differences here, and removing that will just simplify a bit.
Third, there are validations on the
Also, I think it's great that this PR punts the "import" step for now, which will have to handle the "sync" process where data points change over time. We can do that separately, so focusing on the processor class seems like great scoping.
@edavidsonsawyer re test failures, this is what I see here: https://travis-ci.org/studentinsights/studentinsights/jobs/561886950#L3144
So these are just linter failures, you can fix most of the minor things like whitespace by running
Here's our notes talking through the cases for update. I think we can do the "importer" work without doing this first, and just add the data points we find.
That won't handle if we run it twice - we'll get extra data points, but that's okay as a good first step.
After that I might be:
Those cases might be as simple as: "delete everything that ever came from this form." But there might be important edge cases we want to handle, like the teacher deleting a row because a student changed homeroom. And we'll have to translate those to simple guidance for educators too.
Alternately, we might just want to say "if the student is in the sheet, update their records for that time period in the sheet."
We don't have to handle all those edge cases, simple is better! We could say "if you do X, Y will happen" and "here are two edge cases that would break, but we're not handling now."
kevinrobinson left a comment
Hey @edavidsonsawyer! This looks like good progress towards step 1. I left comments inline, let me know if you want to chat more. I'll still plan on spending some time Friday morning checking in with where we're at and then seeing if I can start trying to deploy step 1 incrementally.
Also, I tried to fix the flaky tests on master, so merging should resolve that. Thanks for working around the noise and feel free to ping if it comes up again.