Create a data model for the info required to collect "metadata" from FOIA offices #16
Comments
|
What we have so far is a FOIA Office content type (screenshot of configured fields is below) which corresponds to what was referred to as a component above. The FOIA Office is associated to a Department/Agency in many-to-one manner. The FOIA Office also has an entity reference field (called the Request Submission Form) to a webform entity. This is a one-to-one relationship.
The Department/Agency is a just a taxonomy term with a name, description, and an image upload for the agency seal. Additional fields can be added if needed. An example of a Department/Agency as rendered without any styling work is shown below.
|
|
Here's an example of the USDA OIG component. This is pulled from {
"Office of the Inspector General, Department of Agriculture ( USDA )" : {
"name" : "Office of the Inspector General",
"fax" : "202-690-6305",
"emails" : [
"alison.decker@oig.usda.gov"
],
"summary" : {
"abbreviation" : "USDA",
"description" : "The Department of Agriculture provides leadership on food, agriculture, natural resources, and related issues.",
"name" : "Department of Agriculture",
"usa_id" : "49015"
},
"address" : {
"street" : "1400 Independence Avenue, SW",
"city" : "Washington",
"address_lines" : [
"Alison Decker",
"FOIA Officer",
"Room 441-E Whitten Bldg."
],
"state" : "DC",
"zip" : "20250"
},
"phone" : "202-720-5677",
"request_time_stats" : {
"2011" : {
"complex_highest_days" : "489",
"complex_average_days" : "47",
"simple_highest_days" : "305",
"complex_median_days" : "20",
"expedited_processing_median_days" : "0",
"expedited_processing_average_days" : "0",
"simple_average_days" : "22",
"expedited_processing_lowest_days" : "0",
"simple_lowest_days" : "2",
"complex_lowest_days" : "1",
"expedited_processing_highest_days" : "0",
"simple_median_days" : "16"
},
"2010" : {
"complex_median_days" : "100",
"expedited_processing_median_days" : "0",
"expedited_processing_average_days" : "0",
"simple_average_days" : "14",
"expedited_processing_lowest_days" : "0",
"expedited_processing_highest_days" : "0",
"simple_lowest_days" : "2",
"complex_lowest_days" : "41",
"simple_median_days" : "8",
"complex_highest_days" : "778",
"complex_average_days" : "156",
"simple_highest_days" : "34"
},
"2014" : {
"simple_median_days" : "61",
"complex_lowest_days" : "42",
"simple_lowest_days" : "2",
"simple_average_days" : "85",
"complex_median_days" : "141",
"simple_highest_days" : "413",
"complex_average_days" : "214",
"complex_highest_days" : "682"
},
"2013" : {
"simple_median_days" : "22",
"simple_average_days" : "25",
"simple_lowest_days" : "6",
"complex_lowest_days" : "33",
"complex_median_days" : "42",
"complex_average_days" : "46",
"simple_highest_days" : "79",
"complex_highest_days" : "283"
},
"2012" : {
"complex_average_days" : "68",
"simple_highest_days" : "30",
"complex_highest_days" : "133",
"simple_median_days" : "13",
"simple_average_days" : "13",
"complex_lowest_days" : "31",
"simple_lowest_days" : "1",
"complex_median_days" : "64"
}
},
"top_level" : false
}
}The There are new fields for submissions methods and the request form that @bsweger and I will need to articulate here. Look for a few comments on this ticket as we aggregate. |
|
A note on the screenshot you've posted @ba66e77. The fields for expedited processing and fee waiver instructions feel like they belong on the request form model, not at this office level. I don't have strong feelings about the model per se, but the presentation context of those fields to the agency officers feels like it belongs to the form, not the agency itself. |
|
Here's another example. Note that keys like {
"National Finance Center, Department of Agriculture ( USDA )" : {
"name" : "National Finance Center",
"reading_rooms" : [
[
"Click here to access the NFC Electronic Reading Room",
"https://www.nfc.usda.gov/FOIA/foiaereading.html"
]
],
"summary" : {
"usa_id" : "49015",
"name" : "Department of Agriculture",
"description" : "The Department of Agriculture provides leadership on food, agriculture, natural resources, and related issues.",
"abbreviation" : "USDA"
},
"emails" : [
"cheri.alsobrook@nfc.usda.gov"
],
"fax" : "504-426-9706",
"service_center" : {
"phone" : [
"504-426-0300"
],
"name" : "Sandy Francois"
},
"website" : "https://www.nfc.usda.gov/FOIA/FOIA_home.html",
"address" : {
"city" : "New Orleans",
"street" : "P.O. Box 60000",
"zip" : "70160",
"state" : "LA",
"address_lines" : [
"Cheri Alsobrook",
"FOIA Officer"
]
},
"public_liaison" : {
"name" : "Shri Alsobrook, Sandy Francois",
"phone" : [
"504-426-0327"
]
},
"request_form" : "https://efoia-pal.usda.gov/palMain.aspx",
"top_level" : false,
"foia_officer" : {
"name" : "John Hemstreet",
"phone" : [
"504-426-0168"
]
},
"phone" : "504-426-0374"
}
} |
|
The things we don't want from https://github.com/18F/2015-foia/blob/master/contacts/data/USDA.yaml are the |
|
@pkarman we debated the placement of the fee waver and processing fields. In the end, we put them on the office because we didn't see any instances where the form was presented without the office context and putting them on the office made the content editing UI a little nicer. If that first observation is wrong or there are other factors we're not aware of then we can change the placement. |
|
@pkarman and @ba66e77 ... am compiling a single field list from the various links here and have a few questions to make sure I'm capturing things correctly.
|
|
@ba66e77 Following up on the fee waiver and processing fields, am I thinking correctly that these will vary based on the specific FOIA requests? If that's true (and please let me know if it's not!), these do seem like attributes that should be defined as part of the request schema. I don't have strong UI feelings, since I haven't participating in any testing to date. But we're just talking logical model/schema here? |
yes. IIUC, that's represented in the screenshot by the "entity relationship" to the "Agency/Department". I.e. an office can have a parent office.
It goes one level deep.
A single request goes to a single office (component). If a requester needed to get records from multiple components within a single agency, they would need to file one request for each component.
I like flexibility, but I like to use it to discover patterns that the model doesn't yet have explicit slots to represent. So if folks have been using |
|
@bsweger The fee waiver and expedited processing instructions are static text that will be displayed on the request form, to help the requester understand how much a FOIA request might cost. It's help text. So it doesn't change from request to request, but does change from office to office (possibly) and from agency to agency (likely). |
|
@pkarman aaaah---thanks for the info re: fee waiver/processing....missed that they are static instructions. |
|
@pkarman Thanks for the info re: the hierarchies.
I phrased my original question poorly...let me try again. For an agency like USDA, would we expect that USDA itself (i.e., the top level of a decentralized agency) receive FOIA requests? Put another way, can a requester choose to route a request to the top level of a decentralized agency? Or is the choice limited only to the 2nd tier departments? I'm asking because in the sample USDA metadata, there's a series of departments with all of the fields needed to created a request. Would we expect to have this information defined for USDA proper as well? Does this question make sense? |
|
It does make sense @bsweger thanks for clarifying. Each FOIA office can receive requests. So the main office in a decentralized agency can receive requests like any of their components. |
|
Ok, here's a first pass at a complete list of fields. The names aren't meant to represent what we should use; I added the prefixes to better show, in this flattened format, the 1:n relationships between a department and some of the descriptive names needed to render the form. This exercise raised a few questions about data structures/models that we'll use to store the info, but perhaps I'm jumping ahead. Would love to use some of our sync time today to go through this and better understand how schemas are represented in the current Drupal implementation and how we can capture their change history, since we'll no doubt have to iterate as we get feedback. Also @pkarman great point about the stats; agree that we need some thinking there too.
|
|
@bsweger, it seems like there are two conceptual entities here that the list is trying to represent as a single entity. To my thinking, Agency is an entity and Office/component is an entity, so we should be developing two schemas here. The Agency would contain an array of Office entities to reflect the one-to-many, has-a relationship between the Agency and its Offices. The yml files I've seen for agencies reflect the structure you've outlined here, but what do you think of something structured more like the below? |
|
@ba66e77 Yes, for sure. The table above is a flat list of all the fields I think we'll need. Modeling the relationships seems like the next logical step. In addition to the one to many between department and offices, there are also one to many relationships between and office and submission format, etc. Would be helpful to schedule some time with @pkarman or use a meeting already on the calendar to flush out some acceptance criteria for this story? When we say "create an agency data model" are we talking about the json/yaml representation? A logical relational database model? |
|
I think what we're talking about is a pseudo-code representation. YAML, JSON, bullet-points, whatever. So long as we don't end up conflating defining the model with how the model will be communicated between the front end and the back end. |
|
@ba66e77 good point about not conflating how the data is stored vs how it is serialized. Thanks @bsweger for putting together all the marbles into one list. I agree; some talking time to make sure we're all aligned about how to represent the relationships seems fine. I agree that my mental model so far has been Agency and Offices, where an Agency might have one Office (centralized) or multiple (decentralized). |
|
I also realize that we have used |
|
As I'm putting these fields into into a more hierarchical format, I still have questions about I know I'm beating a dead horse here, but if an office is the entity that receives a FOIA request and a FOIA request can also be made at the department/agency level, even in a decentralized agency, would we expect to see main agencies listed as offices? For example: Centralized Agency Decentralized Agency |
|
You can't make a request at the agency level. I.e. in a decentralized model, you can't request records from across the entire agency (all offices) by submitting to a single main office. It's really up to the agency how they organize/name themselves, but each office must have a FOIA officer, by law, so it's the individual FOIA office to which you must send a request. So in your example, yes, USDA might be listed twice, once as the name of the agency and once as the name of the main office. The naming of the office is up to them, and may or not be misleading. Note I have tried to use the word |
|
@pkarman you're right--there are way too many words for agencies, bureaus, etc. I'll use @ba66e77 spoke with Peter b/c I needed a synchronous convo to clarify this point about the main FOIA office in a decentralized agency. It turns out that I was thinking of an agency and it's main FOIA office as one in the same. Once Peter set me straight on that, things became much clearer. Next stab at the fields forthcoming... |
|
@pkarman @ba66e77 Ok, here's an "unflattened" list of the complete universe of attributes, as I understand them right now. I'm still unable to suss out some of the relationships from reading through the various YAML/JSON examples, so still have a few questions:
Wanted to ask the above explicitly because some of the JSON/YAML artifacts indicate that the answer can be yes, but wouldn't that be pretty confusing from a requester perspective? The format below assumes that Also pinging @adborden for his 👀 , since he was involved in this during discovery. Look forward to hearing what I missed, etc. |
|
This is great @bsweger thanks. I think the answer to all your questions is "yes" but this feels like a great topic to talk with OIP about to clarify. |
|
Ok, will ask OIP about those questions. Had a very high-level convo with Matt to your earlier point, @pkarman:
If I understood OIP correctly:
It sounds like previously there wasn't necessarily a use for the more granular office-level stats once they'd been used to derive the agency-level info. But now there are two additional things to consider:
Which is just to say that you're right--we need more thinking here. I'll leave the |
|
@bsweger conversation from today re: foia officer sounds like we should keep it in as is for now. I don't think their role is restricted to paper submissions. |
bsweger
changed the title
|
@adborden Since you were involved in creating the sample .yaml files in discovery, was wondering if you have any insight into these fields from the draft above: Was the intent that |
|
@malikkotob Thanks for the pings that remind me to update this issue. To be more explicit about where things stand:
In other words, it sounds like there might be 3 over-arching entities: agencies, offices, and the pool of common FOIA questions. Does that sound right? Looks like you've opened new stories for getting this info into Drupal. Does that mean the fourth acceptance criteria here is no longer applicable? |
|
Oh, I forgot to add the caveat that I'd expect schema changes and updates to be part of the process. Particularly around the 90% of questions, since the research/design folks are actively testing those. If they learn that the current wording needs updated or that the office metadata schema needs to be expanded to cover things like adding an ability to customize the ordering of the common questions, we'll revisit. But I think we're arrived at a good starting point. |
|
Hi @bsweger! I was about to comment to say just that. I went ahead and split the implementation of those over-arching entities into their own actionable issues, so yes - we can go ahead and remove that fourth criteria from this ticket. To split hairs with regards to the three entities, the entirety of the request form that an office uses to accept FOIA requests (the 90% that is common, plus the 10% that is unique) is the third entity we will capture in Drupal, as opposed to only the common 90%. Also, no issue with with changes being part of the process. I think as those come up we can adjust in Drupal as needed, using issues in Github to track those changes. I'll look over the 90% and the structure from your comment and let you know if I have any questions/comments. Nice work getting this along! |
|
@malikkotob Great, thanks. I'll remove the fourth acceptance criteria. Also, I moved this story into Review/QA on the board, since you'll be reviewing when you get back to the office. |
|
I spoke to @bsweger offline about #16 (comment) but want to make sure to document the answer for everyone else:
Yes.
Yes, but we can negotiate how this is implemented. The intent of the metadata file from the recommendations point of view is that the metadata file contains everything you would need to generate a request form for an agency and component. That implies that the 90% fields would be in the metadata file. It doesn't say anything about whether the 90% fields should be exposed in the backstage to FOIA officers. From a data modeling perspective, we might want to capture the 90% fields as a separate entity from the request form entity. As mentioned in #16 (comment), there might be some research that would require us to have some customization of the 90% fields and that data would land in the request form entity. As an example, in our experience map/backstage demo, there was a possibility that a FOIA officer could choose one of three options to render the name fields (single field, first/last, or prefix/first/last/suffix). That kind of thing would be captured in the request form entity. |
|
@malikkotob assigning to you for acceptance. Do you have what you need to move forward? Can we call this done? (part of our sprint review) |
|
@RyanSibley reminded me that we weren't planning on putting the annually reported request processing data in the metadata file and we would not collect it from the agency backstage. Agencies already provide this information through annual and quarterly reporting processes so it does not make sense to burden agencies with that duplication. While this data is not intended to be provided in the metadata file, the current thinking is that we wanted it to display to requesters in some form on the Portal. It's probably okay to punt on this until we know more about how we want to use that historical data. It makes sense to store that data as we have modeled, but there's a missing piece about how that data is updated in the future and how it would be delivered to the front end. I've opened #46 to capture that conversation. |
|
Since @malikkotob is out for a couple days, I'll take review of this one |
|
Coalescing from above, the attributes of each entity are: Data StructureAgency
example Agency Component (formerly Office)
Person
QuestionsAgency
Agency Component
|
Fields for the Request Submission Form Template (aka, the 90%)
Questions
|
|
@ba66e77 Thanks for the thoughtful questions. Adding my perspective to the your feedback in this comment re: agency and agency component. My understanding is that we want a user, be it a developer working on the portal front-end or a 3rd party, to be able to:
The sample data we're going back and forth on in this thread represents what a person requesting agency component metadata would see. Thus, I see things like request form fields being part of the agency component metadata. Yes, the form fields are part of a specific FOIA request, but a list of form fields--required and otherwise--is also descriptive information about an agency component's FOIA process and belongs in the metadata. How this is all modeled in the backend is an implementation detail in the purview of you and the Drupal team. But I would expect an API call to say, I think part of the back and forth here is that the data model part of the issue title means different things to different people, so we're all coming in from a different perspective. To that end, I've added a story to create a higher-level dataflow diagram of this process. I don't think it needs to block this, but I would find it personally helpful to look at the metadata/form process in the context of something like the journey map we saw last week. |
|
Trying to articulate more of what I think we're after here. We're not really in the business of building a FOIA request form, as I understand it. For this piece of the project, we're building a service that describes how to a build a FOIA request form. Presumably, we ourselves will be users of this service as we build the portal. So if there's some kind of question list that's a separate database entity, that's a backend design call. But from the perspective of using this agency component metadata service, there's not separate "form" per se, if I understand the discovery team's suggested approach. |
|
Right, @bsweger, I'm hearing three areas of discussion in this issue:
My understanding is this issue is really about number 1 only, although we've touched on all the other aspects too. The other points might be covered in other issues already but it might be helpful to break them out into explicit stories if we think that is needed. I think the diagram of the data flow will also help to clarify the big picture. |
|
Quoting Barrett Smith (2017-08-07 05:47:14)
## Questions
### Agency
1. FOIA.gov indicates there are additional attributes (below). Is there a reason not to add these to the data structure and metadata file?
* logo/seal
* description/about text
* Chief FOIA Officer
These all seem reasonable to add. I would prefer FOIA officers **not**
control the upload of a logo/seal for quality control, but it makes
sense to store it (we might be able to fetch that from an API, I'm not
sure).
### Agency Component
1. Confirm we are removing:
* "Required fields" and "additional fields" - as these are attributes of the form rather than the Agency Component
* Request stats
I spoke with @bsweger on this, and we are OK to move the "required
fields" and "additional fields" to the request form. Also okay to drop
Request stats for now.
2. I'm suggesting we model FOIA Liaisons etc as a consistent Person type. Dissent?
+1, Person is just a name, title, phone, and email?
3. Do we want to attach a name or label to each reading room?
I don't think a name or label is useful (again quality control). I think if we show a list of
raw URLs to a requester, that will be OK.
4. We'll need to add an API endpoint field for those Agency Components which want us to send them submissions via POST request to their API.
Yes, we have an `api` type captured in submission methods[1].
4. We'll need to add an indicator of how the Agency Component wants us to send data to them. I suggest a `preferred submission format` field with option for either email or API.
Mind if we leave this out for now? Email is going to be the first to be
implemented, so I'd rather punt on this for now.
[1]: https://github.com/18F/foia-recommendations/blob/master/schemas.md#submission-methods
…--
Aaron D Borden
Human and Hacker
|
|
Quoting Barrett Smith (2017-08-07 06:56:30)
## Questions
1. For Middle Initial and Middle Name:
* These are technically two different fields. Can we standardize on one of them?
* The note says they are frequently required for "First Party Requests". How do we identify what is a First Party Request?
Yeah, I think we will standardize on one based on the research coming
up. All these fields will go through some research validation as to
whether they should really be in the 90% bucket.
A first-party request is simply a request about yourself, e.g.
requesting your file from the FBI. I don't think there's any
programmatic way to determine it. It's just the nature of the request
and different agencies get different types of requests. Some may get no
first-party requests.
My hunch (and the research will inform us better) is that the 90% fields
won't be represented simply as fields when presented to FOIA officers.
There might be some way customize these fields at the agency component's
discretion e.g. if the agency component receives a lot of first-party
requests, they would choose to include the full
prefix/first/middle/last/suffix fields, but agencies that don't receive
first-party requests might choose a simple single full name field. But
you wouldn't get a granular menu of each individual field.
In the case of supporting documents, we might just leave these in the
10% bucket. Agencies could specify they'd like a generic Attachment
field for file upload (FOIA Online includes this by default for every
agency), or maybe several file upload fields for specific kinds of
documents.
/end speculation
2. phone numbers - if we are enforcing formatting, we need to consider if we need to handle international numbers (with country codes) and extensions
+1 to storing international numbers, unless it adds complexity. Would
Drupal return the numbers via API un-formatted?
3. In general - are we supporting conditionally required fields? @adborden, I believe you had said previously you thought these kind of fields were out of scope.
I don't think we understand the use case for conditionally required
fields yet. If we allow them, I don't think we would expose it for
everything. For example we might give some pre-canned options to select
from regarding collecting requester contact information conditionally.
4. Processing fees - I presume these should be indicated as being USD?
I was hoping bitcoin ;)
5. Attachments/Supporting documentation - is there any guidance on number of files that can be uploaded, their types (PDF, Word, Excel, .txt), or their maximum file size?
I don't think we know enough about this yet. I suspect we want some
reasonable limit on file size.
…--
Aaron D Borden
Human and Hacker
|
|
Responding to @ba66e77's questions:
Let's standardize to
@adborden answer ^^ that above
Do these issues help: 18F/identity-idp#1568 and https://github.com/18F/identity-private/issues/2252?
Are you referring to the 10 percent of fields required by some agencies? Then yes, I believe that is the goal. See issue #51
Bitcoin would be cool.
I'm not sure about this one. I'll look around for guidance. |
|
@RyanSibley I think by conditionally required fields, I think we mean "requesters must provide at least one of email address, mailing address". So email address is required only when requesters do not provide a mailing address. Mailing address is required only when requesters do not provide an email address. |
* FOIA-270: Adds Modal styling and functionality. * FOIA-270: Undoing autoformatting changes. * FOIA-270: Adding modal props.
[FOIA22-127] Style agency links in summary HTML (wip)
The starting point is in the schema document.
Examples are available.
The assumption for the data model is that an agency may have multiple components (the decentralized FOIA office model, e.g. USDA) or a single component (the centralized model, e.g. GSA). Either way, it makes some sense for the agency-to-component relationship to be a one-to-many.
Note: Removed the 4th acceptance criteria (the build-out in Drupal), since there are now discrete issues for those (#40 and #41)
The text was updated successfully, but these errors were encountered: