-
Notifications
You must be signed in to change notification settings - Fork 5
Add PSEO to data schema #92
Comments
From: Stephen R Tibbets (CENSUS/CES FED) In-line responses below: From: Andrew Foote (CENSUS/CES FED) Stephen - Thank you for your feedback on this. I think a lot of the comments on variables would be helped along by us having a schema. For instance, the institution_id variable is sourced from the Department of Education (it is technically the Office of Post-Secondary Education ID (OPEID), and so it is a unique (national) identifier in the data. They don't have leading zeros in the reports I have, but we can add leading zeros to make it uniform. It's good to know they are nationally unique, so that larger issue should not be one. If they do not use a leading zero, we probably should not - you never know, if we assume it is 6 digits, and they assign a number more than a million we're stuck. We should describe it as a numeric variable in the schema, rather than character - you can't really tell otherwise from a csv which would be appropriate. As for the CIP codes, I did some poking to see if the ACS has a naming convention (because they have field of study), but it appears that they use a completely different coding scheme for field of study, which is not at all correlated with the CIP codes. Even on the NCES website, the 4 digit CIP codes are just referred to as CIP codes, so we could change that to variable to "cip" and it would be clear to people. "CIP" is probably good enough, but I don't have a strong opinion on that one. As for having full labels on output files, that's fine to relegate to schema - I put them up there initially because we didn't really have any infrastructure built. I do think the csv's should not have labels. One option is to provide supplemental files in XLSX format as well, and include labels on those. They can be created directly from SAS (assuming we get 9.4, eventually - or we can do it manually for these early betas). As for the file names, is one option having an additional variable that indicates the vintage the data was created on. For instance, the current data would have an 03262018 vintage tag, and new cells will have a different tag. We could have that on the internal version if that makes more sense. One other small note - all the earnings data will be brought up to current year dollars when we update the tables. Right now it is in 2016 dollars, but it will need to be updated. One important thing to note is that the data will always (before pre-release) be in 2016 dollars, because the histogram is in 2016 dollars. Then everything will be transformed into current year dollars. We typically don't include the "vintage" info in the release data itself, rather providing the info in an accompanying metadata file. Given the data is so small, we could add a variable tagging it directly. If we do roll up multiple cohorts into the same file, we can do a little thinking about how best to tag them. If records from this initial release later get cast into future dollars, those numbers should be tagged differently than this first release, but potentially we would want them to be tagged differently from future cohorts. We can do some brainstorming about that. You are right about earnings - we should round to the whole dollar. Easy enough And we can add a separate flag for suppressed, rather than my solution, which isn't really one. I think each cell should have a status flag (we release all variables or none). You've just combined two separate concepts into a single variable, in a way. Sometimes better to break it out - not a big deal. Andrew Foote, Economist, LEHD Center for Economic Studies U.S. Census Bureau Cell 202.384.4784 Room 5K058E census.gov All, I did have some thoughts on things we might consider adjusting in the next PSEO release:
Our current full QWI/J2J data schema is here: https://lehd.ces.census.gov/data/schema/V4.2.0/lehd_public_use_schema.html Given how simple this file is, it should not be a big deal to formally document, but we should do it. Please let me know if you have any thoughts on my suggestions. Thanks, |
What is the timeline on this? V4.3 or later? |
The initial PSEO is already out, and the next one is in development, for release later this summer. Can we perhaps stick it onto a -draft schema, while we finalize things? The files were dropped onto the website without a lot of thought to how the releases were going to evolve, so I'm a little torn about how exactly we want to handle this. I don't know we can have all the questions answered before 4.3 needs to be posted.
…________________________________
From: Lars Vilhuber <notifications@github.com>
Sent: Monday, June 4, 2018 2:02:46 PM
To: labordynamicsinstitute/qwi_schemas
Cc: Stephen R Tibbets (CENSUS/CES FED); Assign
Subject: Re: [labordynamicsinstitute/qwi_schemas] Add PSEO to data schema (#92)
What is the timeline on this? V4.3 or later?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub<#92 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AVlyoXeeQp3ClPmKLVJCqCqKZAiaZl5Iks5t5XZGgaJpZM4UW7eD>.
|
Pick a date, and we'll work backwards from there. 4.3 was a suggestion, but Q4, 2019Q1, etc. are all options.
…--
Lars Vilhuber, Economist
Cornell University, Executive Director, Labor Dynamics Institute
and ILR School - Department of Economics
American Economic Association - Data Editor
Journal of Privacy and Confidentiality - Managing Editor
e: lars.vilhuber@cornell.edu
p: +1.607-330-5743
v: https://cornell.zoom.us/my/larsvilhuber
w: http://lars.vilhuber.com/ <http://lars.vilhuber.com/>
Assistant: ldi@cornell.edu | +1.607-255-2744
________________________________
From: srt1 <notifications@github.com>
Sent: Monday, June 4, 2018 2:08:56 PM
To: labordynamicsinstitute/qwi_schemas
Cc: Lars Vilhuber; Assign
Subject: Re: [labordynamicsinstitute/qwi_schemas] Add PSEO to data schema (#92)
The initial PSEO is already out, and the next one is in development, for release later this summer. Can we perhaps stick it onto a -draft schema, while we finalize things? The files were dropped onto the website without a lot of thought to how the releases were going to evolve, so I'm a little torn about how exactly we want to handle this. I don't know we can have all the questions answered before 4.3 needs to be posted.
________________________________
From: Lars Vilhuber <notifications@github.com>
Sent: Monday, June 4, 2018 2:02:46 PM
To: labordynamicsinstitute/qwi_schemas
Cc: Stephen R Tibbets (CENSUS/CES FED); Assign
Subject: Re: [labordynamicsinstitute/qwi_schemas] Add PSEO to data schema (#92)
What is the timeline on this? V4.3 or later?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub<#92 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AVlyoXeeQp3ClPmKLVJCqCqKZAiaZl5Iks5t5XZGgaJpZM4UW7eD>.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub<#92 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AGsoeMkWa6V5zv3QepJE3Aebhkca7tuTks5t5Xe4gaJpZM4UW7eD>.
|
Q4 (October 2018) is probably the best target. The next data release may happen before that, and we could initially release the standard label_* files in a non-standard place (say, on the PSEO beta site) before they get incorporated into the full schema. But I would like to be settled on a pretty stable specification for the output files with this next release.
I might also recommend re-releasing the Texas data in the revised standard, as well - we could keep the initial release on the website - the data would be the same. One thought on that, every PSEO release could potential include re-releases all of the historical files, even though there are no changes to the data contents of the earlier ones.
…________________________________
From: Lars Vilhuber <notifications@github.com>
Sent: Monday, June 4, 2018 2:10:23 PM
To: labordynamicsinstitute/qwi_schemas
Cc: Stephen R Tibbets (CENSUS/CES FED); Assign
Subject: Re: [labordynamicsinstitute/qwi_schemas] Add PSEO to data schema (#92)
Pick a date, and we'll work backwards from there. 4.3 was a suggestion, but Q4, 2019Q1, etc. are all options.
--
Lars Vilhuber, Economist
Cornell University, Executive Director, Labor Dynamics Institute
and ILR School - Department of Economics
American Economic Association - Data Editor
Journal of Privacy and Confidentiality - Managing Editor
e: lars.vilhuber@cornell.edu
p: +1.607-330-5743
v: https://cornell.zoom.us/my/larsvilhuber
w: http://lars.vilhuber.com/ <http://lars.vilhuber.com/>
Assistant: ldi@cornell.edu | +1.607-255-2744
________________________________
From: srt1 <notifications@github.com>
Sent: Monday, June 4, 2018 2:08:56 PM
To: labordynamicsinstitute/qwi_schemas
Cc: Lars Vilhuber; Assign
Subject: Re: [labordynamicsinstitute/qwi_schemas] Add PSEO to data schema (#92)
The initial PSEO is already out, and the next one is in development, for release later this summer. Can we perhaps stick it onto a -draft schema, while we finalize things? The files were dropped onto the website without a lot of thought to how the releases were going to evolve, so I'm a little torn about how exactly we want to handle this. I don't know we can have all the questions answered before 4.3 needs to be posted.
________________________________
From: Lars Vilhuber <notifications@github.com>
Sent: Monday, June 4, 2018 2:02:46 PM
To: labordynamicsinstitute/qwi_schemas
Cc: Stephen R Tibbets (CENSUS/CES FED); Assign
Subject: Re: [labordynamicsinstitute/qwi_schemas] Add PSEO to data schema (#92)
What is the timeline on this? V4.3 or later?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub<#92 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AVlyoXeeQp3ClPmKLVJCqCqKZAiaZl5Iks5t5XZGgaJpZM4UW7eD>.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub<#92 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AGsoeMkWa6V5zv3QepJE3Aebhkca7tuTks5t5Xe4gaJpZM4UW7eD>.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub<#92 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AVlyoReJJ_9nMadbhvMIKXPj2f901Yy1ks5t5XgPgaJpZM4UW7eD>.
|
FOR DISCUSSION ONLY There will be (for now) two separate "products," PSEO and PSEOD, the latter having the destinations of employment (all names and such are subject to revision at this point). I created "identifiers" and "variables" files, as we do for J2J and QWI. I have also included "label" files for the various characteristic variables. All up for discussion at this point, this is just where we are so far. |
This comment has been minimized.
This comment has been minimized.
Latest version of schema files. A few comments:
Output files will be organized by the state of the institution, with some multi-state institutions placed into a "US/multi" level table, of some sort. I'm sure there are other issues and quirks.. will add when I think of them. |
I'm running into a few issues where encoding is crashing the build. I don't have it narrowed down yet but these files should be checked to confirm they're valid utf-8. Here are some problematic rows: $ grep -axv '.*' ./*csv
Update: Yeah, the label_institution.csv is what was crashing the asciidoctor build. |
This is in your fork, right?
…--
Lars Vilhuber, Economist
Cornell University, Executive Director, Labor Dynamics Institute
and ILR School - Department of Economics
American Economic Association - Data Editor
Journal of Privacy and Confidentiality - Managing Editor
e: lars.vilhuber@cornell.edu
p: +1.607-330-5743
v: https://cornell.zoom.us/my/larsvilhuber
w: http://lars.vilhuber.com/ <http://lars.vilhuber.com/>
Assistant: ldi@cornell.edu | +1.607-255-2744
________________________________
From: Jody Hoon-Starr <notifications@github.com>
Sent: Thursday, May 16, 2019 15:21
To: labordynamicsinstitute/qwi_schemas
Cc: Lars Vilhuber; Assign
Subject: Re: [labordynamicsinstitute/qwi_schemas] Add PSEO to data schema (#92)
I'm running into a few issues where encoding is crashing the build. I don't have it narrowed down yet but these files should be checked to confirm they're valid utf-8. Here are some problematic rows:
$ grep -axv '.*' ./*csv
label_institution.csv:G06710,FUNDACI�N UNIVERSIDAD DE LAS AM�RICAS PU,SAN ANDR�S CHOLULA,MX,I,
label_institution.csv:003945,UNIVERSITY OF PUERTO RICO - MEDICAL SCIE,R�O PIEDRAS,PR,I,72
label_institution.csv:007108,UNIVERSITY OF PUERTO RICO - R�O PIEDRAS,R�O PIEDRAS,PR,I,72
label_institution.csv:008246,DIN� COLLEGE,TSAILE,AZ,I,04
label_institution.csv:014534,INSTITUTO DE BANCA Y COMERCIO,R�O PIEDRAS,PR,I,72
label_institution.csv:015868,COMMONWEALTH OF PR DEPT OF EDUC - ITPR R,R�O PIEDRAS,PR,I,72
label_institution.csv:025875,UNIVERSIDAD METROPOLITANA,R�O PIEDRAS,PR,I,72
label_institution.csv:035313,UNIVERSIDAD PENTECOSTAL MIZPA,R�O PIEDRAS,PR,I,72
label_institution.csv:036083,CARIBBEAN FORENSIC & TECHNICAL COLL,R�O PIEDRAS,PR,I,72
label_institution.csv:037765,"UNIVERSIDAD DE LA SALLE BAJIO, A.C.",LE�N,MX,I,
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub<#92?email_source=notifications&email_token=ABVSQ6ELZXUZZV44GQ6O3P3PVWX5DA5CNFSM4FC3W6B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVSZ4FA#issuecomment-493198868>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABVSQ6BHHBU45AQ2MXZ5OP3PVWX5DANCNFSM4FC3W6BQ>.
|
Just dropping the csvs into the directory got them caught by the write_schemadoc script. Will untangle this and split it out into its own doc generator. |
Yes, this is on the fork https://github.com/LEDApplications/qwi_schemas/tree/V4.5.0-draft |
I forgot the utf-8 option on the csv creation. I'll repost with a new one for you shortly.
…________________________________
From: Jody Hoon-Starr <notifications@github.com>
Sent: Thursday, May 16, 2019 3:47 PM
To: labordynamicsinstitute/qwi_schemas
Cc: Stephen R Tibbets (CENSUS/CES FED); Assign
Subject: Re: [labordynamicsinstitute/qwi_schemas] Add PSEO to data schema (#92)
Yes, this is on the fork https://github.com/LEDApplications/qwi_schemas/tree/V4.5.0-draft
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub<#92?email_source=notifications&email_token=AFMXFIMVJQRCYPKM2TVCPT3PVW253A5CNFSM4FC3W6B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVS34YI#issuecomment-493207137>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFMXFIIHKW2PT26WLW3F4TTPVW253ANCNFSM4FC3W6BQ>.
|
label_institution.zip |
This shouldn't be a problem for the doc creation.
* on the main doc, filter out
* on the pseo doc, pre-process (you have over 6000 lines, so you want to do a similar pre-processing as for the county names)
…--
Lars Vilhuber, Economist
Cornell University, Executive Director, Labor Dynamics Institute
and ILR School - Department of Economics
American Economic Association - Data Editor
Journal of Privacy and Confidentiality - Managing Editor
e: lars.vilhuber@cornell.edu
p: +1.607-330-5743
v: https://cornell.zoom.us/my/larsvilhuber
w: http://lars.vilhuber.com/ <http://lars.vilhuber.com/>
Assistant: ldi@cornell.edu | +1.607-255-2744
________________________________
From: Jody Hoon-Starr <notifications@github.com>
Sent: Thursday, May 16, 2019 15:47
To: labordynamicsinstitute/qwi_schemas
Cc: Lars Vilhuber; Assign
Subject: Re: [labordynamicsinstitute/qwi_schemas] Add PSEO to data schema (#92)
Yes, this is on the fork https://github.com/LEDApplications/qwi_schemas/tree/V4.5.0-draft
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub<#92?email_source=notifications&email_token=ABVSQ6G7P54ARHBNMHCLTU3PVW253A5CNFSM4FC3W6B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVS34YI#issuecomment-493207137>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABVSQ6CJL7W6ZWJA5GPVHKTPVW253ANCNFSM4FC3W6BQ>.
|
Opened pull reques for pseo schema related changes: #116 |
I have pulled in #116. However, there are a few significant issues: @jodyhoonstarr @andrewfoote
|
Section 6 Categorical Variables Add grad_cohort to this section. Text below: This is a 4-digit number representing the first year of the graduation cohort. The length of the cohort will vary by degree_level as follows:
For tabulations across all cohorts, the grad_cohort will be recorded as 0000. |
Changes to be implemented:
CSV files to be updated (identifiers, pseoe, pseof) |
Above changes implemented in #119 |
Please pick up the following schema files from the M: drive:
|
zip of the above files |
changes can be found in #120 |
Changes: |
Section 5.3.6, 5.3.7: Please keep status flag in display version |
We want to include formal documentation of the PSEO outputs in the data schema. Note, I have made some comments to Andrew about the first release, suggesting some adjustments for future releases. Not sure what schema number we want to call this, or if we want a separate schema. It's a pretty simple release, though the future iterations are a little different to QWI/J2J.
See:
https://lehd.ces.census.gov/data/pseo_beta.html
https://lehd.ces.census.gov/data/pseo/
The text was updated successfully, but these errors were encountered: