Survey Datasets Transformation and Re-Configuration

[evaristoc]

We are working on data transformation to fit the requirements for the analysis.

People who have been working on this so far:

• @erictleung
• @koustuvsinha
• @koustuvsinha made a first separation of the datasets and made a first cleaning of the variables (see https://gitter.im/FreeCodeCamp/DataScience?at=571d25cc7469496137b8c45f)
• @erictleung took the files and started modifications using R (see some discussion at https://gitter.im/FreeCodeCamp/DataScience?at=571f0955d47413c07dcd4755)

We are currently working on @erictleung's fork but reporting through this channel to preserve history.

[evaristoc]

@erictleung identified several discrepancies, particularly related to data parsing. Those that he found, he solved. We must be following this until being sure that all corrections are complete.

[evaristoc]

These are main tasks:

Currently working on best way to encode variables and values to facilitate analysis. We are working on:

• Variable re-naming and or Creation
• Variable labelling
• Value encoding
• Value labelling
• Missing value encoding and labelling
• Metadata File Management

We should take care that changes made to values and variables should be updated in the metadata file manager.

[QuincyLarson]

@evaristoc @erictleung Before you do this, I recommend you get a final dump of the survey. We've had several hundred additional responses since I last did a dump. I will upload the files now.

[QuincyLarson]

@erictleung @evaristoc OK - I've added the updated data.

[erictleung]

Just for reference, here is the branch of my fork of the repo to clean and combine the data. I'm primarily doing the data processing in R, for those interested.

[evaristoc]

We found several inconsistencies for the answers given to:

• Resources : Others
• PodCasts : Others
• Code Events : Others

We are working on parsing the data and trying to give some consistency to those inconsistency based on general assumptions. This also means that answers that can not be responded for those assumptions will be considered as missing.

This procedure might not be perfect but it is the best we can do.

[evaristoc]

Sorry for the two recent messages (5 min ago): they were deleted and discussion moved to #29

[SamAI-Software]

Reference for questions about data #41