Create script to check project files for missing data

[cnk]

Overview

We need an easy way of finding out when data is missing from project cards so that we can present an uniform UX and so that functions that rely on this data can work across all projects.

Action Items

• Determine desired output and process to follow
• Identify data types (links, leadership, description, etc)
• Identify which items projects might not have (e.g. live site)
• Propose value to use for fields where data is intentionally missing

Requirements

• Frequency: 1x/week
• Output: ?

[zempo]

I will make a test repo with mock data to try some solutions
It will expand upon @cnk 's previous attempt to crawl through the project's directory.

• research options
• make test repo
• check with Bonnie

[rblaz001]

I'm just getting over some pretty bad food poisoning so I haven't worked on this as much as I would have liked. I still need to make a 100 Automation's repo to continue developing my python script.

Action Item Progress

• Determine desired output and process to follow
• Identify data types (links, leadership, description, etc)
• Identify which items projects might not have (e.g. live site)
• Propose value to use for fields where data is intentionally missing

I have chosen to stick to a spreadsheet for the desired output of our script. I'm currently outputting a spreadsheet that contains two columns, one for the name of the project and the other for the list of missing data types.

The data types that we are using are currently hard coded in the script and were obtained by using a script that collected a set of used data types across all projects.

Set of Data Types

Project Missing Data Types

Note: I'm currently not taking account of nested data types. For example name, role, and links that are under "Leadership" in the following image. This is something I want to look in to, but I might have to reconsider how I deal with my output and determine what nested data types are important.

[rblaz001]

I now have a version of the script that outputs unused data types and missing data types for all projects.
I still need create a 100 automation repo and add my work to it.
Remaining work is quality of life edits to the script, including proper command line support.

The data types that we are using are currently hard coded in the script and were obtained by using a script that collected a set of used data types across all projects. The nested data types are denoted by comma delimited entries.

Set of Data Types

Project Unused Data Types

Project Missing Data Types (Currently Only One Missing)

• need to change output to properly where the missing data is nested.

Example of role missing from above entry

[rblaz001]

Added 100Automations repo. https://github.com/100Automations/script-gather-missing-data-types

[alexandrastubbs]

@ExperimentsInHonesty will talk to @rblaz001 about topic tags

[ExperimentsInHonesty]

@rblaz001 Please fill this out, so that we can get you listed on the 100automations.org site that will soon exist:
create a project card for a 100 automations project

[rblaz001]

Spreadsheet for current unused/missing data types.

Note: Unused Data Types are data types that are never used in a project card. Missing Data Types are nested data types that are missing even though the parent data types are present. (This can be one or more instances)

missingDataTypes.xlsx

[rblaz001]

Notes From Bonnie:

• Research Github Action integration for creating issues
• Add new validation method. Provide a source .md collection file with every data type in project with a require value or ignore value. This will be used to determine how script parses missing data types.
• Add functionality to output report of missing data types of source .md file. This will make sure that source .md file is kept up to date with requirements.
• Add additional output option as JSON
• Add Github Action integration

[alexandrastubbs]

@rblaz001 do you have a list of all the data types? If so, can you upload to google drive and share the link here?

[rblaz001]

@alexandrastubbs I have to make a small adjustment to the script to output all the currently used data types. I'm not home at the moment but I'll be able to upload the file and link it before tomorrow morning.

[rblaz001]

@alexandrastubbs I added an extra sheet labeled "Used Data Types" that has a list of data types used across the project cards.

https://docs.google.com/spreadsheets/d/1w59UfwbGiEAoBXjAxl4uwQe3xk5j6yFd3NxBMOm510Y/edit?usp=sharing

Note: The data types are not ordered. Also comma separated data types represent the nesting
For example One,Two,Three would look like the following in the project card

One:
    Two:
        Three: value

[rblaz001]

Progress Report
Progress - Was able to finish refactor, add JSON support, and update README. I was also able to update repo name and logo to include Jekyll
Blockers - Need to show progress to PM and get notes
Availability: 3-4 days depending on possible job interviews
ETA:

• Automation Card: End of week at most but shooting for much sooner
• Validate Through Template Feature: 7-9 days, difficulty level is relatively low but I'm catching up on documentation from refactor plus food-oasis tasks.
• GitHub Action Issues: TBD, began research

Work In Progress
Progress with automation card:
100Automations/Website#29

Finishing refactor documentation:
100Automations/jekyll-gather-data-types#2

Then working on validate through template feature:
100Automations/jekyll-gather-data-types#3

Was also able to look in to github action api and it should be possible to automate issues for missing data types relatively easy. The only blocker I'm having right now is thinking through how to manage issues so redundant issues are not created.

[alexandrastubbs]

@rblaz001 to rename from 'used data types' to 'all data types'

[rblaz001]

Progress - My availability changed this last week, I was unable to get any work done
Blockers - None
Availability: No more interviews or scheduling conflicts, will be available all week

Priority:

• Finish automation card
• Rename 'used data types' to 'all data types'
• Add validate through template feature

Task after priority finished:

• GitHub actions for automatic issue creation

[ExperimentsInHonesty]

@rblaz001 checking to see if you are able to put in an update on this issue before tomorrow.

[rblaz001]

Progress

• Finished automation card, renamed 'used data types' to 'all data types ' in output, and finished updating documentation.

Next step is to add validate through template.

• I will work on this tonight and either should finish first draft of it tonight, or will update with expected completion time.
• Finished MVP for validate through template, still need to update documentation and README before merging to master.

Then begin work on implementing GitHub actions

Blockers - Moving to Austin, still trying to find an apartment so I have very limited time

Availability - Limited

[ExperimentsInHonesty]

Do UX requirement gathering.
Bonnie will post on CfA slack asking for other people who have Jekyll sites and use collections.

[rblaz001]

template.md

---
alt: important
alt-hero: important
completed-contact: important
description: important
hide: ignore
identification: important
image: important
image-hero: important
leadership:
  - links:
      github: important
      linkedin: important
      slack: important
    name: important
    picture: important
    role: important
links:
  - name: important
    url: important
location: important
looking:
  - category: important
    skill: important
partner: important
status: important
technologies: important
title: important
tools: important
---

[rblaz001]

Updated Project Card Data Types - 9/13/2020
https://docs.google.com/spreadsheets/d/1F8ba2Kn6fjpp15zpseVqoYP7DsWVq1tpsKPusMCHF5Q/edit?usp=sharing

Updated Template Data Types - 9/13/2020
https://docs.google.com/spreadsheets/d/1KqqzZJOZKhX5QAB9X1wOwn_RuovRH1ikGwnQdUudbaM/edit?usp=sharing

[cnk]

What are the items like:

{
    "filename": "food-oasis.md",
    "errors": [
        "is not of a type(s) string",
        "is not of a type(s) array"
    ]
}

[akibrhast]

ah.. my bad... let me work on a bit more detailed error report.
But in answer to your question. The two errors in question are happening because these fields['looking','role'] in the image below are empt/null.
Where my json schema is expecting them to be a string. @cnk

[akibrhast]

Edited #386 (comment) , to give a bit more detailed report.
I was really hoping to get something along the line of

"error":[
 {
 "message": "is not of type string",
 "expected": "string",
 "given": "null",
 "stack": "on line 256 ,in 'message': null"
 }
]

but can't seem to find the relevant information... :(

[cnk]

"looking" (and probably "technologies") can be empty. Can you express that in a json schema? optional fields can exist but be empty?

[akibrhast]

yewp, not a problem. I can make the looking be either string/null . But a project should always at least have technology, no?? I would be hesitant on allowing that to be empty...

[akibrhast]

Updated - > #386 (comment)
An empty looking field does not report an error. Although it is still a required key.

[akibrhast]

Currently I have a on: workflow dispatch action running that checks the files and generates a validation report. @ruben1s

• https://github.com/akibrhast/website/actions/runs/652747129
• https://github.com/akibrhast/website/blob/gh-pages/_data/project_validation_report.json

Require feedback on where you want to go next with this @cnk? @ExperimentsInHonesty ?

My Proposal

• Have this github action run ever 3.5days
• On action start
  1. Check and see if an issue created by the action already exists
  2. If an open issue exists that was created by this bot-action
     • Quit Action

This should be fairly easy to implement using this endpoint

 https://api.github.com/repos/hackforla/website/issues?state=open&creator=akibrhast

3. If an open issue does not exist that was created by this bot-action
   • Generate validation report and create an issue with the report data

Example Issue that would be created :

Project Validation Failing[month/date/year]

Overview

The following report was generated due to project schema not matching validation schema. Please review the report listed below under Project Validation Error Report and apply the necessary fixes.

Project Validation Error Report

[
    {
        "file": "100 Automations",
        "errors": [
            {
                "message": "requires property \"partner\"",
                "stack": "instance requires property \"partner\""
            },
            {
                "message": "requires property \"tools\"",
                "stack": "instance requires property \"tools\""
            }
        ]
    },
    {
        "file": "311 Data",
        "errors": [
            {
                "message": "requires property \"tools\"",
                "stack": "instance requires property \"tools\""
            }
        ]
    }
]

Action Items

• Please apply the fixes to the files listed in the resources below using the error report above

Resources/Instructions

• https://github.com/hackforla/website/blob/gh-pages/_projects/100-automations.md
• https://github.com/hackforla/website/blob/gh-pages/_projects/311-data.md

[ExperimentsInHonesty]

@akibrhast I like the proposal.

• Still trying to figure out how this would be used.
  • concerned that if it has one issue that has a lot of different projects in it, it will be hard to parse out. We can't get 10 PMs to weigh in on one issue.
  • concerned about us determining that a project was not going to have something in it (like no partner) and no way to get it to stop flagging it.

Interested to hear your suggestions for improvement based on these concerns.

[akibrhast]

concerned about us determining that a project was not going to have something in it (like no partner) and no way to get it to stop flagging it.

Currently the schema is created based on this comment #386 (comment) ('Current Project Must Have's') .

According to that comment, all current projects must have those keys. However it seems like if you look at the image below that key is commented out for some reason. https://raw.githubusercontent.com/hackforla/website/gh-pages/_projects/100-automations.md . That should not be the case, it's fine if the partner value is empty, but key should still exist.

According to the original issue Action Item #386 (comment)

• Identify which items projects might not have (e.g. live site)

Some details that needs to be consolidated are

• what are the keys that a current project MUST ALWAYS have
• what are the values that a current project MUST ALWAYS have (example: title,id)
• what are the values that current project may be OPTIONAL(ex: partners)
• what are the type of the value a field accepts for example a slack link should always look like https://hackforla.slack.com/team/xxxxxx .

Same questions apply to Completed Projects and On Hold Projects.

Changing the schema is relatively simple. As you can see, a comment by @cnk #386 (comment) was made and the schema was soon changed after that.

---

concerned that if it has one issue that has a lot of different projects in it, it will be hard to parse out. We can't get 10 PMs to weigh in on one issue.

Maybe generate a single issue per file ?

[akibrhast]

Progress

None since 5 days ago
#386 (comment)

Blockers

Awaiting Feedback from @ExperimentsInHonesty @cnk

Availability

Here and There

ETA

N/A

[ExperimentsInHonesty]

Met with @akibrhast - he will add notes to this doc.