-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Audit task flags for fine grained label handling implemented and controllable through Admin dashboard. #3507
Conversation
…Webpage into 2508-street-specific-flags
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I made a few small changes:
- Updated some wording
- Fixed the stale flag in the GSV label view bc the button didn't do anything
- The flags now only show up in the label view if you're an admin (they showed up for everyone in LabelMap)
- For low_quality and stale data, instead of fully removing them from being validated, I made it so that they are just deprioritized
The rest of the comments are throughout the code! It looks great!
…Webpage into 2508-street-specific-flags
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great! There are probably a couple visual things we could do differently, but ultimately it's an admin tool that is primarily intended for me, so we don't need to sweat them :)
I'm going to wait to merge this though! We have a couple of mapathons in the next week, so I want to make sure that I can push a small bug fix to prod easily if need be without adding something big like this as well. Just trying to keep things stable. But I plan to merge next week after the mapathons are over!
…Webpage into 2508-street-specific-flags
Resolves #2508
Right now, the only way a user's tasks and labels can be marked as lower quality is by marking the entire user as a "low quality user." This presents the issue that users whose labels get better over time will still be marked as low quality, making all of their new good labels useless, or being marked as high quality and their bad labels will be used when they shouldn't be. This also limits the options we have for marking user audits as either high quality or low quality.
The addition of task flags allows an Admin to manually derive when a user's tasks improve in quality and mark older tasks as low quality, meaning a user can continue making meaningful contributions even if their earlier performance was not up to par. New flags, "incomplete" and "stale", were also incorporated, which increase the options for managing user tasks by having different effects on the task's re-audit, validation, clustering, and leaderboard effects. The effects and differences of each flag listed below:
Before/After screenshots (if applicable)
Admin User GSV Label View:
![image](https://private-user-images.githubusercontent.com/77756028/308001282-f93c7491-d8ec-4c1c-b1a3-119980923d7e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI3ODkxMjAsIm5iZiI6MTcyMjc4ODgyMCwicGF0aCI6Ii83Nzc1NjAyOC8zMDgwMDEyODItZjkzYzc0OTEtZDhlYy00YzFjLWIxYTMtMTE5OTgwOTIzZDdlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODA0VDE2MjcwMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTAzNDU0MGQzMmQ0NDFhMDYyNzliZWE5MzI2OGE4OWIxN2IzMjRkZjEzMWNiZDIzMDc2NTEzYWUxYzI1OTBkYjImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.H7KoFjA-I6tSdrlDF_G-CBehdziLAbg1pn2-b9m6GaI)
Admin User Page (under "General Info and Stats")
![image](https://private-user-images.githubusercontent.com/77756028/308001347-69808728-f126-48cf-ac33-c06bcd45b3dd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI3ODkxMjAsIm5iZiI6MTcyMjc4ODgyMCwicGF0aCI6Ii83Nzc1NjAyOC8zMDgwMDEzNDctNjk4MDg3MjgtZjEyNi00OGNmLWFjMzMtYzA2YmNkNDViM2RkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODA0VDE2MjcwMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWE2OTVmZDU0ZDhlNWI4NDdkNWJmOGI1NzU5YzM5M2Q4OWNlOWNhNDM3NzlhMjEzM2M4NDI4NDljNzI0MDg3ZGImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.oKQ1CBLuXN9EkZg1LSD5VdXGwHv1LGY6XTVWZs38hmU)
Testing instructions
Things to check before submitting the PR