Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Audit task flags for fine grained label handling implemented and controllable through Admin dashboard. #3507

Merged
merged 18 commits into from
May 7, 2024

Conversation

davphan
Copy link
Collaborator

@davphan davphan commented Feb 27, 2024

Resolves #2508

Right now, the only way a user's tasks and labels can be marked as lower quality is by marking the entire user as a "low quality user." This presents the issue that users whose labels get better over time will still be marked as low quality, making all of their new good labels useless, or being marked as high quality and their bad labels will be used when they shouldn't be. This also limits the options we have for marking user audits as either high quality or low quality.

The addition of task flags allows an Admin to manually derive when a user's tasks improve in quality and mark older tasks as low quality, meaning a user can continue making meaningful contributions even if their earlier performance was not up to par. New flags, "incomplete" and "stale", were also incorporated, which increase the options for managing user tasks by having different effects on the task's re-audit, validation, clustering, and leaderboard effects. The effects and differences of each flag listed below:

Flag Name Re-audit? Include in Clustering Validate Include in Dashboard Include in Leaderboard Notes
Incomplete The labels placed on the street were good, but a lot of labels were missed (false negatives)
Low Quality Street was otherwise audited poorly
Stale To be applied automatically in the future to old data, but can be applied manually now.
Before/After screenshots (if applicable)

Admin User GSV Label View:
image

Admin User Page (under "General Info and Stats")
image

Testing instructions
  1. Sign in as an Admin account. Go to the Admin dashboard, then the "Users" tab on the left.
  2. Select a user, preferably the top user with a lot of audits.
  3. Scroll down to the "Set Flags by Date" section.
  4. Click either of the "Set Flags" buttons. This will apply the specified flag to all tasks started before the current date (so all tasks).
  5. Scroll up to the label map. Click on a label and scroll down to the "Manually Change Task Flags" section.
  6. Check that the button for the flag(s) you modified in step 4 are red, indicating they are applied.
  7. Click the buttons to manually change the flag of the task that the label is in.
Things to check before submitting the PR
  • I've written a descriptive PR title.
  • I've added/updated comments for large or confusing blocks of code.
  • I've included before/after screenshots above.
  • I've asked for and included translations for any user facing text that was added or modified.
  • I've updated any logging. Clicks, keyboard presses, and other user interactions should be logged. If you're not sure how (or if you need to update the logging), ask Mikey. Then make sure the documentation on this wiki page is up to date for the logs you added/updated.

Copy link
Member

@misaugstad misaugstad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made a few small changes:

  • Updated some wording
  • Fixed the stale flag in the GSV label view bc the button didn't do anything
  • The flags now only show up in the label view if you're an admin (they showed up for everyone in LabelMap)
  • For low_quality and stale data, instead of fully removing them from being validated, I made it so that they are just deprioritized

The rest of the comments are throughout the code! It looks great!

app/models/street/StreetEdgePriorityTable.scala Outdated Show resolved Hide resolved
app/models/attribute/UserClusteringSessionTable.scala Outdated Show resolved Hide resolved
app/models/user/UserStatTable.scala Outdated Show resolved Hide resolved
app/models/audit/AuditTaskTable.scala Outdated Show resolved Hide resolved
app/models/user/UserStatTable.scala Outdated Show resolved Hide resolved
public/javascripts/Admin/src/Admin.GSVLabelView.js Outdated Show resolved Hide resolved
public/javascripts/Admin/src/Admin.GSVLabelView.js Outdated Show resolved Hide resolved
app/controllers/AdminController.scala Outdated Show resolved Hide resolved
public/javascripts/Admin/src/Admin.User.js Outdated Show resolved Hide resolved
public/stylesheets/admin.css Outdated Show resolved Hide resolved
Copy link
Member

@misaugstad misaugstad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! There are probably a couple visual things we could do differently, but ultimately it's an admin tool that is primarily intended for me, so we don't need to sweat them :)

I'm going to wait to merge this though! We have a couple of mapathons in the next week, so I want to make sure that I can push a small bug fix to prod easily if need be without adding something big like this as well. Just trying to keep things stable. But I plan to merge next week after the mapathons are over!

@misaugstad misaugstad merged commit 727e8bf into develop May 7, 2024
@misaugstad misaugstad deleted the 2508-street-specific-flags branch May 7, 2024 20:28
@misaugstad misaugstad mentioned this pull request May 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Want to ignore data larger than a label but smaller than a user
2 participants