Custom Flagging Callbacks & Flagging for Hugging Face Spaces

[abidlabs]

Motivation

We would like the FLAG button on Gradio interfaces to be flexible. In particular, on Hugging Face Spaces, the FLAG button should take the sample and save it to a Hugging Face Dataset. This can be useful for collecting data e.g. for building a crowd-sourced dataset of samples on which a machine learning demo makes mistakes.

Proposal

We create a new FlaggingCallback abstract class whose subclasses must define the following two methods:

• setup(flagging_dir)
• flag(self, interface, input_data, output_data, flag_option, flag_index, username)

The setup() method is called once when the Interface is launched and can do things like create a directory, while the flag() method is what is called when the button is actually called and can do things like append to a CSV file.

The Interface object is constructed (through the flagging_callback parameter) with a FlaggingCallback object. The default callback is an object of a class known as CSVLogger which implements the current behavior of the FLAG button. We also define a SimpleCSVLogger for pedagogical reasons.

Most importantly, we define a new HuggingFaceDatasetSaver callback which a user can use to enable the behavior described above (save a flagged sample to a HuggingFace Dataset). In order to do this, a user simply needs to instantiate a HuggingFaceDatasetSaver with hf_foken (their Hugging Face token, which can be provided through Secrets), and dataset_name. The class will take care of the rest utilizing the hub API.

This is work in progress -- feedback welcome.

link T-34

[height]

This pull request has been linked to 1 task:

• T-34 HF Flagging <> Dataset Integration (unlink task)

💡Tip: Add "Close T-34" to the pull request title or description, to a commit message, or in a comment to mark this task as "Done" when the pull request is merged.

[abidlabs]

Link T-34

[aliabid94]

LGTM. Possibly allow them to also pass just a function instead having to implement the whole class (since only the flag option will be necessary in most cases), but your call.

[abidlabs]

While this still needs to be polished, merging so that @aliabid94 and other can build off of this.