TextFlint

TextFlint is an unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing . TextFlint offers 20 general transformations, 60 task-specific transformations and thousands of their combinations, and provides over 67,000 evaluation results generated by the transformation on 24 classic datasets from 12 tasks.

---

This repository contains all code and data used to build the TextFlint website, hosted here: https://www.textflint.io

Adding or modifying results

TextFlint allows you to upload evaluation results on new models or transformations. You can now edit files directly in GitHub to create pull requests. Please follow the instructions blow, We will check the correctness of the results. If the data is correct, it will be synchronized to the website in a few days.

Contributing to TextFlint

Thanks for contributing to TextFlint! Here are some guidelines to get your pull request accepted. You can now edit files directly in GitHub to create pull requests. All data is in the folder ./Tasks.

How to add evaluation result

Before adding a evaluation result, make sure the result is reliable and to comfirm the following information of the result.

• Task
• Dataset
• Model
• Metric
• Download link of dataset before and after transformation

The result files are in the ./Tasks/{your_task}/results/{your_data} folder. According to the metirc and transformation type, open the corresponding JSON file, following fields will be shown ( take ABSA as an example ).

{transformation_type}_{metric}.json

[
    {
        "model":"LCF-BERT",
        "attack_results":
            {
                "RevTgt":{
                            "ori":81.97,
                            "trans":48.93,
                            "sample_num": ,
                            "ori_download_link": "https://www.textflint.com/static/Tasks/ABSA/trans_dataset/SemEval2014-Laptop/ori_RevTgt.json",
                            "trans_download_link": "https://www.textflint.com/static/Tasks/ABSA/trans_dataset/SemEval2014-Laptop/trans_RevTgt.json",
                            "contributor": {"rui zheng":{"github": "https://github.com/ruizheng20"}}
                        }
            }
    }
]

Find the model to which the evaluation result belong (If it's a new model, fill in all the fields above and insert it after the other models), add this result to attack_results field, followed by other transformation. The meanings of the attack_results field is as follows. Make sure the added files follow the JSON strictly.

• attack_results: Results you should add
  • key: Name of transformation
  • value:
    • ori: Evaluation result before transformation
    • trans: Evaluation result after transformation
    • sample_num: Number of data that can be transformed
    • ori_download_link: Download link of dataset before transformation, please provide available download link, or contact us
    • trans_download_link: Download link of dataset after transformation
    • contributor:
      • key: Name of contributor
      • value:
        • github: Github homepage link of contributor

If your model or transformation are not in our list, you need to upload a description of the model or transformation

How to add model description

The model description files are in the ./Tasks/{your_task}/models folder. Create a JSON file named by your model, and finish the JSON file by imitating the fields below.

your_model_name.json

{
    "desc":"The key idea of these proposals are to learn aspect embeddings and let aspects participate in computing attention weights.",
    "contributor":[{"name":"rui zheng", "github":"https://github.com/ruizheng20"}]
}

• desc: Description of model
• contributor
  • name: Name of contributor
  • github: Github homepage link of contributor

How to add transformation description

The transformation description files are in the ./Tasks/{your_task}/transformations folder. Create a json file named by your transformation, and finish the JSON file by imitating the fields below.

your_transformation_name.json

{
    "desc":"AddDiff: Add aspects with the opposite sentiment from the target aspect.",
    "examples":[
        {
            "ori":"BEST spicy tuna roll, great asian salad.（Target: spicy tuna roll)",           
             "trans":"TBEST spicy tuna roll, great asian salad, but this small place is packed, on a cold day, the seating by the entrance way can be pretty drafty and bad service."
        },
        {
            "ori":"The food was extremely tasty, creatively presented and the wine excellent. (Target: wine)",
            "trans":"The food was extremely tasty, creatively presented and the wine excellent, but yeah, sometimes the service can be slow, a lentil dish was salty beyond edibility and the red curry is weak and tasteless."
        }
    ],
    "contributor":[
        {
            "name":"rui zheng",
            "github":"https://github.com/ruizheng20"
        }
    ]
}

• desc: Description of transformation
• examples: Please provide at least two examples
  • ori: Text before transformation
  • trans: Text after transformation
• contributor
  • name: Name of contributor
  • github: Github homepage link of contributor

How to add human evaluation

Please follow the same standard as https://www.textflint.com/human_evaluation for the human evaluation. The human evaluation files are in the ./Tasks/{your_task}/human_evaluations folder. Create a json file named by your transformation, and finish the JSON file by imitating the fields below. your_transformation_name.json

{
    "ori_list": [3.9, 4.0, 4.030100334, 4.0], 
    "trans_list": [3.84, 4.0, 3.806666667, 4.0]
}

• ori_list: Human evaluation results of data before transformation. The elements in the array represent each other in turn: the mean of grammar score, the median of grammar score, the mean of plausibility score, the median of plausibility score.
• trans_list: Human evaluation results of data after transformation. The elements in the array represent each other in turn: the mean of grammar score, the median of grammar score, the mean of plausibility score, the median of plausibility score.