LLM-as-a-Coauthor: The Challenges of Detecting LLM-Human Mixcase

This repository is a official code of the research presented in the paper [arXiv]. The goal is to provide a transparent, open-source implementation for the community to explore and build upon.

Dataset: MixSet

Overview

The MixSet dataset is a comprehensive collection designed for advanced machine learning experiments. It's structured to support a variety of tasks including text classification, natural language understanding, and more.

Location

The dataset is located in the ./data/MixSet/ directory relative to the project's root. Ensure that this path exists and contains the necessary data files before running any scripts that depend on the MixSet dataset.

Usage

To use the MixSet dataset in your experiments, refer to the scripts or modules specifically designed to read and process this data. Ensure that any path or configuration settings in your scripts point to the correct ./data/MixSet/ location.

Customization

If you customize or extend the dataset, consider documenting your changes here. This might include adding new files, filtering or processing data, or generating synthetic data points.

Acknowledgements

If the MixSet dataset is sourced from or inspired by an external project, publication, or another dataset, be sure to acknowledge the original creators or contributors here.

Contact

For any issues, questions, or suggestions related to the MixSet dataset, please contact Dongping Chen or open an issue in the project's repository.

Usage

Prerequisites

• Python = 3.9
• Other dependencies specified in requirements.txt

Installation

To set up your environment to run the code, follow these steps:

1. Clone the Repository:

git clone https://github.com/Dongping-Chen/MixSet.git
cd MixSet

2. Create and Activate a Virtual Environment (optional but recommended) and Install the Required Packages:

conda create --name mixset python=3.9
conda activate mixset
pip install -r requirements.txt

3. Download Datasets To download the MGT datasets, please refer to this link, then move the dataset folders to <YOUR PATH>/MixSet/data/MGT_datasets/. To download the HWT datasets, please refer to this link, then move the dataset folders to <YOUR PATH>/MixSet/data/pure_processed_HWT/.

4. Download Checkpoints of GPT-Sentinel Download the pretrained GPT-Sentinel t5-small and put to <YOUR PATH>/MixSet/.

Experiment Reproduce

Experiment 1

To reproduce the first experiments, run:

./Ex1_run.sh

You should run GPT-Zero by:

./Ex1_run_GPTzero

As for Ghostbuster, we will update the code as soon as possible.

Experiment 2

To reproduce the second experiments for binary classification, run:

./Ex2_binary_run

To reproduce the second experiments for three-class classification, run:

./Ex2_three_class_run

Experiment 3

To reproduce the third experiments for operation-wise transfer learning, run:

./Ex3_operation_train.sh
./Ex3_operation_test.sh

To reproduce the third experiments for LLM-wise transfer learning, run:

./Ex3_LLM_transfer.sh

Storage Requirements for Experiment 3 and 4 Scripts

Please be aware that the scripts for Experiment 3 and 4 require storing trained checkpoints in the folder path. This may occupy more than 20GB of space. It is essential to ensure that you have sufficient storage available on your device. Failing to allocate the necessary space might lead to interruptions during the code execution. We highly recommend checking and freeing up adequate space before running these scripts to ensure a smooth and uninterrupted experience.

Experiment 4

To reproduce the fourth experiments for ablation study, run:

./Ex4_auto_train.sh
./Ex4_auto_test.sh

Script Parameters Description

Below are the parameters used in the script along with their descriptions:

• --Mixcase_filename: Specifies the filename for the Mixcase data. Default is None.
• --MGT_only_GPT: If set, the script will only use MGT (Model Generated Text) from GPT-family models.
• --test_only: If set, the script will only perform testing, skipping any training procedures.
• --train_threshold: Specifies the threshold for training. Default is 10000.
• --no_auc: If set, the script will only calculate the Mixcase scenarios, which means no Area Under the ROC Curve (AUC) metrics.
• --only_supervised: If set, the script will perform only supervised learning without any unsupervised techniques.
• --train_with_mixcase: If set, the script will include Mixcase data in the training process.
• --seed: Sets the seed for random number generation to ensure reproducibility. Default is 0.
• --ckpt_dir: Specifies the directory to save checkpoints. Default is "./ckpt".
• --log_name: Specifies the name of the log file. Default is 'Log'.
• --mixcase_threshold: Sets the threshold for considering data as Mixcase. Default is 0.8.
• --transfer_filename: Specifies the filename for transfer learning. Default is None.
• --three_classes: If set, the script will use a three-class classification scheme instead of binary classification.
• --finetune: If set, the script will fine-tune the supervised model.
• --mixcase_as_mgt: If set, Mixcase data will be treated as Model Generated Text (MGT).

Contributing

Contributions to this project are welcome. Please consider the following ways to contribute:

• Reporting issues
• Improving documentation
• Proposing new features or improvements

Acknowledgements

This project is based on the findings and methodologies presented in the paper MixSet. Part of the code is borrowed from MGTBench.

Citation

@misc{gao2024llmasacoauthor,
      title={LLM-as-a-Coauthor: The Challenges of Detecting LLM-Human Mixcase}, 
      author={Chujie Gao and Dongping Chen and Qihui Zhang and Yue Huang and Yao Wan and Lichao Sun},
      year={2024},
      eprint={2401.05952},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}