User Feedback Space #2
Replies: 9 comments 11 replies
-
|
问题：修正以下这句话：我做错了事情，真是湖图 |
Beta Was this translation helpful? Give feedback.
-
|
感谢反馈！目前句子太短确实准确率会低一些，因为单个句子能利用的特征太少了 |
Beta Was this translation helpful? Give feedback.
-
|
感觉可以根据不同NLP任务增加对应的语料，比如提供的这个人类的改法肯定是改成糊涂。 |
Beta Was this translation helpful? Give feedback.
-
|
The detector can be tricked by inserting square brackets [ ] for example for citiations. While round brackets ( ) still keep the decision that a Text was generated by a LLM with high confidence, square brackets can change this confidence from 99,9% ChatGPT to 99,9 human. Example of it is shown on your Twitter post https://twitter.com/sufeheisenberg/status/1616369062409273345 |
Beta Was this translation helpful? Give feedback.
-
|
Appreciate it so much for your feedback! We'll consider to add some perturbation into text to improve robustness of the detector in next version, and find a better way to deal with punctuations. |
Beta Was this translation helpful? Give feedback.
-
|
用Sinlge-text version试了一下火车上的广播：
|
Beta Was this translation helpful? Give feedback.
-
|
感谢反馈！！Sinlge-text 这个训练的分类器，确实缺少一些GLTR里的先验知识，泛化性不够好。语料中目前只有QA的训练数据，对于论文、公告等其他文体还没数据，我们也在计划扩充。 |
Beta Was this translation helpful? Give feedback.
-
|
Is your work published under a specific license? For example CC BY 4.0? |
Beta Was this translation helpful? Give feedback.
-
|
If the source datasets used in this corpus has a specific license which is stricter than CC-BY-SA, our products follow the same. If not, they follow CC-BY-SA license. https://github.com/Hello-SimpleAI/chatgpt-comparison-detection#dataset-copyright |
Beta Was this translation helpful? Give feedback.
-
|
问题：三体里面大刘是否对科技爆炸太乐观了，经济还有兴衰周期，没有经济支持科技能一直指数式增长吗？用更加口语化的说话方式并多用短句回答问题，不少于中文的 250 字。 |
Beta Was this translation helpful? Give feedback.
-
|
测试文本： 事故发生后，中共中央总书记、国家主席、中央军委主席习近平立即作出重要指示，惊悉东航MU5735航班失事，要立即启动应急机制，全力组织搜救，妥善处置善后。国务院委派领导同志靠前协调处理，尽快查明事故原因，举一反三，加强民用航空领域安全隐患排查，狠抓责任落实，确保航空运行绝对安全，确保人民生命绝对安全。 中共中央政治局常委、国务院总理李克强作出批示，要求全力以赴搜寻幸存者，尽一切可能救治伤员，妥善处理善后事宜，做好遇难者家属安抚和服务，实事求是、及时准确发布信息，认真严肃查明事故原因，采取有力措施加强民航安全管理。 独立文本版：ChatGPT 语言学版： 猜测：对于应用文来说，假阳性的可能性应该更大。但是应用文也没必要核实是否是人写的吧？ |
Beta Was this translation helpful? Give feedback.
-
|
文体太精炼了。。。。。。。 |
Beta Was this translation helpful? Give feedback.
-
|
语言学版data文件夹里的五个文件没有地方下载？怎么解决 |
Beta Was this translation helpful? Give feedback.
-
|
开发者你好，首先非常感谢你们的创作，对我的实际业务起到了很大帮助。我对比测试了3个版本的效果，singletext在我的业务中准确率大约有84%，非常符合业务的预期。 建议开发者可以考虑：支持在特定业务场景下，能在分句判断的基础上对一段文字进行整体判断，可类似gptzero输出四分类结果。（我目前根据字数判断的处理比较简单也不够合理） |
Beta Was this translation helpful? Give feedback.
-
|
您好，这确实是一个好建议！针对一篇文章，不仅仅要知道一个最终的预测结果，还希望得到具体哪里导致了这样的预测结果。后期我们如果开发应用的话，会加上这样的考虑，感谢！ |
Beta Was this translation helpful? Give feedback.
-
|
你好，我想看一下Linguistic Analysis的代码，想问一下能上传吗 |
Beta Was this translation helpful? Give feedback.
-
|
仅供参考（不可运行，仅用于原理展示）： # Copyright (c) Hello-SimpleAI Org. 2023.
# Licensed under the Apache License, Version 2.0.
import os
import pickle
import re
from typing import Callable, List, Tuple
from nltk.data import load as nltk_load
import numpy as np
from sklearn.linear_model import LogisticRegression
import torch
from transformers.utils import cached_file
from transformers import GPT2LMHeadModel, GPT2Tokenizer
DET_LING_ID = 'Hello-SimpleAI/chatgpt-detector-ling'
def download_file(filename):
return cached_file(DET_LING_ID, filename)
NLTK = nltk_load(download_file('english.pickle'))
sent_cut_en = NLTK.tokenize
LR_GLTR_EN, LR_PPL_EN, LR_GLTR_ZH, LR_PPL_ZH = [
pickle.load(open(download_file(f'{lang}-gpt2-{name}.pkl'), 'rb'))
for lang, name in [('en', 'gltr'), ('en', 'ppl'), ('zh', 'gltr'), ('zh', 'ppl')]
]
NAME_EN = 'gpt2'
TOKENIZER_EN = GPT2Tokenizer.from_pretrained(NAME_EN)
MODEL_EN = GPT2LMHeadModel.from_pretrained(NAME_EN)
NAME_ZH = 'IDEA-CCNL/Wenzhong-GPT2-110M'
TOKENIZER_ZH = GPT2Tokenizer.from_pretrained(NAME_ZH)
MODEL_ZH = GPT2LMHeadModel.from_pretrained(NAME_ZH)
# code borrowed from https://github.com/blmoistawinde/HarvestText
def sent_cut_zh(para: str) -> List[str]:
para = re.sub('([。！？\?!])([^”’)\]）】])', r"\1\n\2", para) # 单字符断句符
para = re.sub('(\.{3,})([^”’)\]）】….])', r"\1\n\2", para) # 英文省略号
para = re.sub('(\…+)([^”’)\]）】….])', r"\1\n\2", para) # 中文省略号
para = re.sub('([。！？\?!]|\.{3,}|\…+)([”’)\]）】])([^，。！？\?….])', r'\1\2\n\3', para)
# 如果双引号前有终止符，那么双引号才是句子的终点，把分句符\n放到双引号后，注意前面的几句都小心保留了双引号
para = para.rstrip() # 段尾如果有多余的\n就去掉它
# 很多规则中会考虑分号;，但是这里我把它忽略不计，破折号、英文双引号等同样忽略，需要的再做些简单调整即可。
sentences = para.split("\n")
sentences = [sent.strip() for sent in sentences]
sentences = [sent for sent in sentences if len(sent.strip()) > 0]
return sentences
CROSS_ENTROPY = torch.nn.CrossEntropyLoss(reduction='none')
def gpt2_features(
text: str, tokenizer: GPT2Tokenizer, model: GPT2LMHeadModel, sent_cut: Callable
) -> Tuple[List[int], List[float]]:
# Tokenize
input_max_length = tokenizer.model_max_length - 2
token_ids, offsets = list(), list()
sentences = sent_cut(text)
for s in sentences:
tokens = tokenizer.tokenize(s)
ids = tokenizer.convert_tokens_to_ids(tokens)
difference = len(token_ids) + len(ids) - input_max_length
if difference > 0:
ids = ids[:-difference]
offsets.append((len(token_ids), len(token_ids) + len(ids))) # 左开右闭
token_ids.extend(ids)
if difference >= 0:
break
input_ids = torch.tensor([tokenizer.bos_token_id] + token_ids)
logits = model(input_ids).logits
# Shift so that n-1 predict n
shift_logits = logits[:-1].contiguous()
shift_target = input_ids[1:].contiguous()
loss = CROSS_ENTROPY(shift_logits, shift_target)
all_probs = torch.softmax(shift_logits, dim=-1)
sorted_ids = torch.argsort(all_probs, dim=-1, descending=True) # stable=True
expanded_tokens = shift_target.unsqueeze(-1).expand_as(sorted_ids)
indices = torch.where(sorted_ids == expanded_tokens)
rank = indices[-1]
counter = [
rank < 10,
(rank >= 10) & (rank < 100),
(rank >= 100) & (rank < 1000),
rank >= 1000
]
counter = [c.long().sum(-1).item() for c in counter]
# compute different-level ppl
text_ppl = loss.mean().exp().item()
sent_ppl = list()
for start, end in offsets:
nll = loss[start: end].sum() / (end - start)
sent_ppl.append(nll.exp().item())
max_sent_ppl = max(sent_ppl)
sent_ppl_avg = sum(sent_ppl) / len(sent_ppl)
if len(sent_ppl) > 1:
sent_ppl_std = torch.std(torch.tensor(sent_ppl)).item()
else:
sent_ppl_std = 0
mask = torch.tensor([1] * loss.size(0))
step_ppl = loss.cumsum(dim=-1).div(mask.cumsum(dim=-1)).exp()
max_step_ppl = step_ppl.max(dim=-1)[0].item()
step_ppl_avg = step_ppl.sum(dim=-1).div(loss.size(0)).item()
if step_ppl.size(0) > 1:
step_ppl_std = step_ppl.std().item()
else:
step_ppl_std = 0
ppls = [
text_ppl, max_sent_ppl, sent_ppl_avg, sent_ppl_std,
max_step_ppl, step_ppl_avg, step_ppl_std
]
return counter, ppls # type: ignore
def lr_predict(
f_gltr: List[int], f_ppl: List[float], lr_gltr: LogisticRegression, lr_ppl: LogisticRegression,
id_to_label: List[str]
) -> List:
x_gltr = np.asarray([f_gltr])
gltr_label = lr_gltr.predict(x_gltr)[0]
gltr_prob = lr_gltr.predict_proba(x_gltr)[0, gltr_label]
x_ppl = np.asarray([f_ppl])
ppl_label = lr_ppl.predict(x_ppl)[0]
ppl_prob = lr_ppl.predict_proba(x_ppl)[0, ppl_label]
return [id_to_label[gltr_label], gltr_prob, id_to_label[ppl_label], ppl_prob]
def predict_en(text: str) -> List:
with torch.no_grad():
feat = gpt2_features(text, TOKENIZER_EN, MODEL_EN, sent_cut_en)
out = lr_predict(*feat, LR_GLTR_EN, LR_PPL_EN, ['Human', 'ChatGPT'])
return out
def predict_zh(text: str) -> List:
with torch.no_grad():
feat = gpt2_features(text, TOKENIZER_ZH, MODEL_ZH, sent_cut_zh)
out = lr_predict(*feat, LR_GLTR_ZH, LR_PPL_ZH, ['人类', 'ChatGPT'])
return out摘自我们huggingface上space里的代码（https://huggingface.co/spaces/Hello-SimpleAI/chatgpt-detector-ling/edit/main/app.py） |
Beta Was this translation helpful? Give feedback.
-
|
我主要是想了解关于依存距离那部分的代码，可以精确点吗 |
Beta Was this translation helpful? Give feedback.
-
|
我对其中的计算原理比较好奇，而且没有看到相关的引用论文 |
Beta Was this translation helpful? Give feedback.
-
|
哦抱歉，我看错了，我以为你要linguistic classifier GLTR的代码。我联系相关同学分享一下。 |
Beta Was this translation helpful? Give feedback.
-
👋
Hey, dear friends!
We've set up a feedback space here to collect everyone's feedback on our detector and the upcoming dataset.
嗨，亲爱的朋友！
我们在这里提供了一个反馈区，来收集大家对我们的检测器以及即将到来的数据集的反馈。
Beta Was this translation helpful? Give feedback.
All reactions