对中文的NLP资源做个汇总，以备不时之需

• 中文NLP语料资源

• NLP工具包

• 中文分词

• QA & Chatbot 问答和聊天机器人

• NLP相关论文

• Python-Tutorial

• matplotlib-Ipython

• pandas

• scikit-learn

https://mp.weixin.qq.com/s/CODsntmNYwHga9jHxDdbgw

使用方法
下载语料 https://pan.baidu.com/s/1szmNZQrwh9y994uO8DFL_A 提取码：f2ex

执行命令即可
python main.py

生成结果
每个来源的语料分别生成一个独立的*.tsv文件，都放在新生成的clean_chat_corpus文件夹下。
生成结果格式为 tsv格式，每行是一个样本，先是query，再是answer

由哈工大SCIR博士生侯宇泰收集整理的一个任务型对话数据集大全

这份数据集大全涵盖了到目前在任务型对话领域的所有常用数据集的主要信息。 此外，为了帮助研究者更好的把握领域进展的脉络，以Leaderboard的形式给出了几个数据集上的State-of-the-art实验结果。

数据集的地址如下：

https://github.com/AtmaHou/Task-Oriented-Dialogue-Dataset-Survey

• 中文同义词表，反义词表，否定词表

• 1.4亿三元组中文知识图谱（https://github.com/ownthink/KnowledgeGraphData）

• Dbpedia（https://wiki.dbpedia.org/develop/datasets/dbpedia-version-2016-10）：多语知识图谱数据，共有130亿个三元组，但大部分都是英语。有760个类，1105个关系，1622个属性。

• 开放的中文知识图谱社区（http://www.openkg.cn/）：这里有很多垂直领域图谱数据，我就不一一放上来了。

• 腾讯词向量

  • 腾讯AI实验室公开的中文词向量数据集包含800多万中文词汇，其中每个词对应一个200维的向量。

  • 下载地址：https://ai.tencent.com/ailab/nlp/embedding.html

• THULAC 中文词法分析工具包 by 清华 (C++/Java/Python)

• LTP 语言技术平台 by 哈工大 (C++) pylyp LTP的python封装

• BaiduLac by 百度 Baidu's open-source lexical analysis tool for Chinese, including word segmentation, part-of-speech tagging & named entity recognition.

• HanLP (Java)

• SnowNLP (Python) Python library for processing Chinese text

• 小明NLP (Python) 轻量级中文自然语言处理工具

• chinese_nlp (C++ & Python) Chinese Natural Language Processing tools and examples

• CoreNLP by Stanford (Java) A Java suite of core NLP tools.

• Stanza by Stanford (Python) A Python NLP Library for Many Human Languages

• spaCy (Python) Industrial-Strength Natural Language Processing with a online course

• gensim (Python) Gensim is a Python library for topic modelling, document indexing and similarity retrieval with large corpora.

• Kashgari - Simple and powerful NLP framework, build your state-of-art model in 5 minutes for named entity recognition (NER), part-of-speech tagging (PoS) and text classification tasks. Includes BERT and word2vec embedding.

• Jieba 结巴中文分词 (Python及大量其它编程语言衍生) 做最好的 Python 中文分词组件

• 北大中文分词工具 (Python) 高准确度中文分词工具，简单易用，跟现有开源工具相比大幅提高了分词的准确率。

• A neural network model for Chinese named entity recognition

• bert-chinese-ner 使用预训练语言模型BERT做中文NER

• Information-Extraction-Chinese Chinese Named Entity Recognition with IDCNN/biLSTM+CRF, and Relation Extraction with biGRU+2ATT 中文实体识别与关系提取

• Rasa NLU (Python) turn natural language into structured data, a Chinese fork at Rasa NLU Chi

• Rasa Core (Python) machine learning based dialogue engine for conversational software

• Chatstack A Full Pipeline UI for building Chinese NLU System

• Snips NLU (Python) Snips NLU is a Python library that allows to parse sentences written in natural language and extracts structured information.

• DeepPavlov (Python) An open source library for building end-to-end dialog systems and training chatbots.

• ChatScript Natural Language tool/dialog manager, a rule-based chatbot engine.

• Chatterbot (Python) ChatterBot is a machine learning, conversational dialog engine for creating chat bots.

• Chatbot (Python) 基於向量匹配的情境式聊天機器人

• QA-Snake (Python) 基于多搜索引擎和深度学习技术的自动问答

• 使用TensorFlow实现的Sequence to Sequence的聊天机器人模型 (Python)

• 使用深度学习算法实现的中文阅读理解问答系统 (Python)

• AnyQ by Baidu 主要包含面向FAQ集合的问答系统框架、文本语义匹配工具SimNet。

• QASystemOnMedicalKG (Python) 以疾病为中心的一定规模医药领域知识图谱，并以该知识图谱完成自动问答与分析服务。

• GPT2-chitchat (Python) 用于中文闲聊的GPT2模型

1. SEO, Minjoon, et al. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603, 2016.

2. ZADEH, Amir, et al. Multi-attention recurrent network for human communication comprehension. arXiv preprint arXiv:1802.00923, 2018.

3. CHEN, Kehai, et al. Syntax-Directed Attention for Neural Machine Translation. arXiv preprint arXiv:1711.04231, 2017.

1. AutoExtend_ACL2014

2. AutoExtend_ACL2017

3. WELLER-DI MARCO, Marion; FRASER, Alexander; IM WALDE, Sabine Schulte. Addressing Problems across Linguistic Levels in SMT: Combining Approaches to Model Morphology, Syntax and Lexical Choice. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. 2017. p. 625-630.

4. YAGHOOBZADEH, Yadollah; SCHÜTZE, Hinrich. Multi-level representations for fine-grained typing of knowledge base entities. arXiv preprint arXiv:1701.02025, 2017.

5. TISSIER, Julien; GRAVIER, Christophe; HABRARD, Amaury. Dict2vec: Learning Word Embeddings using Lexical Dictionaries. In: Conference on Empirical Methods in Natural Language Processing (EMNLP 2017). 2017. p. 254-263.

6. PINTER, Yuval; GUTHRIE, Robert; EISENSTEIN, Jacob. Mimicking word embeddings using subword RNNs. arXiv preprint arXiv:1707.06961, 2017.

7. CHIU, Billy, et al. How to train good word embeddings for biomedical NLP. In: Proceedings of the 15th Workshop on Biomedical Natural Language Processing. 2016. p. 166-174.

8. XIE, Ruobing, et al. Lexical sememe prediction via word embeddings and matrix factorization. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence. AAAI Press, 2017. p. 4200-4206.

9. CHE, Zhengping, et al. Exploiting convolutional neural network for risk prediction with medical feature embedding. arXiv preprint arXiv:1701.07474, 2017.

10. YU, Liang-Chih, et al. Refining word embeddings for sentiment analysis. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017. p. 534-539.

11. FARUQUI, Manaal, et al. Retrofitting word vectors to semantic lexicons. arXiv preprint arXiv:1411.4166, 2014.

12. ABEND, Omri; RAPPOPORT, Ari. The state of the art in semantic representation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017. p. 77-89.

13. WU, Ledell, et al. StarSpace: Embed All The Things!. arXiv preprint arXiv:1709.03856, 2017.

14. CHEN, Liang-Wei; LEE, Wei-Chun; HWANG, Hsiang-Wei. When Word Embedding Meets Lexical Networks.

1. 王坤峰, et al. 生成式对抗网络 GAN 的研究进展与展望. 自动化学报, 2017, 43.3: 321-332.

2. CHEN, Xinchi, et al. Adversarial multi-criteria learning for chinese word segmentation. arXiv preprint arXiv:1704.07556, 2017.

3. LIU, Pengfei; QIU, Xipeng; HUANG, Xuanjing. Adversarial multi-task learning for text classification. arXiv preprint arXiv:1704.05742, 2017.

4. LI, Zheng, et al. End-to-end adversarial memory network for cross-domain sentiment classification. In: Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). 2017. p. 2237.

5. GUI, Tao, et al. Part-of-speech tagging for twitter with adversarial neural networks. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017. p. 2411-2420.

6. KIM, Joo-Kyung, et al. Cross-Lingual Transfer Learning for POS Tagging without Cross-Lingual Resources. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017. p. 2832-2838.

7. CRESWELL, Antonia, et al. Generative Adversarial Networks: An Overview. IEEE Signal Processing Magazine, 2018, 35.1: 53-65.

1. CRICHTON, Gamal, et al. A neural network multi-task learning approach to biomedical named entity recognition. BMC bioinformatics, 2017, 18.1: 368.

2. Chen, X., Qiu, X., & Huang, X. (2016). A feature-enriched neural model for joint Chinese word segmentation and part-of-speech tagging. arXiv preprint arXiv:1611.05384.

3. RUDER, Sebastian. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.

4. LONG, Mingsheng, et al. Learning Multiple Tasks with Multilinear Relationship Networks. In: Advances in Neural Information Processing Systems. 2017. p. 1593-1602.

5. AGUILAR, Gustavo, et al. A Multi-task Approach for Named Entity Recognition in Social Media Data. In: Proceedings of the 3rd Workshop on Noisy User-generated Text. 2017. p. 148-153.

1. WU, Yi; BAMMAN, David; RUSSELL, Stuart. Adversarial training for relation extraction. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017. p. 1778-1783.

2. HUANG, Yi Yao; WANG, William Yang. Deep Residual Learning for Weakly-Supervised Relation Extraction. arXiv preprint arXiv:1707.08866, 2017.

3. HUANG, Yi Yao; WANG, William Yang. Deep Residual Learning for Weakly-Supervised Relation Extraction. arXiv preprint arXiv:1707.08866, 2017.

4. HE, Zhengqiu, et al. SEE: Syntax-aware Entity Embedding for Neural Relation Extraction. arXiv preprint arXiv:1801.03603, 2018.

5. GANEA, Octavian-Eugen; HOFMANN, Thomas. Deep Joint Entity Disambiguation with Local Neural Attention. arXiv preprint arXiv:1704.04920, 2017.

6. ADEL, Heike; SCHÜTZE, Hinrich. Global Normalization of Convolutional Neural Networks for Joint Entity and Relation Classification. arXiv preprint arXiv:1707.07719, 2017.

7. Zeng, W., Lin, Y., Liu, Z., & Sun, M. (2016). Incorporating relation paths in neural relation extraction. arXiv preprint arXiv:1609.07479.

8. TAY, Yi; LUU, Anh Tuan; HUI, Siu Cheung. Learning to Attend via Word-Aspect Associative Fusion for Aspect-based Sentiment Analysis. arXiv preprint arXiv:1712.05403, 2017.

9. Zeng, X., He, S., Liu, K., & Zhao, J. (2018). Large Scaled Relation Extraction with Reinforcement Learning. Relation, 2, 3.

1. KIM, Joo-Kyung, et al. Cross-Lingual Transfer Learning for POS Tagging without Cross-Lingual Resources. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017. p. 2832-2838.

2. YANG, Zhilin; SALAKHUTDINOV, Ruslan; COHEN, William W. Transfer learning for sequence tagging with hierarchical recurrent networks. arXiv preprint arXiv:1703.06345, 2017.

3. PAN, Sinno Jialin; YANG, Qiang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 2010, 22.10: 1345-1359.

4. PAN, Sinno Jialin, et al. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 2011, 22.2: 199-210.

1. WANG, Bailin; LU, Wei. Learning Latent Opinions for Aspect-level Sentiment Classification. 2018.