add

Artificial-Intelligence/AI.md

@@ -17,31 +17,16 @@
 * [chatgpt-dingtalk](https://github.com/eryajf/chatgpt-dingtalk)
 * [feishu-chatgpt](https://github.com/Leizhenpeng/feishu-chatgpt)
 
-### GPT
+### LLM
+* [Awesome-Chinese-LLM](https://github.com/HqWu-HITCS/Awesome-Chinese-LLM)
 * [ChatGPT资料汇总](https://github.com/dalinvip/Awesome-ChatGPT)
+* [LLMSurvey](https://github.com/RUCAIBox/LLMSurvey)
 * [awesome-RLHF](https://github.com/opendilab/awesome-RLHF)
 * [langflow](https://github.com/logspace-ai/langflow)
 * [openai-cookbook](https://github.com/openai/openai-cookbook)
 
-### IMAGE
-* [stable-diffusion](https://github.com/CompVis/stable-diffusion)
-  * [awesome](https://github.com/awesome-stable-diffusion/awesome-stable-diffusion)
-  * [civitai](https://civitai.com/)
-  * [stable-diffusion-art](https://stable-diffusion-art.com/)
-  * [stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui)
-  * [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
-  * [multi-frame-rendering](https://xanthius.itch.io/multi-frame-rendering-for-stablediffusion)
-* [visual-chatgpt](https://github.com/microsoft/visual-chatgpt)
-* [DALL·E](https://platform.openai.com/docs/guides/images)
-* [midjourney](https://www.midjourney.com/)
-* [image][https://github.com/openai/CLIP]
-* [segment-anything](https://github.com/facebookresearch/segment-anything)
-
-### Video Voice
-* [video][https://studio.d-id.com/]
-* [ms vallex](https://vallex-demo.github.io/)
-
 ### ChatGPT-web
+* [lmsys](chat.lmsys.org)
 * [chatgpt-demo](https://github.com/ddiu8081/chatgpt-demo)
 * [chatgpt-web](https://github.com/Chanzhaoyu/chatgpt-web)
 * [GUI for ChatGPT API](https://github.com/GaiZhenbiao/ChuanhuChatGPT)
@@ -55,17 +40,16 @@
 * [slickgpt](https://github.com/ShipBit/slickgpt)
 * [chatgpt_academic](https://github.com/binary-husky/chatgpt_academic)
 * [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
-
-### ChatGPT-PDF
-* [gpt4-pdf](https://github.com/mayooear/gpt4-pdf-chatbot-langchain)
-* [chatpdf-minimal-demo](https://github.com/postor/chatpdf-minimal-demo)
-* [ChatPaper](https://github.com/kaixindelele/ChatPaper)
+* [ChatGPT-Next-Web](https://github.com/Yidadaa/ChatGPT-Next-Web)
+* [lobe-chat](https://github.com/lobehub/lobe-chat)
+* [gpts](https://github.com/huggingface/chat-ui)
 
 
 ### Model Quantization And Fine-tune
 * [llama](https://github.com/facebookresearch/llama)
 * [bloomz](https://huggingface.co/bigscience/bloomz)
 * [pythia](https://github.com/EleutherAI/pythia)
+* [RedPajama](https://huggingface.co/togethercomputer)
 * [dolly](https://github.com/databrickslabs/dolly)
 * [Open-Assistant](https://github.com/LAION-AI/Open-Assistant)
 * [MOSS](https://github.com/OpenLMLab/MOSS)
@@ -76,7 +60,63 @@
 * [Chinese-LLaMA-Alpaca](https://github.com/ymcui/Chinese-LLaMA-Alpaca)
 * [Vicuna](https://github.com/lm-sys/FastChat)
 * [ChatGLM-6B](https://github.com/THUDM/ChatGLM-6B)
+* [ChatGLM2](https://github.com/THUDM/ChatGLM2-6B)
 * [LMFlow](https://github.com/OptimalScale/LMFlow)
+* [Linly](https://github.com/CVI-SZU/Linly)
+* [TigerBot](https://github.com/TigerResearch/TigerBot)
+* [OpenBuddy](https://github.com/OpenBuddy/OpenBuddy)
+* [open_llama_13b](https://huggingface.co/openlm-research/open_llama_13b)
+* [Fengshenbang](https://github.com/IDEA-CCNL/Fengshenbang-LM)
+* [Firefly](https://github.com/yangjianxin1/Firefly)
+* [TransGPT](https://github.com/DUOMO/TransGPT)
+* [InternLM](https://github.com/InternLM/InternLM)
+* [WizardLM](https://github.com/nlpxucan/WizardLM)
+* [LLaMA-Efficient-Tuning](https://github.com/hiyouga/LLaMA-Efficient-Tuning)
+* [LLaMA2-Accessory](https://github.com/Alpha-VLLM/LLaMA2-Accessory)
+* [Llama2-Code-Interpreter](https://github.com/SeungyounShin/Llama2-Code-Interpreter)
+* [Baichuan2](https://github.com/baichuan-inc/Baichuan2)
+* [Anima LangChat](https://github.com/lyogavin/Anima/)
+* [Llama2-Chinese](https://github.com/FlagAlpha/Llama2-Chinese)
+* [Firefly-LLaMA2-Chinese](https://github.com/yangjianxin1/Firefly-LLaMA2-Chinese)
+* [01](https://github.com/01-ai/Yi)
+* [DeepSeek](https://github.com/deepseek-ai)
+
+### vertical
+* [DISC-LawLLM](https://github.com/FudanDISC/DISC-LawLLM)
+
+
+### Multimodal
+* [VisualGLM-6B](https://github.com/THUDM/VisualGLM-6B)
+* [Visual-Chinese-LLaMA-Alpaca](https://github.com/airaria/Visual-Chinese-LLaMA-Alpaca)
+* [Vary](https://github.com/Ucas-HaoranWei/Vary)
+* [gligen](https://github.com/mut-ex/gligen-gui)
+* [SDXL-Lightning](https://huggingface.co/ByteDance/SDXL-Lightning)
+* [SUPIR](https://github.com/Fanghua-Yu/SUPIR)
+* [CodeFormer](https://github.com/sczhou/CodeFormer)
+* [EMO](https://github.com/HumanAIGC/EMO)
+
+#### IMAGE
+* [stable-diffusion](https://github.com/CompVis/stable-diffusion)
+  * [awesome](https://github.com/awesome-stable-diffusion/awesome-stable-diffusion)
+  * [civitai](https://civitai.com/)
+  * [stable-diffusion-art](https://stable-diffusion-art.com/)
+  * [stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui)
+  * [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
+  * [multi-frame-rendering](https://xanthius.itch.io/multi-frame-rendering-for-stablediffusion)
+* [visual-chatgpt](https://github.com/microsoft/visual-chatgpt)
+* [DALL·E](https://platform.openai.com/docs/guides/images)
+* [midjourney](https://www.midjourney.com/)
+* [image][https://github.com/openai/CLIP]
+* [segment-anything](https://github.com/facebookresearch/segment-anything)
+* [TripoSR](https://github.com/VAST-AI-Research/TripoSR) image-to-3d
+
+#### Video Voice
+* [video][https://studio.d-id.com/]
+* [ms vallex](https://vallex-demo.github.io/)
+* [audiolm](https://github.com/lucidrains/audiolm-pytorch)
+* [bark](https://github.com/suno-ai/bark)
+* [TTS](https://github.com/netease-youdao/EmotiVoice)
+
 
 ### Tools Apps
 * [popular ai list](https://www.futurepedia.io/)
@@ -85,10 +125,41 @@
 * [Auto-GPT](https://github.com/Torantulino/Auto-GPT)
 * [Auto-GPT](https://github.com/Significant-Gravitas/Auto-GPT)
 * [AgentGPT](https://github.com/reworkd/AgentGPT)
+* [babyagi](https://github.com/yoheinakajima/babyagi)
 * [JARVIS HuggingGPT](https://github.com/microsoft/JARVIS)
 * [code search](https://github.com/BloopAI/bloop)
+* [graphgpt](https://github.com/varunshenoy/graphgpt)
+* [BMTools](https://github.com/OpenBMB/BMTools)
+* [OpenAGI](https://github.com/agiresearch/OpenAGI)
+* [LowCodeLLM](https://github.com/microsoft/TaskMatrix/tree/main/LowCodeLLM)
+* [知识图谱](https://github.com/zjunlp/DeepKE/)
+* [工作流](https://github.com/AndersonBY/vector-vein)
+* [MetaGPT](https://github.com/geekan/MetaGPT)
+* [ChatDev](https://github.com/OpenBMB/ChatDev/)
+* [open-interpreter](https://github.com/KillianLucas/open-interpreter)
+* [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent)
+* [code refact](https://github.com/smallcloudai/refact)
+* [GPTS](https://supertools.therundown.ai/gpts)
+* [RecurrentGPT](https://github.com/aiwaves-cn/RecurrentGPT)
+
+#### ChatGPT-PDF
+* [gpt4-pdf](https://github.com/mayooear/gpt4-pdf-chatbot-langchain)
+* [chatpdf-minimal-demo](https://github.com/postor/chatpdf-minimal-demo)
+* [ChatPaper](https://github.com/kaixindelele/ChatPaper)
+* [quivr](https://github.com/stangirard/quivr)
+* [danswer](https://github.com/danswer-ai/danswer)
+* [FastGPT](https://github.com/labring/FastGPT)
+* [Langchain-Chatchat](https://github.com/chatchat-space/Langchain-Chatchat)
 
+### LLMOps
+* [predibase](https://predibase.com/)
+* [opengpts](https://github.com/langchain-ai/opengpts)
 
+## eval
+* [c-eval](https://github.com/SJTU-LIT/ceval)
+* [flageval](https://flageval.baai.ac.cn/#/trending)
+* [embedding](https://huggingface.co/spaces/mteb/leaderboard)
+* [ragas](https://github.com/explodinggradients/ragas)
 
 
 
@@ -105,6 +176,22 @@
 * [facebook llama](https://github.com/facebookresearch/llama)
 * [Plug and play modules to optimize the performances of your AI systems](https://github.com/nebuly-ai/nebullvm)
 
+### embedding
+* [text2vec](https://github.com/shibing624/text2vec)
+* [uniem](https://github.com/wangyuxinwhy/uniem)
+* [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding)
+* [Chinese-BERT-wwm](https://github.com/ymcui/Chinese-BERT-wwm)
+* [网易有道BCEmbedding](https://huggingface.co/maidalun1020/bce-embedding-base_v1)
+* [jina](https://huggingface.co/jinaai/jina-embeddings-v2-base-zh)
+
+### Text2sql
+* [sqlcoder](http://github.com/defog-ai/sqlcoder)
+* [DB-GPT](https://github.com/eosphoros-ai/DB-GPT)
+* [Awesome-Text2SQL](https://github.com/eosphoros-ai/Awesome-Text2SQL)
+
+### Prompts
+* [GPTs](https://github.com/linexjlin/GPTs)
+
 ## 实践
 * [罪名法务,数据+模型](https://github.com/liuhuanyong/CrimeKgAssitant)
 * [ChatGPT-WEB-APP-Clone](https://github.com/SashenJayathilaka/ChatGPT-Clone)

Artificial-Intelligence/llama_train.md

@@ -0,0 +1,26 @@
+# LLAMA2 Chinese
+
+```SHELL
+conda create -n qlllama2 python=3.8
+conda activate qlllama2
+conda deactivate
+conda remove -n qlllama2 --all
+
+pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
+
+torchrun --nproc_per_node=8 train.py --train_args_file train_args/llama2-13b-ext.yaml
+
+```
+
+## 报错记录
+
+* libbitsandbytes_cpu 版本不对导致的
+
+  ```shell
+  python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cquantize_blockwise_fp16_nf4
+  ```
+  用自己的CUDA版本覆盖
+  ```SHELL
+  cd python3.8/site-packages/bitsandbytes
+  cp libbitsandbytes_cuda121.so libbitsandbytes_cpu.so
+  ```

DevOps/Kubernetes.md

@@ -212,7 +212,20 @@ kubectl create -f multi-pods.yaml
 
 kubectl exec -it pod名称 -c 容器名称 sh或者bash
 ```
+### minikube
+```SHELL
+sudo apt-get install docker.io
+sudo apt  install docker.io  #安装docker
+sudo groupadd docker     #添加用户组 docker
+sudo gpasswd -a $USER docker     #将登陆用户加入到docker用户组中
+newgrp docker     #更新用户组
 
+curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
+sudo install minikube-linux-amd64 /usr/local/bin/minikube
+
+minikube start
+
+```
 ### microk8s
 #### 安装
 Ubuntu系统下安装microk8s非常方便，但大多情况可能会遇到安装完成后一直不能启动的问题，主要是下载k8s所需的 k8s.gcr.io 或 gcr.io 镜像出现被墙问题。

DevOps/运维.md

@@ -15,6 +15,7 @@ nvidia-smi
 DataDog、Sysdig、Prometheus
 
 ## Nginx
+
 ```shell
 location ^~/user/ {
     proxy_set_header Host $host;
@@ -34,3 +35,58 @@ location ^~/user/ {
   proxy_pass http://user;
 }
 ```
+
+## mysql 扩容
+```SHELL
+sudo umount /mnt/mydisk
+
+
+# 查看磁盘（显示所有磁盘）
+fdisk -l
+# 查看磁盘使用情况（未挂载的磁盘不会显示）
+df -h
+
+
+# dev/vdb是前面查出来的
+diskNumber=dev/vdb
+mkdir /data
+parted /$diskNumber mklabel gpt
+parted /$diskNumber mkpart opt 0% 100%
+sleep 1
+mkfs -t ext4 /${diskNumber}1
+mount /${diskNumber}1 /data
+df -h
+blkid /${diskNumber}1 |awk '{print $2 " /'$diskNumber' ext4 defaults 0 0"}' >> /etc/fstab
+cat /etc/fstab
+
+
+# 创建文件夹
+mkdir -p /vdb/mysql_data/mysql
+# 复制mysql未挂载前的数据
+cp -rf /var/lib/mysql/* /vdb/mysql_data/mysql
+# 授权mysql用户组
+chown -R  mysql:mysql /vdb/mysql_data
+# 授权文件夹
+chmod -R 700 /vdb/mysql_data
+
+vim /etc/apparmor.d/usr.sbin.mysqld
+
+# Allow data dir access
+将  /var/lib/mysql/ r,
+    /var/lib/mysql/** rwk,
+修改为
+# Allow data dir access
+    /vdb/mysql_data/mysql/ r,
+    /vdb/mysql_data/mysql/** rwk,
+
+
+service mysql restart
+
+# 查看binlog是否关闭
+show variables like 'log_%';
+# 查看端口是否修改
+show global variables like 'port';
+# 查看挂载目录是否生效
+show global variables like 'datadir';
+
+```

Linux/NetworkNamespace.md

@@ -0,0 +1,34 @@
+# Network Namespace 网络命名空间
+
+
+> 一般情况下，所有数据流量默认通过有线网卡走。然后，对于特定的程序或 Terminal 窗口使用无线网卡，可以通过在启动它们之前设置路由表实现。  
+> 这种操作涉及到 Linux 系统的路由管理和命名空间的概念。Linux 命名空间（namespace）是内核的一种特性，可以隔离进程组，使其只能看到特定命名空间中的全局资源。这里我们使用网络命名空间（network namespace）来实现进程网络隔离。
+
+创建一个新的网络命名空间
+
+```shell
+sudo ip netns add mynamespace
+sudo ip netns list # 查看
+```
+
+可以将你的无线网卡连接到新创建的网络命名空间。假设你的无线网卡是 wlan0：
+
+```shell
+sudo ip link set wlan0 netns mynamespace
+```
+
+可以进入到这个网络命名空间中，并配置网络：
+
+```shell
+sudo ip netns exec mynamespace bash
+sudo ifconfig wlan0 up # 启动网卡
+sudo dhclient wlan0 # 网络DHCP
+ # 无线密码
+sudo wpa_supplicant -B -i wlan0 -c /path/to/your/wpa_supplicant.conf
+
+# 添加本地回环网络
+sudo ip netns exec mynamespace ip link set lo up
+# 设置DNS
+sudo nano /etc/resolv.conf    
+          nameserver 8.8.8.8
+```

Notepad/opensource-awesome.md

@@ -23,6 +23,8 @@ Opensource
 ![GitHub stars](https://img.shields.io/github/stars/sindresorhus/awesome-nodejs?style=flat-square)  
 [golang-open-source-projects](https://github.com/hackstoic/golang-open-source-projects)
 ![GitHub stars](https://img.shields.io/github/stars/hackstoic/golang-open-source-projects?style=flat-square)  
+[awesome-selfhosted](https://github.com/awesome-selfhosted/awesome-selfhosted)
+![GitHub stars](https://img.shields.io/github/stars/awesome-selfhosted/awesome-selfhosted?style=flat-square)  
 [Awesome-Design-Tools](https://github.com/LisaDziuba/Awesome-Design-Tools)
 ![GitHub stars](https://img.shields.io/github/stars/LisaDziuba/Awesome-Design-Tools?style=flat-square)  
 [Beginner-friendly Projects](https://github.com/MunGell/awesome-for-beginners)
@@ -154,7 +156,8 @@ Github Actions
 * [Riot-web](https://github.com/vector-im/riot-web)
 ![GitHub stars](https://img.shields.io/github/stars/vector-im/riot-web)
 * [Wire](https://github.com/wireapp/wire)
-![GitHub stars](https://img.shields.io/github/stars/wireapp/wire)
+![GitHub stars](https://img.shields.io/github/stars/wireapp/wire)  
+* [matrix](https://matrix.org/)
 
 ### Google Drive
 * [Nextcloud](https://github.com/nextcloud/server)
@@ -170,6 +173,8 @@ Github Actions
 ![GitHub stars](https://img.shields.io/github/stars/silverwind/droppy)  
 * [Chevereto](https://github.com/Chevereto/Chevereto-Free)
 ![GitHub stars](https://img.shields.io/github/stars/Chevereto/Chevereto-Free)图床    
+* [minio](https://github.com/minio/minio)
+![GitHub stars](https://img.shields.io/github/stars/minio/minio)
 
 ### [zendesk](https://zen.com/)
 * [Zammad](https://github.com/zammad/zammad)

Python/pandas.md

@@ -400,6 +400,9 @@ df[df.eval("Q1 > `Q2`+@a")]
 
 # 分组取前3
 df_ask_org.groupby('CLASS').apply(lambda x:x.nlargest(3,'述求量'))
+
+df.groupby('street')['community'].apply(lambda x: x.value_counts().nlargest(10))
+
 ```
 
 ### 数据处理、分组、透视