Repositories list

• Megatron-LM

  Public
  Python

  •

  Other

  •0•3•0•1•Updated Nov 7, 2024Nov 7, 2024

• llm-jp-eval

  Public
  Python

  •

  Apache License 2.0

  •38•99•4•4•Updated Nov 6, 2024Nov 6, 2024

• scripts

  Public
  Shell

  •

  Apache License 2.0

  •1•1•4•10•Updated Nov 6, 2024Nov 6, 2024

• Toxicity_Filter

  Public
  A DeBERTa-based toxicity filter trained with LLM-jp Toxicity Dataset

  Python

  •0•0•0•0•Updated Nov 5, 2024Nov 5, 2024

• tuning-competition

  Public
  HTML

  •0•0•0•0•Updated Nov 4, 2024Nov 4, 2024

• llm-jp-data-contamination

  Public
  The data contamination repository for the LLM-JP project. Forked from @eiei7

  Python

  •0•1•0•0•Updated Nov 3, 2024Nov 3, 2024

• awesome-japanese-llm

  Public
  日本語LLMまとめ - Overview of Japanese LLMs

  japanesegenerative-modeljapanese-language+ 16language-modelslanguage-modelgenerative-modelsmultimodalvision-and-languagevision-languagefoundation-models+ 9

  TypeScript

  •

  Apache License 2.0

  •31•1k•0•0•Updated Nov 1, 2024Nov 1, 2024

• text2dataset

  Public
  Easily turn large English text datasets into Japanese text datasets using open LLMs.

  Python

  •

  MIT License

  •0•13•9•0•Updated Oct 29, 2024Oct 29, 2024

• nii-sakura-cluster-configs

  Public
  Shell

  •0•2•0•0•Updated Oct 26, 2024Oct 26, 2024

• moe-recipes

  Public
  Shell

  •0•1•0•0•Updated Oct 24, 2024Oct 24, 2024

• transformers

  Public
  🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

  Python

  •

  Apache License 2.0

  •27k•0•0•0•Updated Oct 14, 2024Oct 14, 2024

• FastChat2

  Public
  An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

  Python

  •

  Apache License 2.0

  •4.5k•0•0•0•Updated Oct 3, 2024Oct 3, 2024

• llm-leaderboard

  Public
  Project of llm evaluation to Japanese tasks

  Python

  •36•0•0•0•Updated Aug 11, 2024Aug 11, 2024

• FastChat

  Public
  An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

  Python

  •

  Apache License 2.0

  •4.5k•0•0•0•Updated Aug 9, 2024Aug 9, 2024

• experiments

  Public
  Issue-Only Pretrain Task Management Repository

  0•0•42•0•Updated Aug 8, 2024Aug 8, 2024

• llm-jp-tokenizer

  Public
  Roff

  •

  Apache License 2.0

  •7•31•1•0•Updated Aug 7, 2024Aug 7, 2024

• llmjp_topic_model

  Public
  Python

  •0•0•0•0•Updated Jul 19, 2024Jul 19, 2024

• modelwg

  Public
  LLM-jp Model-WG Working Directory

  Shell

  •

  Apache License 2.0

  •0•6•0•4•Updated Jul 9, 2024Jul 9, 2024

• DeepSpeed-MII

  Public
  MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

  Python

  •

  Apache License 2.0

  •176•0•0•0•Updated Jul 5, 2024Jul 5, 2024

• vllm

  Public
  A high-throughput and memory-efficient inference and serving engine for LLMs

  Python

  •

  Apache License 2.0

  •4.5k•0•0•0•Updated Jun 21, 2024Jun 21, 2024

• nii-geniac-gcp-lustre

  Public
  GENIAC用のLustreクラスタ構築スクリプト

  Jinja

  •0•0•0•0•Updated Jun 20, 2024Jun 20, 2024

• nii-geniac-hpc-toolkit

  Public
  HCL

  •0•1•0•0•Updated Jun 13, 2024Jun 13, 2024

• llm-jp-sft

  Public
  Python

  •

  Apache License 2.0

  •15•51•0•1•Updated Jun 13, 2024Jun 13, 2024

• memorization-analysis

  Public
  Python

  •0•0•3•1•Updated Jun 7, 2024Jun 7, 2024

• nii-geniac-gcp-maintenance

  Public
  Shell

  •0•0•0•0•Updated Jun 7, 2024Jun 7, 2024

• open-llm-leaderboard-local

  Public
  Open LLM Leaderboard のローカル実行用スクリプト

  Python

  •1•3•0•1•Updated Feb 24, 2024Feb 24, 2024

• llm-jp-model-playground

  Public
  Interactive application to verify multiple LLMs

  TypeScript

  •

  MIT No Attribution

  •2•13•11•2•Updated Feb 20, 2024Feb 20, 2024

• llm-jp-dpo

  Public
  Python

  •

  Apache License 2.0

  •3•5•0•0•Updated Feb 6, 2024Feb 6, 2024

• llm-jp-corpus

  Public
  Python

  •

  Apache License 2.0

  •4•41•3•0•Updated Feb 2, 2024Feb 2, 2024

• sakura-h100-setup

  Public
  0•4•0•0•Updated Jan 5, 2024Jan 5, 2024