Skip to content

Release v0.0.4

Compare
Choose a tag to compare
@fracpete fracpete released this 19 Dec 02:50
· 95 commits to main since this release
  • pairs-to-llama2 filter now has an optional --prefix parameter to use with the instruction
  • added the pretrain-sentences-to-pairs filter for generating artificial prompt/response datasets from pretrain data
  • requires seppl>=0.0.11 now
  • the LDC_MODULES_EXCL environment variable is now used for specifying modules to be excluded from the registration process (e.g., used when generating help screens for derived libraries that shouldn't output the base plugins as well)
  • llm-registry and llm-help now allow specifying excluded modules via -e/--excluded_modules option
  • to-alpaca writer now has the -a/--ensure_ascii flag to enforce ASCII compatibility in the output
  • added global option -u/--update_interval to convert tool to customize how often progress of # records processed is being output in the console (default: 1000)
  • text-length filter now handles None values, i.e., ignores them
  • locations (i.e., input/instructions/output/etc) can be specified now multiple times
  • the llm-help tool can generate index files for all the plugins now; in case of markdown it will link to the other markdown files