TODO AI

Help needed in the area of AI

Choose LLM

Desired result

• Open Source
• Local
• Runs on cell phones / Android (see below)
• Quality roughly comparable to ChatGPT

Options

• Open Assistant / LAION
• Llama, and derivates
• RoBERTa
  • https://huggingface.co/roberta-base
  • Derivates
    • https://huggingface.co/models?other=roberta
    • https://huggingface.co/distilroberta-base
    • https://huggingface.co/microsoft/codebert-base
  • https://github.com/facebookresearch/fairseq/tree/main/examples/roberta
• Others
  • https://huggingface.co

Tasks

• Find more options
• Compare with ChatGPT / GPT4

Dangers

• Open source models may not be good enough
• Llama on shaky legal ground
• GPT is expensive, competing, not local etc.

Structured output

Fine-tune/train the LLM to produce machine-readable and consistent output as JSON.

Desired result

E.g. "Find a hotel in Madrid" returns

{
  action: "book",
  subject: "hotel",
  criteria: {
    location: "https://en.wikipedia.org/Madrid",
    gps: "34.444,55.2343",
  }
}

"Find a red blouse" or "I would like a red blouse" returns

{
  action: "buy",
  subject: "product",
  subjectCategory: [ "cloths", "top", "shirt", "business" ],
  criteria: {
    gender: "female",
    color: "red",
  }
}

Options

• Fine-tune
• LoRA
• Prompt
• Microsoft Guidance
  • https://github.com/microsoft/guidance
  • https://www.heise.de/news/Microsoft-Guidance-legt-ChatGPT-und-Co-gewuenschte-Ketten-an-9060268.html
  • MIT license

Tasks

• Try the above options, and possibly others
• Might involve training or fine-tuning of models
• Run rests to ensure that the output is consistent and reliable.
• (Other team members) Create test cases (input sentences)
• (Other team members) Define and document output formats for many more cases.

Dangers

• LLMs do not produce coherent and consistent output
• LLMs lie -> correctness

LLM speed

Find or create a version of the chosen LLM that can run on most current smart phones.

• Objective: Result in less than 1 second.
• Compare section Android | AI speed

Dangers

• LLMs are too slow for current smart phones
  • ARM CPUs fast enough?
  • Only the latest high end phones have an AI coprocessor
  • Difficult to use the AI coprocessors, due to processor instruction sets and APIs
  • Impossible to use the AI coprocessors, due to security or lack of APIs
• LLMs need too much storage space for current smart phones.
• LLMs need too much RAM for current smart phones.

Variable parameters for LLM

• Pass list of contact names, songs, close locations etc. to LLM
• Desired result: It can recognize those and include them in the structured output.

Variable parameters for Speech recognition

• Pass list of contact names, songs, close locations etc. to LLM
• Desired result: It can recognize those with higher confidence and likeliness.

Wake word

• Listen to audio continously and react only to the specific wake word.
• Make "Parula" the wake word
• Low false position, low false negative
• Runs on PC, Android, Raspberry Pi, ESP32
• Very low CPU / power consumption

Mitigation

• Not needed when on Android home screen, activated by Android device button, or by PC keyboard button.