Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement multimodal models (LLaVA) #3436

Merged
merged 36 commits into from Oct 12, 2023
Merged

Implement multimodal models (LLaVA) #3436

merged 36 commits into from Oct 12, 2023
+10,214 −2

Commits on Oct 2, 2023

  1. WIP: start implementing LLaVA

    @monatis
    monatis committed Oct 2, 2023
    Copy the full SHA
    59aa1ac View commit details
    Browse the repository at this point in the history

  2. rm scratch buf for now, will revert after cleanup

    @monatis
    monatis committed Oct 2, 2023
    Copy the full SHA
    0f0e7c6 View commit details
    Browse the repository at this point in the history

  3. LLaVA image encoder is working. will combine with llama

    @monatis
    monatis committed Oct 2, 2023
    Copy the full SHA
    7e9120f View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2023

  1. Add llava inference code, but it's buggy. debugging

    @monatis
    monatis committed Oct 3, 2023
    Copy the full SHA
    d37ed47 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2023

  1. LLaVA is working e2e, needs to optimize memory allocation + cleanup

    @monatis
    monatis committed Oct 7, 2023
    Copy the full SHA
    8690f42 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2023

  1. Use ggml_allocr + rm unnecessary code

    @monatis
    monatis committed Oct 8, 2023
    Copy the full SHA
    94eeac3 View commit details
    Browse the repository at this point in the history

  2. fix: crlf -> lf

    @monatis
    monatis committed Oct 8, 2023
    Copy the full SHA
    0c2bd79 View commit details
    Browse the repository at this point in the history

  3. fix: new line at EoF

    @monatis
    monatis committed Oct 8, 2023
    Copy the full SHA
    204d08b View commit details
    Browse the repository at this point in the history

  4. fix: trailing whitespace

    @monatis
    monatis committed Oct 8, 2023
    Copy the full SHA
    95da79e View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' into llava

    @monatis
    monatis committed Oct 8, 2023
    Copy the full SHA
    2a04d0b View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2023

  1. Add readme

    @monatis
    monatis committed Oct 9, 2023
    Copy the full SHA
    444dbce View commit details
    Browse the repository at this point in the history

  2. Update readme

    @monatis
    monatis committed Oct 9, 2023
    Copy the full SHA
    8af7e21 View commit details
    Browse the repository at this point in the history

  3. Some cleanup

    @monatis
    monatis committed Oct 9, 2023
    Copy the full SHA
    54495c9 View commit details
    Browse the repository at this point in the history

  4. Are you happy editorconfig?

    @monatis
    monatis committed Oct 9, 2023
    Copy the full SHA
    9b0ec4d View commit details
    Browse the repository at this point in the history

  5. rm unused batch image preprocessing

    @monatis
    monatis committed Oct 9, 2023
    Copy the full SHA
    8278a73 View commit details
    Browse the repository at this point in the history

  6. rm unused import

    @monatis
    monatis committed Oct 9, 2023
    Copy the full SHA
    d78e816 View commit details
    Browse the repository at this point in the history

  7. fix: rm designated initializers

    @monatis
    monatis committed Oct 9, 2023
    Copy the full SHA
    4759bfd View commit details
    Browse the repository at this point in the history

  8. introduce pad-to-square mode for non-square images

    @monatis
    monatis committed Oct 9, 2023
    Copy the full SHA
    325d240 View commit details
    Browse the repository at this point in the history

  9. are you happy editorconfig?

    @monatis
    monatis committed Oct 9, 2023
    Copy the full SHA
    d75a031 View commit details
    Browse the repository at this point in the history

  10. gitignore /llava

    @monatis
    monatis committed Oct 9, 2023
    Copy the full SHA
    ae01c85 View commit details
    Browse the repository at this point in the history

  11. Handle cases where image file does not exist

    @monatis
    monatis committed Oct 9, 2023
    Copy the full SHA
    5009ae9 View commit details
    Browse the repository at this point in the history

  12. add llava target to Makefile

    @monatis
    monatis committed Oct 9, 2023
    Copy the full SHA
    96171de View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2023

  1. add support for 13b model variant

    @monatis
    monatis committed Oct 10, 2023
    Copy the full SHA
    d640aae View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2023

  1. Maybe seed is unlucky?

    @monatis
    monatis committed Oct 11, 2023
    Copy the full SHA
    587bde8 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into llava

    @monatis
    monatis committed Oct 11, 2023
    Copy the full SHA
    f1564bb View commit details
    Browse the repository at this point in the history

  3. Check if apples are compared to apples

    @monatis
    monatis committed Oct 11, 2023
    Copy the full SHA
    ab21587 View commit details
    Browse the repository at this point in the history

  4. are you happy editorconfig?

    @monatis
    monatis committed Oct 11, 2023
    Copy the full SHA
    0409ae0 View commit details
    Browse the repository at this point in the history

  5. Use temperature = 0.1 by default

    @monatis
    monatis committed Oct 11, 2023
    Copy the full SHA
    f0f7834 View commit details
    Browse the repository at this point in the history

  6. command line: use gpt_params_parse()

    @monatis
    monatis committed Oct 11, 2023
    Copy the full SHA
    2bc1710 View commit details
    Browse the repository at this point in the history

  7. Merge master and fix conflicts

    @monatis
    monatis committed Oct 11, 2023
    Copy the full SHA
    1403d87 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2023

  1. minor

    @monatis
    monatis committed Oct 12, 2023
    Copy the full SHA
    dc913ea View commit details
    Browse the repository at this point in the history

  2. handle default n_predict

    @monatis
    monatis committed Oct 12, 2023
    Copy the full SHA
    56ccf97 View commit details
    Browse the repository at this point in the history

  3. fix typo

    @monatis
    monatis committed Oct 12, 2023
    Copy the full SHA
    e9534ea View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into llava

    @ggerganov
    ggerganov committed Oct 12, 2023
    Copy the full SHA
    346e3c1 View commit details
    Browse the repository at this point in the history

  5. llava : code formatting, rename files, fix compile warnings

    @ggerganov
    ggerganov committed Oct 12, 2023
    Copy the full SHA
    4bc5c9c View commit details
    Browse the repository at this point in the history

  6. do not use Wno-cast-qual for MSVC

    @monatis
    monatis committed Oct 12, 2023
    Copy the full SHA
    0bd7e69 View commit details
    Browse the repository at this point in the history