Unit 4: Fusion Text and Vision - Tasks and Models for Image and Text. #151
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added VLM Introduction, pushing to main for reviews!
SuryaKrishna02
requested review from
merveenoyan and
MKhalusova
as code owners
| @@ -0,0 +1,36 @@ | |||
| # MultiModal Tasks and Models Part - 1 | |||
|
|
|||
| In this section, we will briefly look at the different multimodal tasks involving Image and Text modalities, and their corresponding models. Before diving in, let's have small recap on what is meant by "multimodal" which was covered in previous sections. The human world is a symphony of diverse sensory inputs. We perceive and understand through sight, sound, touch, and more. This multimodality is what separates our rich understanding from the limitations of traditional, unimodal AI models. Multimodal models, drawing inspiration from human cognition, aim to bridge this gap by integrating information from multiple sources, like text, images, audio, and even sensor data. This fusion of modalities leads to a more comprehensive and nuanced understanding of the world, unlocking a vast range of tasks and applications. | |||
There was a problem hiding this comment.
Really nice intro 😄
|
|
||
| - **Text-to-Image generation:** Imagine a magical paintbrush that interprets your words and brings them to life! Text-to-image generation is like that, it transform your written descriptions into unique images. It's a blend of language understanding and image creation, where your text unlocks a visual world from photorealistic landscapes to dreamlike abstractions, all born from the power of your words. | ||
|
|
||
| ## Visual Question Anwering (VQA) and Visual Reasoning |
There was a problem hiding this comment.
Excited to read the upcoming sections 😄
There was a problem hiding this comment.
nicely connected to real world. :)
|
|
||
| In this section, we will briefly look at the different multimodal tasks involving Image and Text modalities, and their corresponding models. Before diving in, let's have small recap on what is meant by "multimodal" which was covered in previous sections. The human world is a symphony of diverse sensory inputs. We perceive and understand through sight, sound, touch, and more. This multimodality is what separates our rich understanding from the limitations of traditional, unimodal AI models. Multimodal models, drawing inspiration from human cognition, aim to bridge this gap by integrating information from multiple sources, like text, images, audio, and even sensor data. This fusion of modalities leads to a more comprehensive and nuanced understanding of the world, unlocking a vast range of tasks and applications. | ||
|
|
||
| ## Examples of Tasks |
There was a problem hiding this comment.
From my previous experience, try to avoid heavy words like delve and keep the words as simple as possible for easy reading.
|
@SuryaKrishna02 amazing read, nice introduction and well written 🤗 |
merveenoyan
reviewed
Dec 20, 2023
|
@merveenoyan Thanks for your comments. I have made those changes and completed the rest of the section. Looking forward to your review. |
johko
approved these changes
Dec 25, 2023
Co-authored-by: Johannes Kolbe <2843485+johko@users.noreply.github.com>
|
@SuryaKrishna02 can you fix merge conflicts and we can merge? |
|
@merveenoyan Fixed the merge conflicts. |
9 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hey everyone!
This PR adds the Part 1 of the Second Section on Fusion of Text and Vision for Unit 4: Multimodal Models. introducing the Multimodal Tasks and Models involving Image and Text.
Related to Issue: #54
Best,
Fusion of Text and Vision Team.