🏆 Awarded the 'Most Innovative Approach' at NivHack 2023 hosted at Niveus Solutions Pvt. Ltd., Mangalore.
![](https://private-user-images.githubusercontent.com/83623339/263758712-a36de8a0-41f1-43bf-93b4-51e2516d046d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM4NzU2MTMsIm5iZiI6MTcyMzg3NTMxMywicGF0aCI6Ii84MzYyMzMzOS8yNjM3NTg3MTItYTM2ZGU4YTAtNDFmMS00M2JmLTkzYjQtNTFlMjUxNmQwNDZkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE3VDA2MTUxM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWY4NjI3ZWY0OWRiZjE0OTI0MjQyYjRmMTZmNGUwNzg1Y2IyN2UwOTk0NDVhMWNjYTEzMzE4MjQ0M2IwNTUxNDcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.66xmArb4kY_M3PaRFolhocXnWfVHdP2QJyScHoiYJ90)
- Live URL
- VertexAi API Microservice was built with Flask -- Repo
- Both of the repos were dockerized & deployed on Google Cloud Run
- Final Presentation Pitch Deck
NOTE The deployments might be down right now as the Cloud access was given only during the hackathon.
- Virtually Interactive 3D Character with speech & hand gestures
- Multilingual Chat Sessions in different languages
- Follow-up Questions based on previous conversation
- Persistent chat sessions considering chat history of user
- Personalised medical query & mental health support
- Multilingual Voice Chat Support
- Google PaLM2 based on model chat-bison using Google vertex AI
- Deployed both client & server on Google Cloud Run
- Database on Cloud SQL (PostgreSQL)
- Google OAuth
- Deepgram STT & ElevenLabs TTS
- CI/CD using Google Cloud source repositories & Cloud build
- Google Translate
- T3 Stack: Next.js, tRPC, Prisma, Typescript
- Flask Backend API
- No finetuning available for chat-bison model
- It was our first time using & configuring GCP & Gitlab
- No Direct support of CI/CD of Gitlab on Cloud Run
- Didn't get access to Google TTS & STT due to Quota Limit
- The TTS generation & lipsyncing had a lot of latency
- MedPaLM integration for better accuracy.
- Using 3D model for other usecases like kids' trainer, lifestyle expert etc.
- Break the sentences according to fullstops & punctuations before processing further (to avoid latency)
- Can extend the concept to Metaverse domain
- Nagaraj Pandith
- Swasthik Shetty
- Nidheesha T