MLOps Survey: Preparation - Large Language Models (LLM) - 2023.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOAdVdT9pLNHBrdTuXjiQq/",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/fmind/mlops-survey-chatgpt-2023/blob/main/MLOps%20Survey%3A%20Preparation%20Large%20Language%20Models%20(LLM)%20-%202023.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# FIELDS"
      ],
      "metadata": {
        "id": "G_S3FtvK0FTa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "OPENAI_API_KEY = \"\" #@param {type:\"string\"}\n",
        "OPENAI_MODEL = 'gpt-3.5-turbo' #@param {type:\"string\"}\n",
        "TEMPERATURE = 0 #@param {type:\"number\"}"
      ],
      "metadata": {
        "id": "K6-1MQji0-Yg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5A0CcAP41T3l"
      },
      "outputs": [],
      "source": [
        "RESULTS = '''What is your position/title at your company?,How big is your organization? (number of employees),Are you using LLM at in your organization?,What is your use case/use cases?,Have you integrated or built any internal tools to support LLMs in your org? If so what?,What are some of the main challenges you have encountered thus far when building with LLMs,What are your main concerns with using LLMs in production?,how are you using LLMs?,What tools are you using with LLMs?,What areas are you most interested in learning more about?,How do you deal with reliability of output?,Any stories you have that are worth sharing about working with LLMs?,Any questions you have for the community about LLM in production?,What is the main reason for not using LLMs in your org?,What are some key questions you face when it comes to using LLM in prod?,have you tried LLMs for different use cases in your org?,\"If yes, why did it not work out?\",Any stories you have that are worth sharing about working with LLMs?,Any questions you have for the community about LLM in production?\n",
        ",,,,,,,,,,,,,,,,,,\n",
        "Data scientist,\"1,000+\",No,,,,,,,,,,,Production takes investment,Cost to maintain the service,Yes,Date quality,N/a,\n",
        "ML Engineer,50-500,Yes,Text summarisation,No,\"Current infrastructure not designed for such massive models, having to implement workarounds for quick fixes\",Cost,\"Open source model (GPT-J, etc)\",Seldon,Inferenece for LLMs,,,,,,,,,\n",
        "Founder,1-10,Yes,Data annotation; summarization; search,\"https://chat.mantisnlp.com, internal annotation tools\",\"out of date info, hallucinations, cost, difficulty in deploying on own infrastructure\",they could serve nonsense or out of date information,Open AI API,,Working with OSS LLMs,we provide links to the context from which answers have been drawn.,,,,,,,,\n",
        "Owner,1-10,Yes,Content production and brainstorming,Not yet,not yet,cost of openai API,Open AI API,\"Looking at Pinecone, weaviate, langchain, hugging face\",Embeddings,temperature = 0,,,,,,,,\n",
        "Senior Principal Scientist,\"1,000+\",Yes,Healthcare,\"yes, confidential\",reliability,cost of openai API,In house model,,Working with OSS LLMs,,,,,,,,,\n",
        "Head of Machine Learning,50-500,No,,,,,,,,,,,It's only just now becoming feasible with the ChatGPT API. Before it was too expensive and quality was not good enough.,How to make them reliable.,\"We are still exploring, eg for support in diagnosis of plant diseases.\",So far it's not reliable enough and we don't have spent enough time preparing special content for it. Like rules it should use during diagnosis. The data ChatGPT was trained on seems a bit hit and miss for us.,,\"Anyone have some experience with getting better reliability out of ChatGPT? Anyone succeeded in fine-tuning and getting better quality than ChatGPT? Thinking of llama for example.\n",
        "How can you limit ChatGPT so users will not ask it totally random stuff?\"\n",
        "Senior Machine Learning Engineer,\"1,000+\",Yes,\"Search, classification, NER\",Just a bunch of shitty scripts I need to refactor in the future.,\"Size in all regards. Also training time, response time and that multi gpu debugging is weird.\",\"Whether the cost involved for us justifies the gains we get, considering we mainly make money from ads and not a really good product.\",Other model provider API,\"Hugginface, sagemaker, openapi api, tensorflow.\",Working with OSS LLMs,\"Ill refer to the picture of the xkcd guy \"\"just stir the model until the numbers look right\"\".\",\"I am not sure what ranks as a LLM. Anything bert and beyond? Anyway, we once used setfit for a production task but found that performance was really tricky to debug. The proof of concept was fine. Production was a complete mess. To this day I dont know why- underlying hardware?\",I mainly wonder how everyone is monitoring these. ALSO. For my company GDPR is a big deal. My DPO shoots down our ideas more often than not or limits it to using the last three months of data. How do other people deal with this?,,,,,,\n",
        "Head of Product,10-50,No,,,,,,,,,,,No applications,ROI of more expensive models,Yes,No ROI,N/A,Interested in results of survey\n",
        ",50-500,Yes,,oi,lik,il,Open AI API,loi,Working with OSS LLMs,lo,lo,lo,,,,,,\n",
        "Senior Data Engineer,50-500,Yes,\"text embeddings for search, text generation to help the business people do their job\",\"Yes, we're using OpenAI for a couple of internal tools.\",,Cost. But our issues with productionisation is more general than just for LLMs,Open AI API,streamlit,Inferenece for LLMs,\"All internal tools, reliability is not a major concern for now\",,,,,,,,\n",
        "Machine learning engineer,\"1,000+\",Yes,\"Natural Language interface, domain specific content creation\",,Prompt debugging! We need a new form of debugger.,They are erratic,Open AI API,,Working with OSS LLMs,We have control systems,,,,,,,,\n",
        "Senior Data Scientist,\"1,000+\",Yes,Chatbots and Text classification,One of our use cases is to build text classification models for each of our customers (we are a B2B company). Achieved this using LLMs and some engineering,,,\"Open source model (GPT-J, etc)\",,Embeddings,,,,,,,,,\n",
        "ML Engineer,50-500,No,,,,,,,,,,,,,,,,\n",
        "Snr MLOps Engineer,50-500,Yes,Adapters,\"Yes, training setup as well as inference setup\",Latency and deployment time,Resources usage along with latency and deployment time,\"Open source model (GPT-J, etc)\",,Inferenece for LLMs,Assessment at evaluation,,,,,,,,\n",
        ",\"1,000+\",Yes,,,,,,,Inferenece for LLMs,,,,,,,,,\n",
        "Senior ML engineer,50-500,No,,,,,,,,,,,\"Two main reasons: 1. Very compute & power hungry 2. Cannot be relied upon as it will very confidently return incorrect information or in “existent facts” which is problematic for use in my industry.\n",
        "\n",
        "Eg it took us less than 3mn to get BioGPT to tell us vaccines cause autism\",\"What uses cases does it have for our business? how much gain do we get from using an LLM vs a simpler language model where we can control fact and outputs? How much will it cost (to deploy, and run)\",We abandoned the idea pretty quickly,We got terrible suggestions from the LLM that were false information and would be detrimental to the business,Ask BioGPT about vaccines and autism and it’ll confidently tell you that vaccines cause autism spectrum disorders in a small percentage of children (yikes),How do you force an LLM to only return factual information ?\n",
        "Machine Learning Engineer,50-500,No,,,,,,,,,,,\"We use medium-sized language models (e.g. RoBERTa-base) as the backbone for our deployed NLP models because they can perform inference on CPU, which is cheap. We don't use LLMs (e.g. ChatGPT, GPT-3) because the APIs are expensive, and fine-tuning them to perform NLP tasks other than language generation (e.g. NER, open-set intent classification) is not straightforward.\",How can we ensure the same version of the model is being used? How can we supply a random seed to get deterministic results? How can we minimize LLM API usage costs? How can we fine-tune a LLM on a niche task without having to replicate the weights of the LLM? How can we deploy our own copy of an open-source LLM without breaking the bank?,Yes,\"We evaluated a LLM (GPT-3) against RoBERTa fine-tuned on our small task-specific dataset, and our fine-tuned model got much higher accuracy than GPT-3 did.\",None,\n",
        ",\"1,000+\",No,,,,,,,,,,,Not much usecases for us,,,,,\n",
        "Founding Data Scientist,1-10,Yes,\"Summarization, topic extraction, root-cause analysis\",\"Not much, just some prompt storage and an automated retries to deal with different failure cases.\",\"Hallucinations are a big one. In general, problems are hard to debug without careful manual review, so we're only using them when the amount of results generated is small enough to review by hand.\",\"Reliability, cost, injecting domain-specific language and understanding into them\",Open AI API,,Embeddings,\"Automated retries for some simple & obvious failures, manual review for everything else\",,Where are others drawing the line between training models locally and using LLMs? Have others had success with using LLMs to bootstrap training data for smaller local models?,,,,,,\n",
        "CTO,1-10,Yes,Marketing,Not yet,Still developing,Cost,Open AI API,,Fine tunning LLMS,,,,,,,,,\n",
        "Machine Learning Engineer,\"500-1,000\",No,,,,,,,,,,,Not applicable to the domain,Not yet,,,,Can you actually compete with the giants for a good language model?\n",
        "Head of AI,\"1,000+\",Yes,Scanning large text corpora using an LLM and some cases with text summary for now,Not (yet).,It’s a mix of taming a beast and using alchemy,Consistency of the output quality,Other model provider API,./.,Working with OSS LLMs,HITL,They take every Input equally serious which makes for some great laughs…,,,,,,,\n",
        "Product,\"1,000+\",No,,,,,,,,,,,Production LLM lifecycle and use cases not clear,Data governance,no,,,\n",
        "Lead Data Scientist,50-500,No,,,,,,,,,,,Cost / right use-case,,,,,\n",
        "Co-founder,1-10,Yes,AI for e-commerce,No,Hardware requirements and AI accelerators compatibility,Inference scalability,\"Open source model (GPT-J, etc)\",\"PyTorch, AWS SageMaker, OpenAI's API, Midjourney, CLIP/BLIP/etc models\",Inferenece for LLMs,,,\"Any opinions regarding the newer AI accelerator chips? Many people fighting for GPUs, but not necessarily mentioning AWS Inferentia 2 chips, TPUs, Habana Gaudi, etc.\",,,,,,\n",
        "MLE,50-500,Yes,,dbt pre-processing to get the text right for the prompts and test it,\"explainability, expense, embeddings\",see above,Open AI API,metaflow,Embeddings,,,,,,,,,\n",
        "Snr MLE,50-500,No,,,,,,,,,,,Exploration stage,Putting company data in the right structure and updating the model with it,yes,getting fine tune right and LLMops pipeline,,Is it cheaper to host your own LLM on AWS or just use OpenAI services? With respect to scaling and potential user base of 5000\n",
        "Director of Artificial Intelligence,\"1,000+\",Yes,\"Novelty - ChatGPT is being explored to optimize code, critique written works, etc.\",No,N/A,Complacency and atrophy of critical thinking,\"Open source model (GPT-J, etc)\",Microsoft Azure OpenAI services,Fine tunning LLMS,Educate others on the limitations and proper use.,\"One of our citizen data scientists now (figuratively) \"\"worships\"\" ChatGPT Gods given its capabilities. LOL.\",None at the moment given our priorities are mostly with computer vision and time-series data analysis.,,,,,,\n",
        "Data Scientist,50-500,Yes,\"Summarisation, Email generation, Information retrieval\",Modular python library for interacting with multiple LLM providers/APIs and a standardized prompt generation framework.,Consistency of formatting in freeform LLM text generation. Subtleties in prompt engineering.,\"Detecting truthfulness, hallucination. Output validation. Risk of monopolisation and centralised access to LLM as a service\",Other model provider API,Python,Working with OSS LLMs,,,,,,,,,\n",
        "Co-founder and CTO,1-10,Yes,Prompt autogeneration and synthetic data generation,,Reproducibility,Hallucination and latency,Open AI API,HoneyHive,Fine tunning LLMS,\"Multi-responses shown to user side by side, ghost writer style dropout\",,How are you thinking about tracking user feedback in production?,,,,,,\n",
        ",50-500,Yes,,,,,,,Working with OSS LLMs,,,,,,,,,\n",
        "Senior Software Enginner,50-500,No,,,,,,,,,,,No business need yet,,,,,\n",
        "Co-founder,1-10,No,,,,,,,,,,,We're learning,,,,,\n",
        "ML Developer,\"500-1,000\",Yes,Question Answering,yes,latency,\"latency, factuality\",\"Open source model (GPT-J, etc)\",,Inferenece for LLMs,,,,,,,,,\n",
        ",1-10,No,,,,,,,,,,,\"Cost, Latency, Accuracy\",\"Cost can be high, hallucinations\",,,,\n",
        "data scientist,\"500-1,000\",No,,,,,,,,,,,not really in our domain,N/A,no - closest we've got to dealing with language is fuzzy lookups,N/A,Nope - but I'm excited to learn more,\n",
        "Senior Machine Learning Engineer,10-50,Yes,Question generation for accountants,built internal,reducing latency,not knowing the data used for the initial training,In house model,\"TensorFlow, TFX\",Inferenece for LLMs,yes,,,,,,,,\n",
        "Founder,1-10,Yes,Zero-shot/Few Shot classification for data enrichment,\"Internally, we use BERT to create embeddings, then train a smaller model on customer data.\",Infrastructure costs.,\"Is what I;m building getting obsolete in 3 months,\",In house model,In house Bert (from Huggingface) + Open AI,Fine tunning LLMS,I am stuck with this right now - can I integrate these into the UI for AI Hero.,ChatGPT API output is different every time for the same input - makes it hard to create a reliable service.,How are you managing training/deployment of these?,,,,,,\n",
        "SRE,\"1,000+\",Yes,analyze logs,no,Analyzing results,security,In house model,in house built tool,Fine tunning LLMS,Very manual so far.,confidential,no,,,,,,\n",
        "\"Cofounder (normally product manager, but also wearer of many hats)\",1-10,No,,,,,,,,,,,\"1. gcm diagflow is good for complete basic dialogs, 2. halucinations 3.a. how to make answers be focused on what my company is doing rather than general answers, 3.b. how to keep that evolving as we learn more about the customers 3.c. how to prevent a practical joker who's conversing with my llm to inject some practical joke into it 4. if im relying on openai-s llm, i need to always get a reply what happens if they're busy, whats the proverbial 'failover'\",how to get it to use my company's specific knowledge,not thoroughly enough yet,still working on it,openai - even with monthly subscription it is not predictable output,\"do you use it for chatbots, how do you put guardrails on it for conversations that are going off track or how do you prevent practical jokes if you're re-incorporating the knowledge back into the llm\"\n",
        "Data Architect,\"1,000+\",Yes,Search and chat on proprietary data,No,\"Automation amd scaling of transfer learning, tuning and reinforcement learning\",Scaling,Open AI API,\"Lang chain, k8s, gcp\",Learning from new/private data,Not a concern. In-house,No,\"Best automated/scalable pipeline for tuning,, transfer learning (augmentation), reinforcement feedback\",,,,,,\n",
        "Senior software engineer,\"1,000+\",No,,,,,,,,,,,No need atm,Reliability,,,,\n",
        "consultant,1-10,No,,,,,,,,,,,we'll be using at a startup I'm part of,cost and fit for purpose. Actual market validation,yes. I've used them for classification,they are in prod for that use case,,\n",
        "CTO,50-500,Yes,\"Product (Software) recommendations, Internal Q&A, Marketing content, and Customer support\",\"Langchain, OpenAI Playground, ChatGPT, PromptLayer, Weaviate\",Rapidly changing APIs,Hallucinations misleading customers,Open AI API,PromptLayer,Embeddings,\"Pre/post-processing, constitutional / self-auditing, customer feedback thumbs up/down, custom embedding matrix\",,How might you avoid siloing all of your embeddings when different teams need different tools without building it all in house yet all would benefit from the combined collection?,,,,,,\n",
        "Data scientist / co-founder,1-10,Yes,Information retrieval and text generation.,\"We have some output validation methods and templates, basically regex matching prompts.\",Input length limitation and having to work around it with non-optimal methods.,Hallucinations,Open AI API,\"Langchain, Huggingface, Polars\",Fine tunning LLMS,We implement checking with less stochastic tools.,Content language matters a lot!,,,,,,,\n",
        "Analytics engineer,50-500,Yes,Chatbot,In POC at the moment,Vectorizing CRUD and ci / CD,To get better at semantic search,Open AI API,\"Langchain, pinecone, slack and aws\",Embeddings,,,\"How to better deploy python apps using aws and operationalizing it.\n",
        "How to better manage vector indexes.\",,,,,,\n",
        "\"Director, Innovation\",50-500,No,,,,,,,,,,,Data privacy concerns,Difficulty or ease of maintenance; costs; data privacy,Not yet,na,None yet,Are there any standards/best practices to deal with data privacy?\n",
        "ML & MLOps practice lead,50-500,Yes,\"Text classification, text generation\",,\"Compute performance, fine tuning challenges, explainability\",,\"Open source model (GPT-J, etc)\",,Inferenece for LLMs,,,,,,,,,\n",
        "Founder,1-10,Yes,Social media idea generation,No just used langchain and OpenAI,Prompt engineering; handling very large documents,Bad data/completions; runtime errors parsing completions (we use regex to parse each line ),Open AI API,LangChain,Fine tunning LLMS,We don’t do anything right now,,,,,,,,\n",
        "CTO,1-10,Yes,Semantic data extraction and corporate governance,\"I'm unable to utilise LLM's currently due to data security restrictions, however as a OpenAI beta tester, around 2 years ago I created a business Operational Delivery Framework for a £500M nuclear project. I then transferred the learning framework to £5B project.\",Environment dependency conflicts,Safe and ethical usage,Open AI API,Use case dependent,Fine tunning LLMS,\"AI is replaced by HI(Human) for final check, always.\",\"Yes, the ODF I created was regarded as best in class by regulators and has been suggested to be implemented as new gold standard. I used GPT3 to advise on the design of the framework, and to write all the necessary code to create it.\",\"Nope, no questions, just a great big thankyou, and a virtual high five for all the awesome peeps 🙏\",,,,,,\n",
        "MLOps Engineer,50-500,No,,,,,,,,,,,Still evaluating infrastructure requirements and ways to make it traceable if things go wrong,\"Retraining, tackle hallucinations and avoid wrong results\",No,,,\n",
        "Junior developer,1-10,Yes,Creating a chat bot to query data efficiently for managers,Not yet but its in production. Using langchain.,\"The main challenges are using the different tools from langchain to optimise for the use case. There are a lot of things to consider. Also, we use ts and angular front end so not having all the tools that are available in python required us to have python as the backend now.\",It’s how useful/valuable it will actually be for the client.,Open AI API,Langchain/ts/csharpe/angular.,Fine tunning LLMS,We havent gotten to that yet.,\"They are amazing. But as mentioned, a chatbot alone is probably not enough to be useful, hence building out other capabilities will become important.\",,,,,,,\n",
        "deep learning engineer,\"1,000+\",Yes,extract information from image and text of medical domain,no,deploy models into production,hight training & inference cost,\"Open source model (GPT-J, etc)\",\"transformers, triton inference server\",Working with OSS LLMs,,,,,,,,,\n",
        "Co-Founder,10-50,No,,,,,,,,,,,Lack of use cases that require so much sophistication,\"cost of maintenance, lack of institutional knowledge\",No,,\"Have previously tried using them, however, in many cases, LLMs aren't all that reliable. I tried to use it as a chatbot for a client but it didn't work as expected in many cases which is why didn't end up using it in production.\",None\n",
        "Intern,\"1,000+\",No,,,,,,,,,,,Because I am in education sector.,when you use LLM in production it can fail any time,NO,,,\n",
        "CTO,1-10,Yes,Coding,VSCode Extension,Work with 3 AI providers at the same time,The wrong respond from the model,Open AI API,Code GPT https://codegpt.co,Embeddings,Now it is just responds tickets in github,\"The extension hace more than 200,000 downloads and it work great!\",,,,,,,\n",
        "Software Development Engineer,\"1,000+\",Yes,\"(Not in my org, at work I don’t do this, it’s personal) Cover letter generator based on resume and job description\",,Embedding with appropriate data effectively,Being dependent on one company for the API so far. Having models of equivalent performance whose weights were public would be a great relief,Open AI API,\"Gpt index, langchain\",Embeddings,\"Prompt engineering, allow user to be specific, and of course re-generate\",,,,,,,,\n",
        "Data Science Team Lead,10-50,Yes,Topic sentiment evaluation on text,,Managing training data labeling,explainability,\"Open source model (GPT-J, etc)\",AWS Sagemaker,Fine tunning LLMS,Test each new model candidate on test set,,,,,,,,\n",
        "Sr. ML Eng,50-500,No,,,,,,,,,,,inference cost vs. impact on revenue,how to reduce inference cost,,,,\n",
        "Senior AI Expert,50-500,Yes,Medical Diagnostics Support,\"No, we rely on open source tools\",\"Explainability, Testing, Getting ground truth data\",Trustworthiness,\"Open source model (GPT-J, etc)\",\"Milvus DB, FastAPI, Hugginface, Pytorch, etc.\",Working with OSS LLMs,We try to make the output as transparent as we can (using explainability techniques) and not relying upon LLM for the end decision (these are used as a support tool),,,,,,,,\n",
        "Data Science Engineer,50-500,Yes,Assistant chat bot,\"Langchain. Will serve the bot from a flask server, being set up for the bot, into our existing java platform\",\"Langchain LLM Parse error. Turbo not using multiple tools for some reason, when davinci-003 did\",\"Need HIPAA compliance, so trying to get on Azure. Worried about the contract falling through\",Open AI API,Langchain. All my agent’s tools are custom-built. I hit our internal APIs with them,I want deep knowledge on how to make the ReAct pattern work in various edge cases,\"Will have friendly internal users using it first in the course of serving customers, and giving us per-message feedback\",davinci-003 hallucinating using an existing tool without actually using it!,How should I split my initial prompt up between turbo’s System prompt and the first Human prompt I give it?,,,,,,\n",
        "Head of Data and ML,10-50,No,,,,,,,,,,,\"No Use case, being a fintech startup, besides helping with processing documents\",why waste the money,no,,,\n",
        "ML/MLOps Engineer,\"500-1,000\",No,,,,,,,,,,,No clear use case,\"Running them, fine tuning, cost\",Nope,,,\n",
        "\"Data Scientist, HappyFresh\",\"500-1,000\",No,,,,,,,,,,,Not,,,,,\n",
        "Data Scientist,10-50,Yes,Automation of report writing.,\"Yes.\n",
        "Responsible for the design, building, testing and deployment of a GPT-3.5 application that processes numerical data and free-response text in tabular form to generates commentary and analysis for a visual report. Researchers are only required to upload a zip folder(1-click).\n",
        "Technologies: AWS lambda, Docker (ECR/ECS), GPT3.5, PyTorch.\",\"control of the output for our usecase. Reports need to understand the client's intent and take information from briefing calls and key erquriements. These can help guide the output, but now always.\",Thta it will hallucinate something that we won;t pick up in the report editing phase,Open AI API,Torch and Pandas,Prompt Engineering (Riley Goodside),Have editors comb through it,We aim to turn a report around from survey finishing to client inbox in 24 hour. So far we are at 48 hours.,,,,,,,\n",
        "Data Scientist,50-500,Yes,\"LLM in Marketing, Custom data\",HF Inference Endpoints,Cloud Cost,No proper documentation,Other model provider API,,Fine tunning LLMS,,,,,,,,,\n",
        "Senior Data Scientist,\"500-1,000\",No,,,,,,,,,,,\"We don't have a need for generating unstructured data, we work in small languages, our text is very domain specific, LLMs are very compute intensive -> expensive to run and take a lot of work to set up.\",\"Cost, latency, actual performance in the languages / domains we work with (real languages, not programming)\",Nope. Tried some BERT for classification but that hardly counts as an LLM. And I've made models as big as BERT I think.,,,Why are most of the models on Hugging Face based on WordPiece and BPE tokenizers? Give me bytes or give me death\n",
        "Data Scientist,50-500,Yes,chatbot/voice assistant,just used RASA for chatbot creation and haystack for open domain question answering,\"currently used only pre-trained models, biggest challenge was to obtain data for specific domain fine tuning\",it is very important in my opinion to accurately monitor the kind of usage to prevent frauds or malicious intentions,Other model provider API,Haystack,Fine tunning LLMS,\"it depends from specific context and specific use cases, in my experience I rely on validation set and a validation tool that estimates accuracy of answers based on answers manually scrutinized by humans\",not so far,it would be interesting a discussion about how to control the output and in general the kind of usage the system has been exposed,,,,,,\n",
        "Head of Data Science,\"1,000+\",Yes,\"Entity matching, customer service responses (souped up/targetted FAQ).\",yes - high scale server on K8's,K8's caches... TESTING.,\"cost, latency, truthfulnes - but also\n",
        "\n",
        "Maybe management in the sense of versioning and resolving / attributing issues?\n",
        "\n",
        "Copyright and intellectual property liability - who knows what it's been trained on and what it might spit out... What happens if it provides a Disney script for an advert? (I know the answer - nothing good).\n",
        "\n",
        "Safety : what if it tells someone to kill themselves? Or spits out a recipe to make smallpox? (I am being kind of silly here, but... not entirely)\",\"Open source model (GPT-J, etc)\",\"not sure what this means - we use k8's and cloud functions to run them, also interface to case management solutions.\",calibration of llm output / calbration and testing of llms,badly. We do lots of testing but I constantly feel we are on shaky ground.,\"We had a nice time selecting models, proving it would all work and using TSNE and PCA to reduce the dimensionality of embeddings and get compression for our vector lookup system... Very good and satisfying work, we got prizes and praising from all. Then camesome dreadful times making our vector lookup system work under load for long durations in a proper prod environment (where we couldn't touch it to do things like restart it or reload the index). I think this will be a big issue in the future - we need things that are as reliable as oracle for running the inferences, we do not have them - everything is glued together (I am guessing this is not true if you work for Google...)\",,,,,,,\n",
        "Founder,1-10,Yes,We are making tools to monitor and improve the performance of LLMs,No additional tools,,Hallucinations,\"Open source model (GPT-J, etc)\",UpTrain,Embeddings,We are building a tool for that,,How is everyone thinking about fine-tuning LLMs for their use-cases?,,,,,,\n",
        "CEO,1-10,Yes,\"Embeddings, Feature Extraction, Chatbot\",Database connections,Context length limitations,\"Hallucinations, rate limiting,\",Open AI API,\"Langchain, Llama Index, Deep lake, Weaviate\",Fine tunning LLMS,Error handling,,,,,,,,\n",
        "Founder,1-10,Yes,\"Text summarization, code generation\",No,Dealing with vector stores,Keeping agent on track with system (OpenAI’s ChatGPT) or role,Open AI API,LangChain,Embeddings,N/A,N/A,What are some of the best practices for dealing with LLMs or foundation models?,,,,,,\n",
        "Senior Dara scientist,\"1,000+\",Yes,\"Chatbot, semantic search\",Deploy open source LLM,\"Data privacy, big models\",Fake information,\"Open source model (GPT-J, etc)\",PyTorch huggingaface,Inferenece for LLMs,Generate multiple output,Not really,Chatgpt for domain knowledge,,,,,,\n",
        "ML Engineer,\"1,000+\",Yes,Customer support,We’ve integrated with model store!,Memory requirements during inference,Slow start up times/it’s very different from regular code,In house model,Hugging Face,Inferenece for LLMs,We’re doing classification not generation,,Curious to hear about other use cases!,,,,,,\n",
        "Director,1-10,No,,,,,,,,,,,No need yet,I don't need them,\"Yes, too heavy\",Too heavy,Not really,\n",
        "Technical Marketing Manager,10-50,Yes,Content marketing and text drafting,not at the moment in my department,,costs under control and reliability,\"Open source model (GPT-J, etc)\",,Inferenece for LLMs,,,,,,,,,\n",
        "Head of MLOps,\"1,000+\",No,,,,,,,,,,,\"Currently assessing the risk and potential applications, performing POCs.\",\"What are the potential risks, where are the benefits worth the costs for enterprise grade integration, how do we make these things secure, how do we monitor?\",In the process of doing so.,We’re being thorough.,,Anyone any good resources on analysing the risks and how to monitor LLM based solutions?\n",
        "CDO,\"1,000+\",No,,,,,,,,,,,Wee are exploring the space but don't feel they are ready for production use yeet,\"How to pin them to a domain, how to verify the veracity of teh outcome and how to best chain them with APIs\",We are exploring customer facing and colleague facing applications,Concerns about sticking to a domain and about veracity if the answers,,What use cases do people have? what tooling do they recommend? how do they deal with the halucinations?\n",
        "Owner,1-10,Yes,\"sales pipelines, text generation, data classification and many more\",\"Yes, several\",The need to use Python,\"Cost, accuracy\",Open AI API,Zapier,Finetuning and embeddings,extensive testing,,,,,,,,\n",
        "Agricultue consultant,\"1,000+\",No,,,,,,,,,,,$,$,No,No,No,No\n",
        "Data scientist,1-10,Yes,,Yes,Computing power,,Open AI API,,Fine tunning LLMS,,,,,,,,,\n",
        "Product Owner Data Science,50-500,No,,,,,,,,,,,MLOps and DataScience rather young discipline. We are currently working on an LLM use case though,\"Runtime, Drift detection\",No,,,\n",
        "Team Lead DS & MLE,50-500,No,,,,,,,,,,,Domain specific challenges for what we do. Digging into how we could train LLM to fit the needs.,What is it? There is just the huge gap in understanding about analyzing text vs generating text.,We were using s-BERT in our primary recsys model. Recent modeling efforts have reset to simpler systems that are performing better. Still using it in a 2nd model for a clustering challenge.,,,\n",
        "Co-founder,1-10,Yes,Semantic vector embeddings as a feature into RecSys,Not yet,\"Sparse representations (e.g. tfidf) are easier to work with for our use-cases, because they survive simple aggregation better.\",Explainability of embeddings.,\"Open source model (GPT-J, etc)\",,Embeddings,We have come up with a large list of proxy metrics that we found correlate with performance on our tasks - so we track those.,\"As far as embeddings are concerned, we haven't found a clear advantage of LLMs vs simpler models (tfidf) trained on a relevant dataset. Both require elaborate preprocessing of the embedded documents for example.\",I'd love to connect with others who use LLMs embeddings and talk in more detail :),,,,,,\n",
        "Machine Learning Engineer,50-500,No,,,,,,,,,,,it isn't in focus on my leaders to use LLMs right now,Two key questions I do myself on LLM in prod are: 1 - How is the best strategy to train or fine-tune a LLM ? 2 - How to manage infrastructure costs ?,not yet,.,.,What tools are using to training or fine-tune theses models and a case on deploy a model as an API how has been the issues ?\n",
        "CEO,1-10,No,,,,,,,,,,,\"No current need, but also tooling\",\"Reliability, safety, certainty, hallucinations, etc\",No,,,\n",
        "Principal Cloud Developer,50-500,Yes,\"text resume, make ppts, text quizzes, text to flash cards, video resume etc,etc\",-,-,\"bad output, costs etc,etc\",Open AI API,-,Fine tunning LLMS,-,-,-,,,,,,\n",
        "CPO,1-10,Yes,We help organizations transition Excel reports to Python,\"Yes -- our product, Mito, is a spreadsheet that generates Python code as you edit it. We have integrated the chatgpt api into Mito so users can generate code using the LLM. Then, they can use the Mito spreadsheet to verify that the code it generated is correct.\",We are not yet sure how large enterprises are going to adopt LLMs. Our large enterprise customers are currently not using the Mito AI feature because there is too much uncertainty.,,Open AI API,Just the Mito spreadsheet,Figuring out how to deploy LLMs for enterprise use,We give users a spreadsheet interface to verify the output of the LLM. Spreadsheets are great for understanding changes to data!,,,,,,,,\n",
        "Cloud MLOps Engineer,10-50,Yes,Translate french text to english,Yes we have used opus-mt model in containerised environment for translation,It is pretty hard to work with this models as they need lot of resources to run which significantly increase the cost also,Need of resources,Other model provider API,Docker kubernetes aws resources,Inferenece for LLMs,,,,,,,,,\n",
        "CEO / Co-Founder,1-10,Yes,Writing SQL Queries from Natural Language.,No,Getting it to write a query was relatively easy. Getting it to write a query that would actually be correct AND execute is much harder.,My concerns at this point are primarily focused around getting the LLM to generate an executable query in all cases.,Open AI API,\"Apache Drill, Python.\",Fine tunning LLMS,,,,,,,,,\n",
        "Senior MLOps engineer,\"1,000+\",Yes,Adverse media about suppliers.,\"No, but have plans to do so! I will be involved in the feature store architecture.\",The unpredictable nature of responses.,That they give out results that are way off.,Other model provider API,\"Databricks, Azure\",Embeddings,Not well :D,None yet,,,,,,,\n",
        "machine learning engineer,\"1,000+\",No,,,,,,,,,,,Legal said not to,How do we keep costs down? How do we validate the outputs?,nope,,,\n",
        "CEO,1-10,Yes,Development of production tools for deployment/maintenance of LLMs and other models,We have a platform to develop/run models in production called Statfish,Accuracy and hallucinations,Can't rely on answers 100% of the time,\"Open source model (GPT-J, etc)\",\"Jupyter, kubernetes, haystack\",Inferenece for LLMs,Working on that problem,,,,,,,,\n",
        "ML engineer,50-500,No,,,,,,,,,,,currently using classic ML and I plan to introduce LLM capabilities to my org,\"cost of training and inference, time, gpu, ....\",no,,,\n",
        "Domain Chapter Lead - Data Science and AI,\"1,000+\",Yes,Summarisation of customer calls,nope,View of accuracy and quality of results,Reliability,Open AI API,Azure stack,Embeddings,no framework yet,not much from our side yet,How we manage quality of LLM outputs ?,,,,,,\n",
        "Senior Machine Learning Engineer,\"1,000+\",No,,,,,,,,,,,Lack of applicable business use cases and resources,How do you serve them?,Only for a developer,,,\n",
        "VP of ML,50-500,No,,,,,,,,,,,\"Expensive, no ROI, complex, dangerous\",Is it reliable? How do we know it's reliable?,Yes,Expensive and the business use case was poorly defined. Just a shiny toy,Giggles.,\"Use case examples would be interesting, especially where things have gone wrong.\"\n",
        "Technical Lead,50-500,Yes,\"POC, evaluation for healthtech\",n/a,supplement/support vs replace,user risk,In house model,hf,Fine tunning LLMS,cross fingers,\"not that i can share cheaply, ha\",\"Growing LLM features iteratively & coherently vs Alphabet's \"\"put LLM in all the things!\"\"\",,,,,,\n",
        "Solutions engineer,\"500-1,000\",Yes,Summarizes notes from meetings,no.,Not sure what's the standard,The output can be unpredictable,Open AI API,,Inferenece for LLMs,Human evaluation,,,,,,,,\n",
        "Platform Engineer,50-500,No,,,,,,,,,,,No clear use case for our product,,No,,,How do you decide where to use LLM?\n",
        "Senior MLOps,\"1,000+\",Yes,Chatbot,integrate,\"rate limit,\",\"privacy, trustfulness, latency\",Open AI API,vector databases,Inferenece for LLMs,eval,,,,,,,,\n",
        "Data science manager,\"1,000+\",Yes,classification,no custom tools,scaling :),\"great for batch, terrible for real time\",\"Open source model (GPT-J, etc)\",azure ml suite,Embeddings,human verification of sample of inferences,we fine tune a bert sized model with our data and run the inference in batch. We experimented with quantization and distillation and its pretty impressive how much you can do with much smaller models.,,,,,,,\n",
        "Data Scientist,\"500-1,000\",No,,,,,,,,,,,LLMs and their hallucinations just are not good for our current use cases (healthcare industry),,\"I have not, but I know many of the devs want to; there are some other reasonable use cases too.\",,,How do you get traditional engineers to understand that they shouldn't underestimate the complexity of getting ANYTHING ML-related into prod?\n",
        "Principal Data Scientist,\"1,000+\",Yes,\"Knowledge graph, transcription, topic modeling, embedding everything\",\"Yeah, but it's super secret until we try and sell it\",Slow. They give bad answers when you set the confidence requirements too high,Should we be paying for third party providers or train our own. Alpaca anyone?,Open AI API,\"Nvidia NeMO, Azure OpenAI\",Embeddings,Feedback button or manual review depending on use case,Microsoft's call summarization tool can capture a surprising degree of detail while hilariously completely missing the main point of the meeting,Best practices to prevent leakage for client facing models that employ online learning?,,,,,,\n",
        "Manager Machine learning,\"1,000+\",No,,,,,,,,,,,Less expertise,Large size models latency,,,,\n",
        "Data scientist,1-10,Yes,User facing chat application in a narrow domain.,In progress,,,Open AI API,,Inferenece for LLMs,,,,,,,,,\n",
        "Tech Lead,\"1,000+\",Yes,,,Compliance,\"Compliance, security and legal.\",Open AI API,,Fine tunning LLMS,,,,,,,,,\n",
        "Sr. Director AI/ML,\"1,000+\",Yes,\"ITSM, Sales Prospecting\",chat bot,scalabiltiy,data privacy,\"Open source model (GPT-J, etc)\",kubeflow,Fine tunning LLMS,,,,,,,,,\n",
        "BDM,\"1,000+\",Yes,\"code development, summarisation\",\"development environment, pipelining and deployment tooling\",stack is immature; benchmarks are nonexistent; risk hard/impossible to gauge,evaluating efficacy and ROI; understanding power needs/CO2 emissions,In house model,PyTorch + in-house tools,Inferenece for LLMs,user feedback,\"extremely power-hungry, hard to scale; networking bandwidth is as important as memory when parallelising\",is anyone measuring LLM inference power consumption (and CO2 emissions)? what are the observed ranges of watts/inference of the various model sizes?,,,,,,'''"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# IMPORTS"
      ],
      "metadata": {
        "id": "D_eKbppz0Jqd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install openai"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kH2ipP2L0jgb",
        "outputId": "cd16dd8c-2574-4aa1-c917-1ac93fd4e0b2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: openai in /usr/local/lib/python3.10/dist-packages (0.27.6)\n",
            "Requirement already satisfied: requests>=2.20 in /usr/local/lib/python3.10/dist-packages (from openai) (2.27.1)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from openai) (4.65.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from openai) (3.8.4)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai) (1.26.15)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai) (2022.12.7)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai) (3.4)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai) (2.0.12)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai) (1.3.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai) (6.0.4)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai) (1.3.3)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai) (1.9.2)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai) (23.1.0)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai) (4.0.2)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import openai\n",
        "import pandas as pd\n",
        "from google.colab import files\n",
        "\n",
        "import csv\n",
        "import json\n",
        "import typing as T\n",
        "from io import StringIO\n",
        "from datetime import datetime\n",
        "from functools import partial"
      ],
      "metadata": {
        "id": "jqXt40sU0KS7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# CONFIGS"
      ],
      "metadata": {
        "id": "Ha6xJz028cD8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "openai.api_key = OPENAI_API_KEY"
      ],
      "metadata": {
        "id": "XwDAsPSu8cUa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pd.set_option('display.max_rows', None)\n",
        "pd.set_option('display.max_columns', None)\n",
        "pd.set_option('display.max_colwidth', None)"
      ],
      "metadata": {
        "id": "jMWLO98uOvkY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# DATASETS"
      ],
      "metadata": {
        "id": "9E1p5LVc1oWk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "results = pd.read_csv(StringIO(RESULTS))\n",
        "print('Shape:', results.shape)\n",
        "results.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 878
        },
        "id": "B-_cBlVH0XYF",
        "outputId": "8400509a-f613-4bfc-9fe3-d2c84b6045a6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Shape: (110, 19)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  What is your position/title at your company?  \\\n",
              "0                                          NaN   \n",
              "1                               Data scientist   \n",
              "2                                  ML Engineer   \n",
              "3                                      Founder   \n",
              "4                                        Owner   \n",
              "\n",
              "  How big is your organization? (number of employees)  \\\n",
              "0                                                 NaN   \n",
              "1                                              1,000+   \n",
              "2                                              50-500   \n",
              "3                                                1-10   \n",
              "4                                                1-10   \n",
              "\n",
              "  Are you using LLM at in your organization?  \\\n",
              "0                                        NaN   \n",
              "1                                         No   \n",
              "2                                        Yes   \n",
              "3                                        Yes   \n",
              "4                                        Yes   \n",
              "\n",
              "         What is your use case/use cases?  \\\n",
              "0                                     NaN   \n",
              "1                                     NaN   \n",
              "2                      Text summarisation   \n",
              "3  Data annotation; summarization; search   \n",
              "4    Content production and brainstorming   \n",
              "\n",
              "  Have you integrated or built any internal tools to support LLMs in your org? If so what?  \\\n",
              "0                                                                                      NaN   \n",
              "1                                                                                      NaN   \n",
              "2                                                                                       No   \n",
              "3                                    https://chat.mantisnlp.com, internal annotation tools   \n",
              "4                                                                                  Not yet   \n",
              "\n",
              "                     What are some of the main challenges you have encountered thus far when building with LLMs  \\\n",
              "0                                                                                                           NaN   \n",
              "1                                                                                                           NaN   \n",
              "2  Current infrastructure not designed for such massive models, having to implement workarounds for quick fixes   \n",
              "3                         out of date info, hallucinations, cost, difficulty in deploying on own infrastructure   \n",
              "4                                                                                                       not yet   \n",
              "\n",
              "  What are your main concerns with using LLMs in production?  \\\n",
              "0                                                        NaN   \n",
              "1                                                        NaN   \n",
              "2                                                       Cost   \n",
              "3       they could serve nonsense or out of date information   \n",
              "4                                         cost of openai API   \n",
              "\n",
              "          how are you using LLMs?  \\\n",
              "0                             NaN   \n",
              "1                             NaN   \n",
              "2  Open source model (GPT-J, etc)   \n",
              "3                     Open AI API   \n",
              "4                     Open AI API   \n",
              "\n",
              "                      What tools are you using with LLMs?  \\\n",
              "0                                                     NaN   \n",
              "1                                                     NaN   \n",
              "2                                                  Seldon   \n",
              "3                                                     NaN   \n",
              "4  Looking at Pinecone, weaviate, langchain, hugging face   \n",
              "\n",
              "  What areas are you most interested in learning more about?  \\\n",
              "0                                                        NaN   \n",
              "1                                                        NaN   \n",
              "2                                        Inferenece for LLMs   \n",
              "3                                      Working with OSS LLMs   \n",
              "4                                                 Embeddings   \n",
              "\n",
              "                           How do you deal with reliability of output?  \\\n",
              "0                                                                  NaN   \n",
              "1                                                                  NaN   \n",
              "2                                                                  NaN   \n",
              "3  we provide links to the context from which answers have been drawn.   \n",
              "4                                                      temperature = 0   \n",
              "\n",
              "  Any stories you have that are worth sharing about working with LLMs?  \\\n",
              "0                                                                  NaN   \n",
              "1                                                                  NaN   \n",
              "2                                                                  NaN   \n",
              "3                                                                  NaN   \n",
              "4                                                                  NaN   \n",
              "\n",
              "  Any questions you have for the community about LLM in production?  \\\n",
              "0                                                               NaN   \n",
              "1                                                               NaN   \n",
              "2                                                               NaN   \n",
              "3                                                               NaN   \n",
              "4                                                               NaN   \n",
              "\n",
              "  What is the main reason for not using LLMs in your org?  \\\n",
              "0                                                     NaN   \n",
              "1                             Production takes investment   \n",
              "2                                                     NaN   \n",
              "3                                                     NaN   \n",
              "4                                                     NaN   \n",
              "\n",
              "  What are some key questions you face when it comes to using LLM in prod?  \\\n",
              "0                                                                      NaN   \n",
              "1                                             Cost to maintain the service   \n",
              "2                                                                      NaN   \n",
              "3                                                                      NaN   \n",
              "4                                                                      NaN   \n",
              "\n",
              "  have you tried LLMs for different use cases in your org?  \\\n",
              "0                                                      NaN   \n",
              "1                                                      Yes   \n",
              "2                                                      NaN   \n",
              "3                                                      NaN   \n",
              "4                                                      NaN   \n",
              "\n",
              "  If yes, why did it not work out?  \\\n",
              "0                              NaN   \n",
              "1                     Date quality   \n",
              "2                              NaN   \n",
              "3                              NaN   \n",
              "4                              NaN   \n",
              "\n",
              "  Any stories you have that are worth sharing about working with LLMs?.1  \\\n",
              "0                                                                    NaN   \n",
              "1                                                                    N/a   \n",
              "2                                                                    NaN   \n",
              "3                                                                    NaN   \n",
              "4                                                                    NaN   \n",
              "\n",
              "  Any questions you have for the community about LLM in production?.1  \n",
              "0                                                                 NaN  \n",
              "1                                                                 NaN  \n",
              "2                                                                 NaN  \n",
              "3                                                                 NaN  \n",
              "4                                                                 NaN  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a7869c97-ab3a-4cee-b519-54679038fd15\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>What is your position/title at your company?</th>\n",
              "      <th>How big is your organization? (number of employees)</th>\n",
              "      <th>Are you using LLM at in your organization?</th>\n",
              "      <th>What is your use case/use cases?</th>\n",
              "      <th>Have you integrated or built any internal tools to support LLMs in your org? If so what?</th>\n",
              "      <th>What are some of the main challenges you have encountered thus far when building with LLMs</th>\n",
              "      <th>What are your main concerns with using LLMs in production?</th>\n",
              "      <th>how are you using LLMs?</th>\n",
              "      <th>What tools are you using with LLMs?</th>\n",
              "      <th>What areas are you most interested in learning more about?</th>\n",
              "      <th>How do you deal with reliability of output?</th>\n",
              "      <th>Any stories you have that are worth sharing about working with LLMs?</th>\n",
              "      <th>Any questions you have for the community about LLM in production?</th>\n",
              "      <th>What is the main reason for not using LLMs in your org?</th>\n",
              "      <th>What are some key questions you face when it comes to using LLM in prod?</th>\n",
              "      <th>have you tried LLMs for different use cases in your org?</th>\n",
              "      <th>If yes, why did it not work out?</th>\n",
              "      <th>Any stories you have that are worth sharing about working with LLMs?.1</th>\n",
              "      <th>Any questions you have for the community about LLM in production?.1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Data scientist</td>\n",
              "      <td>1,000+</td>\n",
              "      <td>No</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Production takes investment</td>\n",
              "      <td>Cost to maintain the service</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Date quality</td>\n",
              "      <td>N/a</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ML Engineer</td>\n",
              "      <td>50-500</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Text summarisation</td>\n",
              "      <td>No</td>\n",
              "      <td>Current infrastructure not designed for such massive models, having to implement workarounds for quick fixes</td>\n",
              "      <td>Cost</td>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "      <td>Seldon</td>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Founder</td>\n",
              "      <td>1-10</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Data annotation; summarization; search</td>\n",
              "      <td>https://chat.mantisnlp.com, internal annotation tools</td>\n",
              "      <td>out of date info, hallucinations, cost, difficulty in deploying on own infrastructure</td>\n",
              "      <td>they could serve nonsense or out of date information</td>\n",
              "      <td>Open AI API</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>we provide links to the context from which answers have been drawn.</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Owner</td>\n",
              "      <td>1-10</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Content production and brainstorming</td>\n",
              "      <td>Not yet</td>\n",
              "      <td>not yet</td>\n",
              "      <td>cost of openai API</td>\n",
              "      <td>Open AI API</td>\n",
              "      <td>Looking at Pinecone, weaviate, langchain, hugging face</td>\n",
              "      <td>Embeddings</td>\n",
              "      <td>temperature = 0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a7869c97-ab3a-4cee-b519-54679038fd15')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-a7869c97-ab3a-4cee-b519-54679038fd15 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-a7869c97-ab3a-4cee-b519-54679038fd15');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print('Survey Questions:')\n",
        "questions = list(results.columns)\n",
        "for i, question in enumerate(questions):\n",
        "    print(f'{i}. {question}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NH0oRPCe1-mp",
        "outputId": "fe6cdcfd-eb59-4e00-b5d1-a485cb34c04a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Survey Questions:\n",
            "0. What is your position/title at your company?\n",
            "1. How big is your organization? (number of employees)\n",
            "2. Are you using LLM at in your organization?\n",
            "3. What is your use case/use cases?\n",
            "4. Have you integrated or built any internal tools to support LLMs in your org? If so what?\n",
            "5. What are some of the main challenges you have encountered thus far when building with LLMs\n",
            "6. What are your main concerns with using LLMs in production?\n",
            "7. how are you using LLMs?\n",
            "8. What tools are you using with LLMs?\n",
            "9. What areas are you most interested in learning more about?\n",
            "10. How do you deal with reliability of output?\n",
            "11. Any stories you have that are worth sharing about working with LLMs?\n",
            "12. Any questions you have for the community about LLM in production?\n",
            "13. What is the main reason for not using LLMs in your org?\n",
            "14. What are some key questions you face when it comes to using LLM in prod?\n",
            "15. have you tried LLMs for different use cases in your org?\n",
            "16. If yes, why did it not work out?\n",
            "17. Any stories you have that are worth sharing about working with LLMs?.1\n",
            "18. Any questions you have for the community about LLM in production?.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "duplicated = {col: col.replace('.1', '') for col in results if col.endswith('.1')}\n",
        "print('\\n'.join(f'- {dup} -> {col}' for dup, col in duplicated.items()))\n",
        "results[[col for sub in duplicated.items() for col in sub]].head(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 688
        },
        "id": "uEKftJPu5hQa",
        "outputId": "623916d9-3816-48dd-ed2d-c4fa28011e53"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "- Any stories you have that are worth sharing about working with LLMs?.1 -> Any stories you have that are worth sharing about working with LLMs?\n",
            "- Any questions you have for the community about LLM in production?.1 -> Any questions you have for the community about LLM in production?\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  Any stories you have that are worth sharing about working with LLMs?.1  \\\n",
              "0                                                                    NaN   \n",
              "1                                                                    N/a   \n",
              "2                                                                    NaN   \n",
              "3                                                                    NaN   \n",
              "4                                                                    NaN   \n",
              "5                                                                    NaN   \n",
              "6                                                                    NaN   \n",
              "7                                                                    NaN   \n",
              "8                                                                    NaN   \n",
              "9                                                                    NaN   \n",
              "\n",
              "                                                                                                                                                                                                                     Any stories you have that are worth sharing about working with LLMs?  \\\n",
              "0                                                                                                                                                                                                                                                                                     NaN   \n",
              "1                                                                                                                                                                                                                                                                                     NaN   \n",
              "2                                                                                                                                                                                                                                                                                     NaN   \n",
              "3                                                                                                                                                                                                                                                                                     NaN   \n",
              "4                                                                                                                                                                                                                                                                                     NaN   \n",
              "5                                                                                                                                                                                                                                                                                     NaN   \n",
              "6                                                                                                                                                                                                                                                                                     NaN   \n",
              "7  I am not sure what ranks as a LLM. Anything bert and beyond? Anyway, we once used setfit for a production task but found that performance was really tricky to debug. The proof of concept was fine. Production was a complete mess. To this day I dont know why- underlying hardware?   \n",
              "8                                                                                                                                                                                                                                                                                     NaN   \n",
              "9                                                                                                                                                                                                                                                                                      lo   \n",
              "\n",
              "                                                                                                                                                                                             Any questions you have for the community about LLM in production?.1  \\\n",
              "0                                                                                                                                                                                                                                                            NaN   \n",
              "1                                                                                                                                                                                                                                                            NaN   \n",
              "2                                                                                                                                                                                                                                                            NaN   \n",
              "3                                                                                                                                                                                                                                                            NaN   \n",
              "4                                                                                                                                                                                                                                                            NaN   \n",
              "5                                                                                                                                                                                                                                                            NaN   \n",
              "6  Anyone have some experience with getting better reliability out of ChatGPT? Anyone succeeded in fine-tuning and getting better quality than ChatGPT? Thinking of llama for example.\\nHow can you limit ChatGPT so users will not ask it totally random stuff?   \n",
              "7                                                                                                                                                                                                                                                            NaN   \n",
              "8                                                                                                                                                                                                                                Interested in results of survey   \n",
              "9                                                                                                                                                                                                                                                            NaN   \n",
              "\n",
              "                                                                                                                                                                      Any questions you have for the community about LLM in production?  \n",
              "0                                                                                                                                                                                                                                   NaN  \n",
              "1                                                                                                                                                                                                                                   NaN  \n",
              "2                                                                                                                                                                                                                                   NaN  \n",
              "3                                                                                                                                                                                                                                   NaN  \n",
              "4                                                                                                                                                                                                                                   NaN  \n",
              "5                                                                                                                                                                                                                                   NaN  \n",
              "6                                                                                                                                                                                                                                   NaN  \n",
              "7  I mainly wonder how everyone is monitoring these. ALSO. For my company GDPR is a big deal. My DPO shoots down our ideas more often than not or limits it to using the last three months of data. How do other people deal with this?  \n",
              "8                                                                                                                                                                                                                                   NaN  \n",
              "9                                                                                                                                                                                                                                    lo  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-012e3614-1140-407c-9c35-1ae75e45747f\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Any stories you have that are worth sharing about working with LLMs?.1</th>\n",
              "      <th>Any stories you have that are worth sharing about working with LLMs?</th>\n",
              "      <th>Any questions you have for the community about LLM in production?.1</th>\n",
              "      <th>Any questions you have for the community about LLM in production?</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>N/a</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Anyone have some experience with getting better reliability out of ChatGPT? Anyone succeeded in fine-tuning and getting better quality than ChatGPT? Thinking of llama for example.\\nHow can you limit ChatGPT so users will not ask it totally random stuff?</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>NaN</td>\n",
              "      <td>I am not sure what ranks as a LLM. Anything bert and beyond? Anyway, we once used setfit for a production task but found that performance was really tricky to debug. The proof of concept was fine. Production was a complete mess. To this day I dont know why- underlying hardware?</td>\n",
              "      <td>NaN</td>\n",
              "      <td>I mainly wonder how everyone is monitoring these. ALSO. For my company GDPR is a big deal. My DPO shoots down our ideas more often than not or limits it to using the last three months of data. How do other people deal with this?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Interested in results of survey</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>NaN</td>\n",
              "      <td>lo</td>\n",
              "      <td>NaN</td>\n",
              "      <td>lo</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-012e3614-1140-407c-9c35-1ae75e45747f')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-012e3614-1140-407c-9c35-1ae75e45747f button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-012e3614-1140-407c-9c35-1ae75e45747f');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "results = results.drop(columns=duplicated)\n",
        "print('Shape:', results.shape)\n",
        "results.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 878
        },
        "id": "9muUV7ob2X7F",
        "outputId": "18ca78d1-1ba2-43db-da29-09f9c090e641"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Shape: (110, 17)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  What is your position/title at your company?  \\\n",
              "0                                          NaN   \n",
              "1                               Data scientist   \n",
              "2                                  ML Engineer   \n",
              "3                                      Founder   \n",
              "4                                        Owner   \n",
              "\n",
              "  How big is your organization? (number of employees)  \\\n",
              "0                                                 NaN   \n",
              "1                                              1,000+   \n",
              "2                                              50-500   \n",
              "3                                                1-10   \n",
              "4                                                1-10   \n",
              "\n",
              "  Are you using LLM at in your organization?  \\\n",
              "0                                        NaN   \n",
              "1                                         No   \n",
              "2                                        Yes   \n",
              "3                                        Yes   \n",
              "4                                        Yes   \n",
              "\n",
              "         What is your use case/use cases?  \\\n",
              "0                                     NaN   \n",
              "1                                     NaN   \n",
              "2                      Text summarisation   \n",
              "3  Data annotation; summarization; search   \n",
              "4    Content production and brainstorming   \n",
              "\n",
              "  Have you integrated or built any internal tools to support LLMs in your org? If so what?  \\\n",
              "0                                                                                      NaN   \n",
              "1                                                                                      NaN   \n",
              "2                                                                                       No   \n",
              "3                                    https://chat.mantisnlp.com, internal annotation tools   \n",
              "4                                                                                  Not yet   \n",
              "\n",
              "                     What are some of the main challenges you have encountered thus far when building with LLMs  \\\n",
              "0                                                                                                           NaN   \n",
              "1                                                                                                           NaN   \n",
              "2  Current infrastructure not designed for such massive models, having to implement workarounds for quick fixes   \n",
              "3                         out of date info, hallucinations, cost, difficulty in deploying on own infrastructure   \n",
              "4                                                                                                       not yet   \n",
              "\n",
              "  What are your main concerns with using LLMs in production?  \\\n",
              "0                                                        NaN   \n",
              "1                                                        NaN   \n",
              "2                                                       Cost   \n",
              "3       they could serve nonsense or out of date information   \n",
              "4                                         cost of openai API   \n",
              "\n",
              "          how are you using LLMs?  \\\n",
              "0                             NaN   \n",
              "1                             NaN   \n",
              "2  Open source model (GPT-J, etc)   \n",
              "3                     Open AI API   \n",
              "4                     Open AI API   \n",
              "\n",
              "                      What tools are you using with LLMs?  \\\n",
              "0                                                     NaN   \n",
              "1                                                     NaN   \n",
              "2                                                  Seldon   \n",
              "3                                                     NaN   \n",
              "4  Looking at Pinecone, weaviate, langchain, hugging face   \n",
              "\n",
              "  What areas are you most interested in learning more about?  \\\n",
              "0                                                        NaN   \n",
              "1                                                        NaN   \n",
              "2                                        Inferenece for LLMs   \n",
              "3                                      Working with OSS LLMs   \n",
              "4                                                 Embeddings   \n",
              "\n",
              "                           How do you deal with reliability of output?  \\\n",
              "0                                                                  NaN   \n",
              "1                                                                  NaN   \n",
              "2                                                                  NaN   \n",
              "3  we provide links to the context from which answers have been drawn.   \n",
              "4                                                      temperature = 0   \n",
              "\n",
              "  Any stories you have that are worth sharing about working with LLMs?  \\\n",
              "0                                                                  NaN   \n",
              "1                                                                  NaN   \n",
              "2                                                                  NaN   \n",
              "3                                                                  NaN   \n",
              "4                                                                  NaN   \n",
              "\n",
              "  Any questions you have for the community about LLM in production?  \\\n",
              "0                                                               NaN   \n",
              "1                                                               NaN   \n",
              "2                                                               NaN   \n",
              "3                                                               NaN   \n",
              "4                                                               NaN   \n",
              "\n",
              "  What is the main reason for not using LLMs in your org?  \\\n",
              "0                                                     NaN   \n",
              "1                             Production takes investment   \n",
              "2                                                     NaN   \n",
              "3                                                     NaN   \n",
              "4                                                     NaN   \n",
              "\n",
              "  What are some key questions you face when it comes to using LLM in prod?  \\\n",
              "0                                                                      NaN   \n",
              "1                                             Cost to maintain the service   \n",
              "2                                                                      NaN   \n",
              "3                                                                      NaN   \n",
              "4                                                                      NaN   \n",
              "\n",
              "  have you tried LLMs for different use cases in your org?  \\\n",
              "0                                                      NaN   \n",
              "1                                                      Yes   \n",
              "2                                                      NaN   \n",
              "3                                                      NaN   \n",
              "4                                                      NaN   \n",
              "\n",
              "  If yes, why did it not work out?  \n",
              "0                              NaN  \n",
              "1                     Date quality  \n",
              "2                              NaN  \n",
              "3                              NaN  \n",
              "4                              NaN  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-cd0ae62c-e813-4f4e-a545-9ed98ad28b02\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>What is your position/title at your company?</th>\n",
              "      <th>How big is your organization? (number of employees)</th>\n",
              "      <th>Are you using LLM at in your organization?</th>\n",
              "      <th>What is your use case/use cases?</th>\n",
              "      <th>Have you integrated or built any internal tools to support LLMs in your org? If so what?</th>\n",
              "      <th>What are some of the main challenges you have encountered thus far when building with LLMs</th>\n",
              "      <th>What are your main concerns with using LLMs in production?</th>\n",
              "      <th>how are you using LLMs?</th>\n",
              "      <th>What tools are you using with LLMs?</th>\n",
              "      <th>What areas are you most interested in learning more about?</th>\n",
              "      <th>How do you deal with reliability of output?</th>\n",
              "      <th>Any stories you have that are worth sharing about working with LLMs?</th>\n",
              "      <th>Any questions you have for the community about LLM in production?</th>\n",
              "      <th>What is the main reason for not using LLMs in your org?</th>\n",
              "      <th>What are some key questions you face when it comes to using LLM in prod?</th>\n",
              "      <th>have you tried LLMs for different use cases in your org?</th>\n",
              "      <th>If yes, why did it not work out?</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Data scientist</td>\n",
              "      <td>1,000+</td>\n",
              "      <td>No</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Production takes investment</td>\n",
              "      <td>Cost to maintain the service</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Date quality</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ML Engineer</td>\n",
              "      <td>50-500</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Text summarisation</td>\n",
              "      <td>No</td>\n",
              "      <td>Current infrastructure not designed for such massive models, having to implement workarounds for quick fixes</td>\n",
              "      <td>Cost</td>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "      <td>Seldon</td>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Founder</td>\n",
              "      <td>1-10</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Data annotation; summarization; search</td>\n",
              "      <td>https://chat.mantisnlp.com, internal annotation tools</td>\n",
              "      <td>out of date info, hallucinations, cost, difficulty in deploying on own infrastructure</td>\n",
              "      <td>they could serve nonsense or out of date information</td>\n",
              "      <td>Open AI API</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>we provide links to the context from which answers have been drawn.</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Owner</td>\n",
              "      <td>1-10</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Content production and brainstorming</td>\n",
              "      <td>Not yet</td>\n",
              "      <td>not yet</td>\n",
              "      <td>cost of openai API</td>\n",
              "      <td>Open AI API</td>\n",
              "      <td>Looking at Pinecone, weaviate, langchain, hugging face</td>\n",
              "      <td>Embeddings</td>\n",
              "      <td>temperature = 0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cd0ae62c-e813-4f4e-a545-9ed98ad28b02')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-cd0ae62c-e813-4f4e-a545-9ed98ad28b02 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-cd0ae62c-e813-4f4e-a545-9ed98ad28b02');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# AI MODELS"
      ],
      "metadata": {
        "id": "okhTIwGD9Bbs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "openai.Model.retrieve(OPENAI_MODEL)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cbI7Qshr9Bqp",
        "outputId": "9ef61130-80bc-46b5-cc32-ad9132ed6933"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<Model model id=gpt-3.5-turbo at 0x7ff655339030> JSON: {\n",
              "  \"created\": 1677610602,\n",
              "  \"id\": \"gpt-3.5-turbo\",\n",
              "  \"object\": \"model\",\n",
              "  \"owned_by\": \"openai\",\n",
              "  \"parent\": null,\n",
              "  \"permission\": [\n",
              "    {\n",
              "      \"allow_create_engine\": false,\n",
              "      \"allow_fine_tuning\": false,\n",
              "      \"allow_logprobs\": true,\n",
              "      \"allow_sampling\": true,\n",
              "      \"allow_search_indices\": false,\n",
              "      \"allow_view\": true,\n",
              "      \"created\": 1683391732,\n",
              "      \"group\": null,\n",
              "      \"id\": \"modelperm-Kch774kyIWxK1SMaTV7JKoho\",\n",
              "      \"is_blocking\": false,\n",
              "      \"object\": \"model_permission\",\n",
              "      \"organization\": \"*\"\n",
              "    }\n",
              "  ],\n",
              "  \"root\": \"gpt-3.5-turbo\"\n",
              "}"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model = partial(\n",
        "    openai.ChatCompletion.create,\n",
        "    temperature=TEMPERATURE,\n",
        "    model=OPENAI_MODEL,\n",
        ")"
      ],
      "metadata": {
        "id": "tGz_LQfa9b3q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# FUNCTIONS"
      ],
      "metadata": {
        "id": "yT8DMsZIBenF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def itemize(strings: list[str]) -> str:\n",
        "    \"\"\"Convert a list of strings to items.\"\"\"\n",
        "    return '\\n'.join([f'- {s}' for s in strings])\n",
        "\n",
        "print(itemize(['a', 'b', 'c']))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BlX24muMUYUF",
        "outputId": "f2c49e2d-ddd7-448e-dfac-6e8fa097c226"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "- a\n",
            "- b\n",
            "- c\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def message(content: str, role: str = 'user', name: str | None = None) -> dict[str, str]:\n",
        "    \"\"\"Create a message with the given role, content, and name.\"\"\"\n",
        "    msg = {'role': role, 'content': content}\n",
        "    if name is not None:\n",
        "        msg['name'] = name\n",
        "    return msg\n",
        "\n",
        "user = partial(message, role='user')\n",
        "system = partial(message, role='system')\n",
        "print(user('hello user'), system('hi system'))"
      ],
      "metadata": {
        "id": "ipfaClz-Be7T",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ec180ffb-0f52-4310-c2dc-ae20d266ba61"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'role': 'user', 'content': 'hello user'} {'role': 'system', 'content': 'hi system'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def chat(instructions: str, inputs: list[str]) -> list[dict[str, str]]:\n",
        "    \"\"\"Convert a set of instructions and inputs to a chat.\"\"\"\n",
        "    items = itemize(inputs) # convert to bullet points\n",
        "    messages = [user(instructions), user(items)]\n",
        "    return messages\n",
        "\n",
        "print(chat('Count:', ['1', '2', '3']))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "znoducZepA2_",
        "outputId": "24792e2c-de73-4042-fbd8-2b5a99e2304d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[{'role': 'user', 'content': 'Count:'}, {'role': 'user', 'content': '- 1\\n- 2\\n- 3'}]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def from_jsonlines(jsonlines: str) -> list[dict[str, object]]:\n",
        "    \"\"\"Convert a json line string to python data structures.\"\"\"\n",
        "    return [json.loads(line) for line in jsonlines.splitlines()]\n",
        "\n",
        "print(from_jsonlines('{\"a\": 1, \"b\": 2}\\n{\"c\": 3}'))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oO9zz7PRpwsp",
        "outputId": "6e5c1931-550f-4365-a213-5d5903a77cbe"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[{'a': 1, 'b': 2}, {'c': 3}]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def batch(seq, size: int = 10) -> T.Iterator[tuple[int, T.Any]]:\n",
        "    \"\"\"Generate batch of the given size from a seq.\"\"\"\n",
        "    lenght = len(seq)\n",
        "    number = lenght // size\n",
        "    number += 1 if lenght % size != 0 else 0\n",
        "    for i in range(number):\n",
        "        start = i * size\n",
        "        stop = (i + 1) * size\n",
        "        subset = seq[start:stop]\n",
        "        yield i, subset\n",
        "\n",
        "print(list(batch([1, 2, 3, 4, 5], 2)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BI-tB2diqlc0",
        "outputId": "3e55e620-8a9d-4f00-c463-282a64015b89"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[(0, [1, 2]), (1, [3, 4]), (2, [5])]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def query(model, messages: list[dict[str, str]], **kwargs) -> str:\n",
        "    \"\"\"Query the model with the given messages.\"\"\"\n",
        "    completion = model(messages=messages, **kwargs)\n",
        "    print('- tokens:', completion['usage']['total_tokens'])\n",
        "    content = completion['choices'][0]['message']['content']\n",
        "    return content\n",
        "\n",
        "# query(model=model, messages=[user('Say Hello World!')])"
      ],
      "metadata": {
        "id": "fEObzEdMQOMD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def associate(model, instructions: str, inputs: pd.Series, batch_size: int = 20, limit: int | None = None, **kwargs) -> pd.DataFrame:\n",
        "    \"\"\"Associate the user inputs to the model outputs with the given instructions.\"\"\"\n",
        "    # avoid wasting requests\n",
        "    requests = inputs.dropna()\n",
        "    # apply the limit (optional)\n",
        "    if limit is not None:\n",
        "        requests = requests[:limit]\n",
        "    print('- Inputs:', len(inputs))\n",
        "    # iterate on the requests by batch\n",
        "    dataframes = [] # store partial results\n",
        "    for i, subset in batch(requests, batch_size):\n",
        "        print('- batch:', i, '->', len(subset))\n",
        "        # query the model and extract the records\n",
        "        messages = chat(instructions, subset.tolist())\n",
        "        content = query(model, messages=messages)\n",
        "        records = from_jsonlines(content)\n",
        "        # handle the case where lenght are different\n",
        "        if len(records) != len(subset):\n",
        "            print(f'Warning! Got: {len(records)}, expected: {len(subset)}')\n",
        "            records = records + [{} for i in range(len(subset) - len(records))] \n",
        "        # convert the records to an indexed dataframe\n",
        "        df = pd.DataFrame(records, index=subset.index)\n",
        "        dataframes.append(df)\n",
        "    # combine the results with the inputs\n",
        "    outputs = pd.concat(dataframes, axis='index')\n",
        "    reviews = pd.concat([inputs, outputs], axis='columns')\n",
        "    return reviews"
      ],
      "metadata": {
        "id": "Qd_PTlIhOwM_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# PROMPTS"
      ],
      "metadata": {
        "id": "iqNaWJyu8k6L"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "q0_question = questions[0]\n",
        "q0_instructions = \"\"\"I received answers from an MLOps survey about Large Language Models (LLMs).\n",
        "Your task is to extract title and position from each answer.\n",
        "You should use common title and position names whenever possible to avoid synonyms and acronyms.\n",
        "If no title or position are mentioned, you should use an empty string as a placeholder.\n",
        "You should output these information in the JSON lines format.\n",
        "You should generate one JSON line per answer.\n",
        "You can put other information in a field called \"other\".\n",
        "Here's an example of an answer:\n",
        "- senior MLOps\n",
        "- Head Of Product\n",
        "- ML Engineer Lead\n",
        "- Director of AI\n",
        "- CEO\n",
        "Here's an example of the expected output:\n",
        "{\"title\": \"MLOps Engineer\", \"position\": \"Senior\", \"other\": \"\"}\n",
        "{\"title\": \"\", \"position\": \"Head\", \"other\": \"Product\"}\n",
        "{\"title\": \"Machine Learning Engineer\", \"position\": \"Lead\", \"other\": \"\"}\n",
        "{\"title\": \"\", \"position\": \"Director\", \"other\": \"AI\"}\n",
        "{\"title\": \"CEO\", \"position\": \"\", \"other\": \"\"}\n",
        "Answers:\"\"\"\n",
        "q0_inputs = results.iloc[:, 0]\n",
        "print(f'Question 0:', q0_question)\n",
        "q0_reviews = associate(model, q0_instructions, q0_inputs)\n",
        "q0_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "EErppNkykqsf",
        "outputId": "a6f499ac-2f3f-4f3f-a53f-6ea5bcb948b1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 0: What is your position/title at your company?\n",
            "- Inputs: 110\n",
            "- batch: 0 -> 20\n",
            "- tokens: 656\n",
            "- batch: 1 -> 20\n",
            "- tokens: 668\n",
            "- batch: 2 -> 20\n",
            "- tokens: 666\n",
            "- batch: 3 -> 20\n",
            "- tokens: 642\n",
            "- batch: 4 -> 20\n",
            "- tokens: 671\n",
            "- batch: 5 -> 4\n",
            "- tokens: 320\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                           What is your position/title at your company?  \\\n",
              "1                                                        Data scientist   \n",
              "2                                                           ML Engineer   \n",
              "3                                                               Founder   \n",
              "4                                                                 Owner   \n",
              "5                                            Senior Principal Scientist   \n",
              "6                                              Head of Machine Learning   \n",
              "7                                      Senior Machine Learning Engineer   \n",
              "8                                                       Head of Product   \n",
              "10                                                 Senior Data Engineer   \n",
              "11                                            Machine learning engineer   \n",
              "12                                                Senior Data Scientist   \n",
              "13                                                          ML Engineer   \n",
              "14                                                   Snr MLOps Engineer   \n",
              "16                                                   Senior ML engineer   \n",
              "17                                            Machine Learning Engineer   \n",
              "19                                              Founding Data Scientist   \n",
              "20                                                                  CTO   \n",
              "21                                            Machine Learning Engineer   \n",
              "22                                                           Head of AI   \n",
              "23                                                              Product   \n",
              "24                                                  Lead Data Scientist   \n",
              "25                                                           Co-founder   \n",
              "26                                                                  MLE   \n",
              "27                                                              Snr MLE   \n",
              "28                                  Director of Artificial Intelligence   \n",
              "29                                                       Data Scientist   \n",
              "30                                                   Co-founder and CTO   \n",
              "32                                             Senior Software Enginner   \n",
              "33                                                           Co-founder   \n",
              "34                                                         ML Developer   \n",
              "36                                                       data scientist   \n",
              "37                                     Senior Machine Learning Engineer   \n",
              "38                                                              Founder   \n",
              "39                                                                  SRE   \n",
              "40   Cofounder (normally product manager, but also wearer of many hats)   \n",
              "41                                                       Data Architect   \n",
              "42                                             Senior software engineer   \n",
              "43                                                           consultant   \n",
              "44                                                                  CTO   \n",
              "45                                          Data scientist / co-founder   \n",
              "46                                                   Analytics engineer   \n",
              "47                                                 Director, Innovation   \n",
              "48                                             ML & MLOps practice lead   \n",
              "49                                                              Founder   \n",
              "50                                                                  CTO   \n",
              "51                                                       MLOps Engineer   \n",
              "52                                                     Junior developer   \n",
              "53                                               deep learning engineer   \n",
              "54                                                           Co-Founder   \n",
              "55                                                               Intern   \n",
              "56                                                                  CTO   \n",
              "57                                        Software Development Engineer   \n",
              "58                                               Data Science Team Lead   \n",
              "59                                                           Sr. ML Eng   \n",
              "60                                                     Senior AI Expert   \n",
              "61                                                Data Science Engineer   \n",
              "62                                                  Head of Data and ML   \n",
              "63                                                    ML/MLOps Engineer   \n",
              "64                                           Data Scientist, HappyFresh   \n",
              "65                                                       Data Scientist   \n",
              "66                                                       Data Scientist   \n",
              "67                                                Senior Data Scientist   \n",
              "68                                                       Data Scientist   \n",
              "69                                                 Head of Data Science   \n",
              "70                                                              Founder   \n",
              "71                                                                  CEO   \n",
              "72                                                              Founder   \n",
              "73                                                Senior Dara scientist   \n",
              "74                                                          ML Engineer   \n",
              "75                                                             Director   \n",
              "76                                          Technical Marketing Manager   \n",
              "77                                                        Head of MLOps   \n",
              "78                                                                  CDO   \n",
              "79                                                                Owner   \n",
              "80                                                Agricultue consultant   \n",
              "81                                                       Data scientist   \n",
              "82                                           Product Owner Data Science   \n",
              "83                                                   Team Lead DS & MLE   \n",
              "84                                                           Co-founder   \n",
              "85                                            Machine Learning Engineer   \n",
              "86                                                                  CEO   \n",
              "87                                            Principal Cloud Developer   \n",
              "88                                                                  CPO   \n",
              "89                                                 Cloud MLOps Engineer   \n",
              "90                                                     CEO / Co-Founder   \n",
              "91                                                Senior MLOps engineer   \n",
              "92                                            machine learning engineer   \n",
              "93                                                                  CEO   \n",
              "94                                                          ML engineer   \n",
              "95                            Domain Chapter Lead - Data Science and AI   \n",
              "96                                     Senior Machine Learning Engineer   \n",
              "97                                                             VP of ML   \n",
              "98                                                       Technical Lead   \n",
              "99                                                   Solutions engineer   \n",
              "100                                                   Platform Engineer   \n",
              "101                                                        Senior MLOps   \n",
              "102                                                Data science manager   \n",
              "103                                                      Data Scientist   \n",
              "104                                            Principal Data Scientist   \n",
              "105                                            Manager Machine learning   \n",
              "106                                                      Data scientist   \n",
              "107                                                           Tech Lead   \n",
              "108                                                  Sr. Director AI/ML   \n",
              "109                                                                 BDM   \n",
              "\n",
              "                             title                position  \\\n",
              "1                   Data Scientist                           \n",
              "2        Machine Learning Engineer                           \n",
              "3                          Founder                           \n",
              "4                            Owner                           \n",
              "5              Principal Scientist                  Senior   \n",
              "6         Head of Machine Learning                           \n",
              "7        Machine Learning Engineer                  Senior   \n",
              "8                                                     Head   \n",
              "10                   Data Engineer                  Senior   \n",
              "11       Machine Learning Engineer                           \n",
              "12                  Data Scientist                  Senior   \n",
              "13       Machine Learning Engineer                           \n",
              "14                  MLOps Engineer                  Senior   \n",
              "16       Machine Learning Engineer                  Senior   \n",
              "17       Machine Learning Engineer                           \n",
              "19                  Data Scientist                Founding   \n",
              "20                             CTO                           \n",
              "21       Machine Learning Engineer                           \n",
              "22                                                    Head   \n",
              "23                                                           \n",
              "24                  Data Scientist                    Lead   \n",
              "25                                              Co-founder   \n",
              "26       Machine Learning Engineer                           \n",
              "27       Machine Learning Engineer                  Senior   \n",
              "28                                                Director   \n",
              "29                  Data Scientist                           \n",
              "30                                      Co-founder and CTO   \n",
              "32               Software Engineer                  Senior   \n",
              "33                                              Co-founder   \n",
              "34      Machine Learning Developer                           \n",
              "36                  Data Scientist                           \n",
              "37       Machine Learning Engineer                  Senior   \n",
              "38                                                 Founder   \n",
              "39                                                     SRE   \n",
              "40                                              Co-founder   \n",
              "41                  Data Architect                           \n",
              "42               Software Engineer                  Senior   \n",
              "43                                              Consultant   \n",
              "44                                                     CTO   \n",
              "45                  Data Scientist                           \n",
              "46              Analytics Engineer                           \n",
              "47                                                Director   \n",
              "48        ML & MLOps Practice Lead                           \n",
              "49                         Founder                           \n",
              "50                             CTO                           \n",
              "51                  MLOps Engineer                           \n",
              "52                                                  Junior   \n",
              "53          Deep Learning Engineer                           \n",
              "54                      Co-Founder                           \n",
              "55                                                  Intern   \n",
              "56                             CTO                           \n",
              "57   Software Development Engineer                           \n",
              "58          Data Science Team Lead                           \n",
              "59                      Sr. ML Eng                           \n",
              "60                Senior AI Expert                           \n",
              "61           Data Science Engineer                           \n",
              "62                                                    Head   \n",
              "63               ML/MLOps Engineer                           \n",
              "64                  Data Scientist                           \n",
              "65                  Data Scientist                           \n",
              "66                  Data Scientist                           \n",
              "67                  Data Scientist                  Senior   \n",
              "68                  Data Scientist                           \n",
              "69                                                    Head   \n",
              "70                                                 Founder   \n",
              "71                             CEO                           \n",
              "72                                                 Founder   \n",
              "73           Senior Data Scientist                           \n",
              "74                     ML Engineer                           \n",
              "75                                                Director   \n",
              "76     Technical Marketing Manager                           \n",
              "77                                                    Head   \n",
              "78                                                     CDO   \n",
              "79                                                   Owner   \n",
              "80                                  Agriculture Consultant   \n",
              "81                  Data Scientist                           \n",
              "82      Product Owner Data Science                           \n",
              "83                                               Team Lead   \n",
              "84                                              Co-founder   \n",
              "85       Machine Learning Engineer                           \n",
              "86                             CEO                           \n",
              "87                 Cloud Developer               Principal   \n",
              "88                             CPO                           \n",
              "89                  MLOps Engineer                           \n",
              "90                CEO / Co-Founder                           \n",
              "91                  MLOps Engineer                  Senior   \n",
              "92       Machine Learning Engineer                           \n",
              "93                             CEO                           \n",
              "94       Machine Learning Engineer                      ML   \n",
              "95             Domain Chapter Lead                           \n",
              "96       Machine Learning Engineer                  Senior   \n",
              "97                        VP of ML                           \n",
              "98                  Technical Lead                           \n",
              "99              Solutions Engineer                           \n",
              "100              Platform Engineer                           \n",
              "101                          MLOps                  Senior   \n",
              "102           Data Science Manager                           \n",
              "103                 Data Scientist                           \n",
              "104       Principal Data Scientist                           \n",
              "105       Machine Learning Manager                           \n",
              "106                 Data Scientist                           \n",
              "107                      Tech Lead                           \n",
              "108                                        Senior Director   \n",
              "109                                                    BDM   \n",
              "\n",
              "                       other  \n",
              "1                             \n",
              "2                             \n",
              "3                             \n",
              "4                             \n",
              "5                             \n",
              "6                             \n",
              "7                             \n",
              "8                    Product  \n",
              "10                            \n",
              "11                            \n",
              "12                            \n",
              "13                            \n",
              "14                            \n",
              "16                            \n",
              "17                            \n",
              "19                            \n",
              "20                            \n",
              "21                            \n",
              "22                        AI  \n",
              "23                   Product  \n",
              "24                            \n",
              "25                            \n",
              "26                       MLE  \n",
              "27                            \n",
              "28   Artificial Intelligence  \n",
              "29                            \n",
              "30                            \n",
              "32                            \n",
              "33                            \n",
              "34                            \n",
              "36                            \n",
              "37                            \n",
              "38                            \n",
              "39                            \n",
              "40           Product Manager  \n",
              "41                            \n",
              "42                            \n",
              "43                            \n",
              "44                            \n",
              "45                Co-founder  \n",
              "46                            \n",
              "47                Innovation  \n",
              "48                            \n",
              "49                            \n",
              "50                            \n",
              "51                            \n",
              "52                 Developer  \n",
              "53                            \n",
              "54                            \n",
              "55                            \n",
              "56                            \n",
              "57                            \n",
              "58                            \n",
              "59                            \n",
              "60                            \n",
              "61                            \n",
              "62               Data and ML  \n",
              "63                            \n",
              "64                HappyFresh  \n",
              "65                            \n",
              "66                            \n",
              "67                            \n",
              "68                            \n",
              "69              Data Science  \n",
              "70                            \n",
              "71                            \n",
              "72                            \n",
              "73                            \n",
              "74                            \n",
              "75                            \n",
              "76                            \n",
              "77                     MLOps  \n",
              "78                            \n",
              "79                            \n",
              "80                            \n",
              "81                            \n",
              "82                            \n",
              "83                  DS & MLE  \n",
              "84                            \n",
              "85                            \n",
              "86                            \n",
              "87                            \n",
              "88                            \n",
              "89                     Cloud  \n",
              "90                            \n",
              "91                            \n",
              "92                            \n",
              "93                            \n",
              "94                            \n",
              "95       Data Science and AI  \n",
              "96                            \n",
              "97                            \n",
              "98                            \n",
              "99                            \n",
              "100                           \n",
              "101                           \n",
              "102                           \n",
              "103                           \n",
              "104                           \n",
              "105                           \n",
              "106                           \n",
              "107                           \n",
              "108                    AI/ML  \n",
              "109                           "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f79cb45a-d223-480c-a634-73f950d1466a\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>What is your position/title at your company?</th>\n",
              "      <th>title</th>\n",
              "      <th>position</th>\n",
              "      <th>other</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Data scientist</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ML Engineer</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Founder</td>\n",
              "      <td>Founder</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Owner</td>\n",
              "      <td>Owner</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>Senior Principal Scientist</td>\n",
              "      <td>Principal Scientist</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>Head of Machine Learning</td>\n",
              "      <td>Head of Machine Learning</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Senior Machine Learning Engineer</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>Head of Product</td>\n",
              "      <td></td>\n",
              "      <td>Head</td>\n",
              "      <td>Product</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>Senior Data Engineer</td>\n",
              "      <td>Data Engineer</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>Machine learning engineer</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>Senior Data Scientist</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>ML Engineer</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>Snr MLOps Engineer</td>\n",
              "      <td>MLOps Engineer</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>Senior ML engineer</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>Founding Data Scientist</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td>Founding</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>CTO</td>\n",
              "      <td>CTO</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>Head of AI</td>\n",
              "      <td></td>\n",
              "      <td>Head</td>\n",
              "      <td>AI</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>Product</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Product</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>Lead Data Scientist</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td>Lead</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>Co-founder</td>\n",
              "      <td></td>\n",
              "      <td>Co-founder</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>MLE</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td></td>\n",
              "      <td>MLE</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>Snr MLE</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>Director of Artificial Intelligence</td>\n",
              "      <td></td>\n",
              "      <td>Director</td>\n",
              "      <td>Artificial Intelligence</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>Data Scientist</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>Co-founder and CTO</td>\n",
              "      <td></td>\n",
              "      <td>Co-founder and CTO</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>Senior Software Enginner</td>\n",
              "      <td>Software Engineer</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>33</th>\n",
              "      <td>Co-founder</td>\n",
              "      <td></td>\n",
              "      <td>Co-founder</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>ML Developer</td>\n",
              "      <td>Machine Learning Developer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>data scientist</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>Senior Machine Learning Engineer</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>Founder</td>\n",
              "      <td></td>\n",
              "      <td>Founder</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>SRE</td>\n",
              "      <td></td>\n",
              "      <td>SRE</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>Cofounder (normally product manager, but also wearer of many hats)</td>\n",
              "      <td></td>\n",
              "      <td>Co-founder</td>\n",
              "      <td>Product Manager</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>Data Architect</td>\n",
              "      <td>Data Architect</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>42</th>\n",
              "      <td>Senior software engineer</td>\n",
              "      <td>Software Engineer</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43</th>\n",
              "      <td>consultant</td>\n",
              "      <td></td>\n",
              "      <td>Consultant</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>CTO</td>\n",
              "      <td></td>\n",
              "      <td>CTO</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>Data scientist / co-founder</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td></td>\n",
              "      <td>Co-founder</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>Analytics engineer</td>\n",
              "      <td>Analytics Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>47</th>\n",
              "      <td>Director, Innovation</td>\n",
              "      <td></td>\n",
              "      <td>Director</td>\n",
              "      <td>Innovation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>48</th>\n",
              "      <td>ML &amp; MLOps practice lead</td>\n",
              "      <td>ML &amp; MLOps Practice Lead</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>Founder</td>\n",
              "      <td>Founder</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>CTO</td>\n",
              "      <td>CTO</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>51</th>\n",
              "      <td>MLOps Engineer</td>\n",
              "      <td>MLOps Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>Junior developer</td>\n",
              "      <td></td>\n",
              "      <td>Junior</td>\n",
              "      <td>Developer</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53</th>\n",
              "      <td>deep learning engineer</td>\n",
              "      <td>Deep Learning Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>54</th>\n",
              "      <td>Co-Founder</td>\n",
              "      <td>Co-Founder</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>55</th>\n",
              "      <td>Intern</td>\n",
              "      <td></td>\n",
              "      <td>Intern</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>CTO</td>\n",
              "      <td>CTO</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57</th>\n",
              "      <td>Software Development Engineer</td>\n",
              "      <td>Software Development Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>58</th>\n",
              "      <td>Data Science Team Lead</td>\n",
              "      <td>Data Science Team Lead</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>59</th>\n",
              "      <td>Sr. ML Eng</td>\n",
              "      <td>Sr. ML Eng</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>60</th>\n",
              "      <td>Senior AI Expert</td>\n",
              "      <td>Senior AI Expert</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>Data Science Engineer</td>\n",
              "      <td>Data Science Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>62</th>\n",
              "      <td>Head of Data and ML</td>\n",
              "      <td></td>\n",
              "      <td>Head</td>\n",
              "      <td>Data and ML</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>63</th>\n",
              "      <td>ML/MLOps Engineer</td>\n",
              "      <td>ML/MLOps Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>64</th>\n",
              "      <td>Data Scientist, HappyFresh</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td></td>\n",
              "      <td>HappyFresh</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>Data Scientist</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66</th>\n",
              "      <td>Data Scientist</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>67</th>\n",
              "      <td>Senior Data Scientist</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>Data Scientist</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>69</th>\n",
              "      <td>Head of Data Science</td>\n",
              "      <td></td>\n",
              "      <td>Head</td>\n",
              "      <td>Data Science</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>70</th>\n",
              "      <td>Founder</td>\n",
              "      <td></td>\n",
              "      <td>Founder</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>71</th>\n",
              "      <td>CEO</td>\n",
              "      <td>CEO</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>72</th>\n",
              "      <td>Founder</td>\n",
              "      <td></td>\n",
              "      <td>Founder</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>73</th>\n",
              "      <td>Senior Dara scientist</td>\n",
              "      <td>Senior Data Scientist</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>74</th>\n",
              "      <td>ML Engineer</td>\n",
              "      <td>ML Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75</th>\n",
              "      <td>Director</td>\n",
              "      <td></td>\n",
              "      <td>Director</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>76</th>\n",
              "      <td>Technical Marketing Manager</td>\n",
              "      <td>Technical Marketing Manager</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>77</th>\n",
              "      <td>Head of MLOps</td>\n",
              "      <td></td>\n",
              "      <td>Head</td>\n",
              "      <td>MLOps</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>78</th>\n",
              "      <td>CDO</td>\n",
              "      <td></td>\n",
              "      <td>CDO</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>79</th>\n",
              "      <td>Owner</td>\n",
              "      <td></td>\n",
              "      <td>Owner</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>80</th>\n",
              "      <td>Agricultue consultant</td>\n",
              "      <td></td>\n",
              "      <td>Agriculture Consultant</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>81</th>\n",
              "      <td>Data scientist</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>82</th>\n",
              "      <td>Product Owner Data Science</td>\n",
              "      <td>Product Owner Data Science</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>83</th>\n",
              "      <td>Team Lead DS &amp; MLE</td>\n",
              "      <td></td>\n",
              "      <td>Team Lead</td>\n",
              "      <td>DS &amp; MLE</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>84</th>\n",
              "      <td>Co-founder</td>\n",
              "      <td></td>\n",
              "      <td>Co-founder</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>85</th>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>86</th>\n",
              "      <td>CEO</td>\n",
              "      <td>CEO</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87</th>\n",
              "      <td>Principal Cloud Developer</td>\n",
              "      <td>Cloud Developer</td>\n",
              "      <td>Principal</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>88</th>\n",
              "      <td>CPO</td>\n",
              "      <td>CPO</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>89</th>\n",
              "      <td>Cloud MLOps Engineer</td>\n",
              "      <td>MLOps Engineer</td>\n",
              "      <td></td>\n",
              "      <td>Cloud</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>90</th>\n",
              "      <td>CEO / Co-Founder</td>\n",
              "      <td>CEO / Co-Founder</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>91</th>\n",
              "      <td>Senior MLOps engineer</td>\n",
              "      <td>MLOps Engineer</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>92</th>\n",
              "      <td>machine learning engineer</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>93</th>\n",
              "      <td>CEO</td>\n",
              "      <td>CEO</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>94</th>\n",
              "      <td>ML engineer</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td>ML</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>95</th>\n",
              "      <td>Domain Chapter Lead - Data Science and AI</td>\n",
              "      <td>Domain Chapter Lead</td>\n",
              "      <td></td>\n",
              "      <td>Data Science and AI</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>96</th>\n",
              "      <td>Senior Machine Learning Engineer</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>97</th>\n",
              "      <td>VP of ML</td>\n",
              "      <td>VP of ML</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>Technical Lead</td>\n",
              "      <td>Technical Lead</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>99</th>\n",
              "      <td>Solutions engineer</td>\n",
              "      <td>Solutions Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>100</th>\n",
              "      <td>Platform Engineer</td>\n",
              "      <td>Platform Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>101</th>\n",
              "      <td>Senior MLOps</td>\n",
              "      <td>MLOps</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>102</th>\n",
              "      <td>Data science manager</td>\n",
              "      <td>Data Science Manager</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>103</th>\n",
              "      <td>Data Scientist</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>Principal Data Scientist</td>\n",
              "      <td>Principal Data Scientist</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>105</th>\n",
              "      <td>Manager Machine learning</td>\n",
              "      <td>Machine Learning Manager</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>106</th>\n",
              "      <td>Data scientist</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>107</th>\n",
              "      <td>Tech Lead</td>\n",
              "      <td>Tech Lead</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>108</th>\n",
              "      <td>Sr. Director AI/ML</td>\n",
              "      <td></td>\n",
              "      <td>Senior Director</td>\n",
              "      <td>AI/ML</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>BDM</td>\n",
              "      <td></td>\n",
              "      <td>BDM</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f79cb45a-d223-480c-a634-73f950d1466a')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-f79cb45a-d223-480c-a634-73f950d1466a button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-f79cb45a-d223-480c-a634-73f950d1466a');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q1_question = questions[1]\n",
        "q1_inputs = results.iloc[:, 1]\n",
        "print(f'Question 1:', q1_question)\n",
        "# nothing to do\n",
        "q1_reviews = q1_inputs.to_frame()\n",
        "q1_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "x2iPOd_ppI7D",
        "outputId": "e6842263-2486-4993-f5b9-ad742498d069"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 1: How big is your organization? (number of employees)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "    How big is your organization? (number of employees)\n",
              "1                                                1,000+\n",
              "2                                                50-500\n",
              "3                                                  1-10\n",
              "4                                                  1-10\n",
              "5                                                1,000+\n",
              "6                                                50-500\n",
              "7                                                1,000+\n",
              "8                                                 10-50\n",
              "9                                                50-500\n",
              "10                                               50-500\n",
              "11                                               1,000+\n",
              "12                                               1,000+\n",
              "13                                               50-500\n",
              "14                                               50-500\n",
              "15                                               1,000+\n",
              "16                                               50-500\n",
              "17                                               50-500\n",
              "18                                               1,000+\n",
              "19                                                 1-10\n",
              "20                                                 1-10\n",
              "21                                            500-1,000\n",
              "22                                               1,000+\n",
              "23                                               1,000+\n",
              "24                                               50-500\n",
              "25                                                 1-10\n",
              "26                                               50-500\n",
              "27                                               50-500\n",
              "28                                               1,000+\n",
              "29                                               50-500\n",
              "30                                                 1-10\n",
              "31                                               50-500\n",
              "32                                               50-500\n",
              "33                                                 1-10\n",
              "34                                            500-1,000\n",
              "35                                                 1-10\n",
              "36                                            500-1,000\n",
              "37                                                10-50\n",
              "38                                                 1-10\n",
              "39                                               1,000+\n",
              "40                                                 1-10\n",
              "41                                               1,000+\n",
              "42                                               1,000+\n",
              "43                                                 1-10\n",
              "44                                               50-500\n",
              "45                                                 1-10\n",
              "46                                               50-500\n",
              "47                                               50-500\n",
              "48                                               50-500\n",
              "49                                                 1-10\n",
              "50                                                 1-10\n",
              "51                                               50-500\n",
              "52                                                 1-10\n",
              "53                                               1,000+\n",
              "54                                                10-50\n",
              "55                                               1,000+\n",
              "56                                                 1-10\n",
              "57                                               1,000+\n",
              "58                                                10-50\n",
              "59                                               50-500\n",
              "60                                               50-500\n",
              "61                                               50-500\n",
              "62                                                10-50\n",
              "63                                            500-1,000\n",
              "64                                            500-1,000\n",
              "65                                                10-50\n",
              "66                                               50-500\n",
              "67                                            500-1,000\n",
              "68                                               50-500\n",
              "69                                               1,000+\n",
              "70                                                 1-10\n",
              "71                                                 1-10\n",
              "72                                                 1-10\n",
              "73                                               1,000+\n",
              "74                                               1,000+\n",
              "75                                                 1-10\n",
              "76                                                10-50\n",
              "77                                               1,000+\n",
              "78                                               1,000+\n",
              "79                                                 1-10\n",
              "80                                               1,000+\n",
              "81                                                 1-10\n",
              "82                                               50-500\n",
              "83                                               50-500\n",
              "84                                                 1-10\n",
              "85                                               50-500\n",
              "86                                                 1-10\n",
              "87                                               50-500\n",
              "88                                                 1-10\n",
              "89                                                10-50\n",
              "90                                                 1-10\n",
              "91                                               1,000+\n",
              "92                                               1,000+\n",
              "93                                                 1-10\n",
              "94                                               50-500\n",
              "95                                               1,000+\n",
              "96                                               1,000+\n",
              "97                                               50-500\n",
              "98                                               50-500\n",
              "99                                            500-1,000\n",
              "100                                              50-500\n",
              "101                                              1,000+\n",
              "102                                              1,000+\n",
              "103                                           500-1,000\n",
              "104                                              1,000+\n",
              "105                                              1,000+\n",
              "106                                                1-10\n",
              "107                                              1,000+\n",
              "108                                              1,000+\n",
              "109                                              1,000+"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-c81d2e66-f410-4eae-b878-dd1fcabeaf2c\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>How big is your organization? (number of employees)</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>10-50</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>500-1,000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>33</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>500-1,000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>500-1,000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>10-50</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>42</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>47</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>48</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>51</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>54</th>\n",
              "      <td>10-50</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>55</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>58</th>\n",
              "      <td>10-50</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>59</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>60</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>62</th>\n",
              "      <td>10-50</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>63</th>\n",
              "      <td>500-1,000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>64</th>\n",
              "      <td>500-1,000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>10-50</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>67</th>\n",
              "      <td>500-1,000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>69</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>70</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>71</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>72</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>73</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>74</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>76</th>\n",
              "      <td>10-50</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>77</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>78</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>79</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>80</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>81</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>82</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>83</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>84</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>85</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>86</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>88</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>89</th>\n",
              "      <td>10-50</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>90</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>91</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>92</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>93</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>94</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>95</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>96</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>97</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>99</th>\n",
              "      <td>500-1,000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>100</th>\n",
              "      <td>50-500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>101</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>102</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>103</th>\n",
              "      <td>500-1,000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>105</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>106</th>\n",
              "      <td>1-10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>107</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>108</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>1,000+</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c81d2e66-f410-4eae-b878-dd1fcabeaf2c')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-c81d2e66-f410-4eae-b878-dd1fcabeaf2c button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-c81d2e66-f410-4eae-b878-dd1fcabeaf2c');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q2_question = questions[2]\n",
        "q2_inputs = results.iloc[:, 2]\n",
        "print(f'Question 2:', q2_question)\n",
        "# nothing to do\n",
        "q2_reviews = q2_inputs.to_frame()\n",
        "q2_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "qevAEclHpUFd",
        "outputId": "557f0228-3b9f-49dc-a0a5-8795f63e8943"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 2: Are you using LLM at in your organization?\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "    Are you using LLM at in your organization?\n",
              "1                                           No\n",
              "2                                          Yes\n",
              "3                                          Yes\n",
              "4                                          Yes\n",
              "5                                          Yes\n",
              "6                                           No\n",
              "7                                          Yes\n",
              "8                                           No\n",
              "9                                          Yes\n",
              "10                                         Yes\n",
              "11                                         Yes\n",
              "12                                         Yes\n",
              "13                                          No\n",
              "14                                         Yes\n",
              "15                                         Yes\n",
              "16                                          No\n",
              "17                                          No\n",
              "18                                          No\n",
              "19                                         Yes\n",
              "20                                         Yes\n",
              "21                                          No\n",
              "22                                         Yes\n",
              "23                                          No\n",
              "24                                          No\n",
              "25                                         Yes\n",
              "26                                         Yes\n",
              "27                                          No\n",
              "28                                         Yes\n",
              "29                                         Yes\n",
              "30                                         Yes\n",
              "31                                         Yes\n",
              "32                                          No\n",
              "33                                          No\n",
              "34                                         Yes\n",
              "35                                          No\n",
              "36                                          No\n",
              "37                                         Yes\n",
              "38                                         Yes\n",
              "39                                         Yes\n",
              "40                                          No\n",
              "41                                         Yes\n",
              "42                                          No\n",
              "43                                          No\n",
              "44                                         Yes\n",
              "45                                         Yes\n",
              "46                                         Yes\n",
              "47                                          No\n",
              "48                                         Yes\n",
              "49                                         Yes\n",
              "50                                         Yes\n",
              "51                                          No\n",
              "52                                         Yes\n",
              "53                                         Yes\n",
              "54                                          No\n",
              "55                                          No\n",
              "56                                         Yes\n",
              "57                                         Yes\n",
              "58                                         Yes\n",
              "59                                          No\n",
              "60                                         Yes\n",
              "61                                         Yes\n",
              "62                                          No\n",
              "63                                          No\n",
              "64                                          No\n",
              "65                                         Yes\n",
              "66                                         Yes\n",
              "67                                          No\n",
              "68                                         Yes\n",
              "69                                         Yes\n",
              "70                                         Yes\n",
              "71                                         Yes\n",
              "72                                         Yes\n",
              "73                                         Yes\n",
              "74                                         Yes\n",
              "75                                          No\n",
              "76                                         Yes\n",
              "77                                          No\n",
              "78                                          No\n",
              "79                                         Yes\n",
              "80                                          No\n",
              "81                                         Yes\n",
              "82                                          No\n",
              "83                                          No\n",
              "84                                         Yes\n",
              "85                                          No\n",
              "86                                          No\n",
              "87                                         Yes\n",
              "88                                         Yes\n",
              "89                                         Yes\n",
              "90                                         Yes\n",
              "91                                         Yes\n",
              "92                                          No\n",
              "93                                         Yes\n",
              "94                                          No\n",
              "95                                         Yes\n",
              "96                                          No\n",
              "97                                          No\n",
              "98                                         Yes\n",
              "99                                         Yes\n",
              "100                                         No\n",
              "101                                        Yes\n",
              "102                                        Yes\n",
              "103                                         No\n",
              "104                                        Yes\n",
              "105                                         No\n",
              "106                                        Yes\n",
              "107                                        Yes\n",
              "108                                        Yes\n",
              "109                                        Yes"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-4914ce49-ea97-46e7-8f9b-f435114a2d4c\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Are you using LLM at in your organization?</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>33</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>42</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>47</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>48</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>51</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>54</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>55</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>58</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>59</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>60</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>62</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>63</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>64</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>67</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>69</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>70</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>71</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>72</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>73</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>74</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>76</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>77</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>78</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>79</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>80</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>81</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>82</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>83</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>84</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>85</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>86</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>88</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>89</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>90</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>91</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>92</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>93</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>94</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>95</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>96</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>97</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>99</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>100</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>101</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>102</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>103</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>105</th>\n",
              "      <td>No</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>106</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>107</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>108</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4914ce49-ea97-46e7-8f9b-f435114a2d4c')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-4914ce49-ea97-46e7-8f9b-f435114a2d4c button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-4914ce49-ea97-46e7-8f9b-f435114a2d4c');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q3_question = questions[3]\n",
        "q3_instructions = \"\"\"I received answers from an MLOps survey about Large Language Models (LLMs).\n",
        "Your task is to extract all the Natural Language Processing (NLP) tasks and industry fields from each answer.\n",
        "You should use common NLP tasks and industry fields whenever possible to avoid synonyms and acronyms.\n",
        "If no tasks or fields are mentioned, you should use an empty string as a placeholder.\n",
        "You should output these information in the JSON lines format.\n",
        "You should generate one JSON line per answer.\n",
        "Here's an example of an answer:\n",
        "- analyze logs to answer questions and classify text\n",
        "Here's an example of the expected output:\n",
        "{\"fields\": \"Computer Security\", \"tasks\": \"Question Answering, Text Classification\"}\n",
        "Answers:\"\"\"\n",
        "q3_inputs = results.iloc[:, 3]\n",
        "print(f'Question 3:', q3_question)\n",
        "q3_reviews = associate(model, q3_instructions, q3_inputs)\n",
        "q3_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "0rbUA8bp2wZs",
        "outputId": "d68b3200-b476-40ec-d605-91acb5bddbb8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 3: What is your use case/use cases?\n",
            "- Inputs: 110\n",
            "- batch: 0 -> 20\n",
            "- tokens: 635\n",
            "- batch: 1 -> 20\n",
            "- tokens: 643\n",
            "- batch: 2 -> 20\n",
            "- tokens: 650\n",
            "- batch: 3 -> 1\n",
            "- tokens: 178\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                       What is your use case/use cases?  \\\n",
              "2                                                                                                    Text summarisation   \n",
              "3                                                                                Data annotation; summarization; search   \n",
              "4                                                                                  Content production and brainstorming   \n",
              "5                                                                                                            Healthcare   \n",
              "7                                                                                           Search, classification, NER   \n",
              "10                                 text embeddings for search, text generation to help the business people do their job   \n",
              "11                                                         Natural Language interface, domain specific content creation   \n",
              "12                                                                                     Chatbots and Text classification   \n",
              "14                                                                                                             Adapters   \n",
              "19                                                                 Summarization, topic extraction, root-cause analysis   \n",
              "20                                                                                                            Marketing   \n",
              "22                                    Scanning large text corpora using an LLM and some cases with text summary for now   \n",
              "25                                                                                                    AI for e-commerce   \n",
              "28                                   Novelty - ChatGPT is being explored to optimize code, critique written works, etc.   \n",
              "29                                                               Summarisation, Email generation, Information retrieval   \n",
              "30                                                                  Prompt autogeneration and synthetic data generation   \n",
              "34                                                                                                   Question Answering   \n",
              "37                                                                                  Question generation for accountants   \n",
              "38                                                                Zero-shot/Few Shot classification for data enrichment   \n",
              "39                                                                                                         analyze logs   \n",
              "41                                                                                  Search and chat on proprietary data   \n",
              "44                            Product (Software) recommendations, Internal Q&A, Marketing content, and Customer support   \n",
              "45                                                                           Information retrieval and text generation.   \n",
              "46                                                                                                              Chatbot   \n",
              "48                                                                                 Text classification, text generation   \n",
              "49                                                                                         Social media idea generation   \n",
              "50                                                                    Semantic data extraction and corporate governance   \n",
              "52                                                           Creating a chat bot to query data efficiently for managers   \n",
              "53                                                            extract information from image and text of medical domain   \n",
              "56                                                                                                               Coding   \n",
              "57   (Not in my org, at work I don’t do this, it’s personal) Cover letter generator based on resume and job description   \n",
              "58                                                                                   Topic sentiment evaluation on text   \n",
              "60                                                                                          Medical Diagnostics Support   \n",
              "61                                                                                                   Assistant chat bot   \n",
              "65                                                                                        Automation of report writing.   \n",
              "66                                                                                        LLM in Marketing, Custom data   \n",
              "68                                                                                              chatbot/voice assistant   \n",
              "69                                               Entity matching, customer service responses (souped up/targetted FAQ).   \n",
              "70                                                   We are making tools to monitor and improve the performance of LLMs   \n",
              "71                                                                              Embeddings, Feature Extraction, Chatbot   \n",
              "72                                                                                  Text summarization, code generation   \n",
              "73                                                                                             Chatbot, semantic search   \n",
              "74                                                                                                     Customer support   \n",
              "76                                                                                  Content marketing and text drafting   \n",
              "79                                                  sales pipelines, text generation, data classification and many more   \n",
              "84                                                                  Semantic vector embeddings as a feature into RecSys   \n",
              "87                                      text resume, make ppts, text quizzes, text to flash cards, video resume etc,etc   \n",
              "88                                                             We help organizations transition Excel reports to Python   \n",
              "89                                                                                     Translate french text to english   \n",
              "90                                                                           Writing SQL Queries from Natural Language.   \n",
              "91                                                                                       Adverse media about suppliers.   \n",
              "93                                  Development of production tools for deployment/maintenance of LLMs and other models   \n",
              "95                                                                                      Summarisation of customer calls   \n",
              "98                                                                                       POC, evaluation for healthtech   \n",
              "99                                                                                       Summarizes notes from meetings   \n",
              "101                                                                                                             Chatbot   \n",
              "102                                                                                                      classification   \n",
              "104                                                Knowledge graph, transcription, topic modeling, embedding everything   \n",
              "106                                                                    User facing chat application in a narrow domain.   \n",
              "108                                                                                             ITSM, Sales Prospecting   \n",
              "109                                                                                     code development, summarisation   \n",
              "\n",
              "                                      fields  \\\n",
              "2                                              \n",
              "3                                              \n",
              "4                           Content Creation   \n",
              "5                                 Healthcare   \n",
              "7                                              \n",
              "10                                  Business   \n",
              "11                                             \n",
              "12                          Customer Service   \n",
              "14                                             \n",
              "19                                             \n",
              "20                                 Marketing   \n",
              "22                                             \n",
              "25                                E-commerce   \n",
              "28                                             \n",
              "29                                             \n",
              "30                                             \n",
              "34                                             \n",
              "37                                Accounting   \n",
              "38                                             \n",
              "39                                             \n",
              "41                                             \n",
              "44   E-commerce, Customer Support, Marketing   \n",
              "45                                             \n",
              "46                                             \n",
              "48                                             \n",
              "49                              Social Media   \n",
              "50                      Corporate Governance   \n",
              "52                                             \n",
              "53                                   Medical   \n",
              "56                                             \n",
              "57                                             \n",
              "58                                             \n",
              "60                                   Medical   \n",
              "61                                             \n",
              "65                                             \n",
              "66                    Marketing, Custom Data   \n",
              "68                                             \n",
              "69                          Customer Service   \n",
              "70                                             \n",
              "71                                             \n",
              "72                                             \n",
              "73                                             \n",
              "74                          Customer Support   \n",
              "76                         Content Marketing   \n",
              "79                                     Sales   \n",
              "84                       Recommender Systems   \n",
              "87                                             \n",
              "88                                             \n",
              "89                                             \n",
              "90                                             \n",
              "91                           Risk Management   \n",
              "93                         Model Development   \n",
              "95                          Customer Support   \n",
              "98                                Healthcare   \n",
              "99                                             \n",
              "101                                            \n",
              "102                                            \n",
              "104                                            \n",
              "106                                            \n",
              "108                                       IT   \n",
              "109                                            \n",
              "\n",
              "                                                                                   tasks  \n",
              "2                                                                     Text Summarization  \n",
              "3                                            Data Annotation, Text Summarization, Search  \n",
              "4                                                                                         \n",
              "5                                                                                         \n",
              "7                                  Search, Text Classification, Named Entity Recognition  \n",
              "10                                                      Text Embeddings, Text Generation  \n",
              "11                                          Natural Language Interface, Content Creation  \n",
              "12                                                         Chatbots, Text Classification  \n",
              "14                                                                              Adapters  \n",
              "19                             Text Summarization, Topic Extraction, Root-Cause Analysis  \n",
              "20                                                                                        \n",
              "22                                                     Text Scanning, Text Summarization  \n",
              "25                                                                                        \n",
              "28                                              Code Optimization, Written Work Critique  \n",
              "29                           Text Summarization, Email Generation, Information Retrieval  \n",
              "30                                      Prompt Autogeneration, Synthetic Data Generation  \n",
              "34                                                                    Question Answering  \n",
              "37                                                                   Question Generation  \n",
              "38                                    Zero-shot/Few Shot Classification, Data Enrichment  \n",
              "39                                                                          Log Analysis  \n",
              "41                                                                       Search, Chatbot  \n",
              "44                                   Recommendation, Question Answering, Text Generation  \n",
              "45                                                Information Retrieval, Text Generation  \n",
              "46                                                                               Chatbot  \n",
              "48                                                  Text Classification, Text Generation  \n",
              "49                                                                       Text Generation  \n",
              "50                                                              Semantic Data Extraction  \n",
              "52                                                        Chatbot, Information Retrieval  \n",
              "53                           Information Extraction, Image Analysis, Text Classification  \n",
              "56                                                                                Coding  \n",
              "57                                                                       Text Generation  \n",
              "58                                                    Sentiment Analysis, Topic Modeling  \n",
              "60                                                                   Diagnostics Support  \n",
              "61                                                                               Chatbot  \n",
              "65                                                                     Report Generation  \n",
              "66                                                                       Text Generation  \n",
              "68                                                              Chatbot, Voice Assistant  \n",
              "69                                                  Entity Matching, Text Classification  \n",
              "70                                             Performance Monitoring, Model Improvement  \n",
              "71                                               Embeddings, Feature Extraction, Chatbot  \n",
              "72                                                   Text Summarization, Code Generation  \n",
              "73                                                              Chatbot, Semantic Search  \n",
              "74                                                                                        \n",
              "76                                                                         Text Drafting  \n",
              "79                                                  Text Generation, Data Classification  \n",
              "84                                                            Semantic Vector Embeddings  \n",
              "87   Text Resume, PowerPoint Generation, Text Quizzes, Text to Flash Cards, Video Resume  \n",
              "88                                                     Excel Report Transition to Python  \n",
              "89                                                                           Translation  \n",
              "90                                            SQL Query Generation from Natural Language  \n",
              "91                                                                Adverse Media Analysis  \n",
              "93                                 Production Tool Development for LLMs and Other Models  \n",
              "95                                                                    Call Summarization  \n",
              "98                                                                       POC, Evaluation  \n",
              "99                                                            Meeting Note Summarization  \n",
              "101                                                                              Chatbot  \n",
              "102                                                                       Classification  \n",
              "104                            Knowledge Graph, Transcription, Topic Modeling, Embedding  \n",
              "106                                                         User-Facing Chat Application  \n",
              "108                                                              ITSM, Sales Prospecting  \n",
              "109                                                      Code Development, Summarization  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-2ef39d1e-15fb-4618-8bd3-9010fc52496a\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>What is your use case/use cases?</th>\n",
              "      <th>fields</th>\n",
              "      <th>tasks</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Text summarisation</td>\n",
              "      <td></td>\n",
              "      <td>Text Summarization</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Data annotation; summarization; search</td>\n",
              "      <td></td>\n",
              "      <td>Data Annotation, Text Summarization, Search</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Content production and brainstorming</td>\n",
              "      <td>Content Creation</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>Healthcare</td>\n",
              "      <td>Healthcare</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Search, classification, NER</td>\n",
              "      <td></td>\n",
              "      <td>Search, Text Classification, Named Entity Recognition</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>text embeddings for search, text generation to help the business people do their job</td>\n",
              "      <td>Business</td>\n",
              "      <td>Text Embeddings, Text Generation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>Natural Language interface, domain specific content creation</td>\n",
              "      <td></td>\n",
              "      <td>Natural Language Interface, Content Creation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>Chatbots and Text classification</td>\n",
              "      <td>Customer Service</td>\n",
              "      <td>Chatbots, Text Classification</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>Adapters</td>\n",
              "      <td></td>\n",
              "      <td>Adapters</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>Summarization, topic extraction, root-cause analysis</td>\n",
              "      <td></td>\n",
              "      <td>Text Summarization, Topic Extraction, Root-Cause Analysis</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>Marketing</td>\n",
              "      <td>Marketing</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>Scanning large text corpora using an LLM and some cases with text summary for now</td>\n",
              "      <td></td>\n",
              "      <td>Text Scanning, Text Summarization</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>AI for e-commerce</td>\n",
              "      <td>E-commerce</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>Novelty - ChatGPT is being explored to optimize code, critique written works, etc.</td>\n",
              "      <td></td>\n",
              "      <td>Code Optimization, Written Work Critique</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>Summarisation, Email generation, Information retrieval</td>\n",
              "      <td></td>\n",
              "      <td>Text Summarization, Email Generation, Information Retrieval</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>Prompt autogeneration and synthetic data generation</td>\n",
              "      <td></td>\n",
              "      <td>Prompt Autogeneration, Synthetic Data Generation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>Question Answering</td>\n",
              "      <td></td>\n",
              "      <td>Question Answering</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>Question generation for accountants</td>\n",
              "      <td>Accounting</td>\n",
              "      <td>Question Generation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>Zero-shot/Few Shot classification for data enrichment</td>\n",
              "      <td></td>\n",
              "      <td>Zero-shot/Few Shot Classification, Data Enrichment</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>analyze logs</td>\n",
              "      <td></td>\n",
              "      <td>Log Analysis</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>Search and chat on proprietary data</td>\n",
              "      <td></td>\n",
              "      <td>Search, Chatbot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>Product (Software) recommendations, Internal Q&amp;A, Marketing content, and Customer support</td>\n",
              "      <td>E-commerce, Customer Support, Marketing</td>\n",
              "      <td>Recommendation, Question Answering, Text Generation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>Information retrieval and text generation.</td>\n",
              "      <td></td>\n",
              "      <td>Information Retrieval, Text Generation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>Chatbot</td>\n",
              "      <td></td>\n",
              "      <td>Chatbot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>48</th>\n",
              "      <td>Text classification, text generation</td>\n",
              "      <td></td>\n",
              "      <td>Text Classification, Text Generation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>Social media idea generation</td>\n",
              "      <td>Social Media</td>\n",
              "      <td>Text Generation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>Semantic data extraction and corporate governance</td>\n",
              "      <td>Corporate Governance</td>\n",
              "      <td>Semantic Data Extraction</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>Creating a chat bot to query data efficiently for managers</td>\n",
              "      <td></td>\n",
              "      <td>Chatbot, Information Retrieval</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53</th>\n",
              "      <td>extract information from image and text of medical domain</td>\n",
              "      <td>Medical</td>\n",
              "      <td>Information Extraction, Image Analysis, Text Classification</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>Coding</td>\n",
              "      <td></td>\n",
              "      <td>Coding</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57</th>\n",
              "      <td>(Not in my org, at work I don’t do this, it’s personal) Cover letter generator based on resume and job description</td>\n",
              "      <td></td>\n",
              "      <td>Text Generation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>58</th>\n",
              "      <td>Topic sentiment evaluation on text</td>\n",
              "      <td></td>\n",
              "      <td>Sentiment Analysis, Topic Modeling</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>60</th>\n",
              "      <td>Medical Diagnostics Support</td>\n",
              "      <td>Medical</td>\n",
              "      <td>Diagnostics Support</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>Assistant chat bot</td>\n",
              "      <td></td>\n",
              "      <td>Chatbot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>Automation of report writing.</td>\n",
              "      <td></td>\n",
              "      <td>Report Generation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66</th>\n",
              "      <td>LLM in Marketing, Custom data</td>\n",
              "      <td>Marketing, Custom Data</td>\n",
              "      <td>Text Generation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>chatbot/voice assistant</td>\n",
              "      <td></td>\n",
              "      <td>Chatbot, Voice Assistant</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>69</th>\n",
              "      <td>Entity matching, customer service responses (souped up/targetted FAQ).</td>\n",
              "      <td>Customer Service</td>\n",
              "      <td>Entity Matching, Text Classification</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>70</th>\n",
              "      <td>We are making tools to monitor and improve the performance of LLMs</td>\n",
              "      <td></td>\n",
              "      <td>Performance Monitoring, Model Improvement</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>71</th>\n",
              "      <td>Embeddings, Feature Extraction, Chatbot</td>\n",
              "      <td></td>\n",
              "      <td>Embeddings, Feature Extraction, Chatbot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>72</th>\n",
              "      <td>Text summarization, code generation</td>\n",
              "      <td></td>\n",
              "      <td>Text Summarization, Code Generation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>73</th>\n",
              "      <td>Chatbot, semantic search</td>\n",
              "      <td></td>\n",
              "      <td>Chatbot, Semantic Search</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>74</th>\n",
              "      <td>Customer support</td>\n",
              "      <td>Customer Support</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>76</th>\n",
              "      <td>Content marketing and text drafting</td>\n",
              "      <td>Content Marketing</td>\n",
              "      <td>Text Drafting</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>79</th>\n",
              "      <td>sales pipelines, text generation, data classification and many more</td>\n",
              "      <td>Sales</td>\n",
              "      <td>Text Generation, Data Classification</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>84</th>\n",
              "      <td>Semantic vector embeddings as a feature into RecSys</td>\n",
              "      <td>Recommender Systems</td>\n",
              "      <td>Semantic Vector Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87</th>\n",
              "      <td>text resume, make ppts, text quizzes, text to flash cards, video resume etc,etc</td>\n",
              "      <td></td>\n",
              "      <td>Text Resume, PowerPoint Generation, Text Quizzes, Text to Flash Cards, Video Resume</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>88</th>\n",
              "      <td>We help organizations transition Excel reports to Python</td>\n",
              "      <td></td>\n",
              "      <td>Excel Report Transition to Python</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>89</th>\n",
              "      <td>Translate french text to english</td>\n",
              "      <td></td>\n",
              "      <td>Translation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>90</th>\n",
              "      <td>Writing SQL Queries from Natural Language.</td>\n",
              "      <td></td>\n",
              "      <td>SQL Query Generation from Natural Language</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>91</th>\n",
              "      <td>Adverse media about suppliers.</td>\n",
              "      <td>Risk Management</td>\n",
              "      <td>Adverse Media Analysis</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>93</th>\n",
              "      <td>Development of production tools for deployment/maintenance of LLMs and other models</td>\n",
              "      <td>Model Development</td>\n",
              "      <td>Production Tool Development for LLMs and Other Models</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>95</th>\n",
              "      <td>Summarisation of customer calls</td>\n",
              "      <td>Customer Support</td>\n",
              "      <td>Call Summarization</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>POC, evaluation for healthtech</td>\n",
              "      <td>Healthcare</td>\n",
              "      <td>POC, Evaluation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>99</th>\n",
              "      <td>Summarizes notes from meetings</td>\n",
              "      <td></td>\n",
              "      <td>Meeting Note Summarization</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>101</th>\n",
              "      <td>Chatbot</td>\n",
              "      <td></td>\n",
              "      <td>Chatbot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>102</th>\n",
              "      <td>classification</td>\n",
              "      <td></td>\n",
              "      <td>Classification</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>Knowledge graph, transcription, topic modeling, embedding everything</td>\n",
              "      <td></td>\n",
              "      <td>Knowledge Graph, Transcription, Topic Modeling, Embedding</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>106</th>\n",
              "      <td>User facing chat application in a narrow domain.</td>\n",
              "      <td></td>\n",
              "      <td>User-Facing Chat Application</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>108</th>\n",
              "      <td>ITSM, Sales Prospecting</td>\n",
              "      <td>IT</td>\n",
              "      <td>ITSM, Sales Prospecting</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>code development, summarisation</td>\n",
              "      <td></td>\n",
              "      <td>Code Development, Summarization</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2ef39d1e-15fb-4618-8bd3-9010fc52496a')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-2ef39d1e-15fb-4618-8bd3-9010fc52496a button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-2ef39d1e-15fb-4618-8bd3-9010fc52496a');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q4_question = questions[4]\n",
        "q4_instructions = \"\"\"I received answers from an MLOps survey about Large Language Models (LLMs).\n",
        "Your task is to extract all solutions and purposes mentionned in each answer.\n",
        "You should use keywords as much as possible to summarize the user answers.\n",
        "If no solutions or purposes are mentioned, you should use an empty string as a placeholder.\n",
        "You should also return if a solution was integrated or not using \"Yes\" and \"No\".\n",
        "You should output these information in the JSON lines format.\n",
        "You should generate one JSON line per answer.\n",
        "Here's an example of an answer:\n",
        "- In POC at the moment\n",
        "- No just used langchain and OpenAI\n",
        "- just used RASA for chatbot creation and haystack for open domain question answering\n",
        "- Internally, we use BERT to create embeddings, then train a smaller model on customer data.\n",
        "Here's an example of the expected output:\n",
        "{\"integrated\": \"No\", \"solutions\": \"\", \"purposes\": \"\"}\n",
        "{\"integrated\": \"No\", \"solutions\": \"LangChain, OpenAI\", \"purposes\": \"\"}\n",
        "{\"integrated\": \"Yes\", \"solutions\": \"RASA, Haystack\", \"purposes\": \"Chatbot, Question Answering\"}\n",
        "{\"integrated\": \"Yes\", \"solutions\": \"BERT\", \"purposes\": \"Embeddings\"}\n",
        "Answers:\"\"\"\n",
        "q4_inputs = results.iloc[:, 4]\n",
        "print(f'Question 4:', q4_question)\n",
        "q4_reviews = associate(model, q4_instructions, q4_inputs)\n",
        "q4_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "LLoj8lKi8goI",
        "outputId": "4fe7b9f6-a586-4952-ef1f-6ab82cca38e5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 4: Have you integrated or built any internal tools to support LLMs in your org? If so what?\n",
            "- Inputs: 110\n",
            "- batch: 0 -> 20\n",
            "- tokens: 881\n",
            "Warning! Got: 18, expected: 20\n",
            "- batch: 1 -> 20\n",
            "- tokens: 1029\n",
            "Warning! Got: 18, expected: 20\n",
            "- batch: 2 -> 18\n",
            "- tokens: 827\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                                                                                                                                                    Have you integrated or built any internal tools to support LLMs in your org? If so what?  \\\n",
              "2                                                                                                                                                                                                                                                                                                                                                         No   \n",
              "3                                                                                                                                                                                                                                                                                                      https://chat.mantisnlp.com, internal annotation tools   \n",
              "4                                                                                                                                                                                                                                                                                                                                                    Not yet   \n",
              "5                                                                                                                                                                                                                                                                                                                                          yes, confidential   \n",
              "7                                                                                                                                                                                                                                                                                           Just a bunch of shitty scripts I need to refactor in the future.   \n",
              "9                                                                                                                                                                                                                                                                                                                                                         oi   \n",
              "10                                                                                                                                                                                                                                                                                                   Yes, we're using OpenAI for a couple of internal tools.   \n",
              "12                                                                                                                                                                                               One of our use cases is to build text classification models for each of our customers (we are a B2B company). Achieved this using LLMs and some engineering   \n",
              "14                                                                                                                                                                                                                                                                                                            Yes, training setup as well as inference setup   \n",
              "19                                                                                                                                                                                                                                                         Not much, just some prompt storage and an automated retries to deal with different failure cases.   \n",
              "20                                                                                                                                                                                                                                                                                                                                                   Not yet   \n",
              "22                                                                                                                                                                                                                                                                                                                                                Not (yet).   \n",
              "25                                                                                                                                                                                                                                                                                                                                                        No   \n",
              "26                                                                                                                                                                                                                                                                                      dbt pre-processing to get the text right for the prompts and test it   \n",
              "28                                                                                                                                                                                                                                                                                                                                                        No   \n",
              "29                                                                                                                                                                                                                                   Modular python library for interacting with multiple LLM providers/APIs and a standardized prompt generation framework.   \n",
              "34                                                                                                                                                                                                                                                                                                                                                       yes   \n",
              "37                                                                                                                                                                                                                                                                                                                                            built internal   \n",
              "41                                                                                                                                                                                                                                                                                                                                                        No   \n",
              "44                                                                                                                                                                                                                                                                                              Langchain, OpenAI Playground, ChatGPT, PromptLayer, Weaviate   \n",
              "45                                                                                                                                                                                                                                                                   We have some output validation methods and templates, basically regex matching prompts.   \n",
              "46                                                                                                                                                                                                                                                                                                                                      In POC at the moment   \n",
              "49                                                                                                                                                                                                                                                                                                                         No just used langchain and OpenAI   \n",
              "50                                                                                   I'm unable to utilise LLM's currently due to data security restrictions, however as a OpenAI beta tester, around 2 years ago I created a business Operational Delivery Framework for a £500M nuclear project. I then transferred the learning framework to £5B project.   \n",
              "52                                                                                                                                                                                                                                                                                                           Not yet but its in production. Using langchain.   \n",
              "53                                                                                                                                                                                                                                                                                                                                                        no   \n",
              "56                                                                                                                                                                                                                                                                                                                                          VSCode Extension   \n",
              "60                                                                                                                                                                                                                                                                                                                          No, we rely on open source tools   \n",
              "61                                                                                                                                                                                                                                              Langchain. Will serve the bot from a flask server, being set up for the bot, into our existing java platform   \n",
              "65   Yes.\\nResponsible for the design, building, testing and deployment of a GPT-3.5 application that processes numerical data and free-response text in tabular form to generates commentary and analysis for a visual report. Researchers are only required to upload a zip folder(1-click).\\nTechnologies: AWS lambda, Docker (ECR/ECS), GPT3.5, PyTorch.   \n",
              "66                                                                                                                                                                                                                                                                                                                                    HF Inference Endpoints   \n",
              "68                                                                                                                                                                                                                                                                       just used RASA for chatbot creation and haystack for open domain question answering   \n",
              "69                                                                                                                                                                                                                                                                                                                           yes - high scale server on K8's   \n",
              "70                                                                                                                                                                                                                                                                                                                                       No additional tools   \n",
              "71                                                                                                                                                                                                                                                                                                                                      Database connections   \n",
              "72                                                                                                                                                                                                                                                                                                                                                        No   \n",
              "76                                                                                                                                                                                                                                                                                                                        not at the moment in my department   \n",
              "79                                                                                                                                                                                                                                                                                                                                              Yes, several   \n",
              "81                                                                                                                                                                                                                                                                                                                                                       Yes   \n",
              "84                                                                                                                                                                                                                                                                                                                                                   Not yet   \n",
              "87                                                                                                                                                                                                                                                                                                                                                         -   \n",
              "88                                                                                     Yes -- our product, Mito, is a spreadsheet that generates Python code as you edit it. We have integrated the chatgpt api into Mito so users can generate code using the LLM. Then, they can use the Mito spreadsheet to verify that the code it generated is correct.   \n",
              "89                                                                                                                                                                                                                                                                               Yes we have used opus-mt model in containerised environment for translation   \n",
              "90                                                                                                                                                                                                                                                                                                                                                        No   \n",
              "91                                                                                                                                                                                                                                                                        No, but have plans to do so! I will be involved in the feature store architecture.   \n",
              "93                                                                                                                                                                                                                                                                                    We have a platform to develop/run models in production called Statfish   \n",
              "95                                                                                                                                                                                                                                                                                                                                                      nope   \n",
              "99                                                                                                                                                                                                                                                                                                                                                       no.   \n",
              "101                                                                                                                                                                                                                                                                                                                                                integrate   \n",
              "102                                                                                                                                                                                                                                                                                                                                          no custom tools   \n",
              "104                                                                                                                                                                                                                                                                                                     Yeah, but it's super secret until we try and sell it   \n",
              "106                                                                                                                                                                                                                                                                                                                                              In progress   \n",
              "108                                                                                                                                                                                                                                                                                                                                                 chat bot   \n",
              "109                                                                                                                                                                                                                                                                                               development environment, pipelining and deployment tooling   \n",
              "\n",
              "    integrated                                                     solutions  \\\n",
              "2           No                                                                 \n",
              "3           No                          MantisNLP, internal annotation tools   \n",
              "4           No                                                                 \n",
              "5          Yes                                                  Confidential   \n",
              "7           No                                        Refactoring of scripts   \n",
              "9           No                                                            OI   \n",
              "10         Yes                                                        OpenAI   \n",
              "12         Yes                                                          LLMs   \n",
              "14         Yes                                                          LLMs   \n",
              "19          No                             Prompt storage, automated retries   \n",
              "20          No                                                                 \n",
              "22          No                                                                 \n",
              "25          No                                                                 \n",
              "26          No                                            dbt pre-processing   \n",
              "28          No                                                                 \n",
              "29          No                                        Modular python library   \n",
              "34         Yes                                                          BERT   \n",
              "37          No                                                                 \n",
              "41          No                                                                 \n",
              "44          No  Langchain, OpenAI Playground, ChatGPT, PromptLayer, Weaviate   \n",
              "45          No                                                                 \n",
              "46          No                                                                 \n",
              "49          No                                             Langchain, OpenAI   \n",
              "50          No                                                                 \n",
              "52         Yes                                                     Langchain   \n",
              "53          No                                                                 \n",
              "56          No                                                                 \n",
              "60          No                                                     Langchain   \n",
              "61         Yes                 GPT3.5, PyTorch, AWS Lambda, Docker (ECR/ECS)   \n",
              "65          No                                        HF Inference Endpoints   \n",
              "66         Yes                                                RASA, Haystack   \n",
              "68         Yes                                                                 \n",
              "69          No                                                                 \n",
              "70          No                                                                 \n",
              "71         Yes                                               Open source LLM   \n",
              "72         Yes                                                   Model store   \n",
              "76          No                                                                 \n",
              "79         Yes                                                   chatgpt api   \n",
              "81         Yes                                                 opus-mt model   \n",
              "84          No                                                                 \n",
              "87                                                                             \n",
              "88         Yes                                                   chatgpt api   \n",
              "89         Yes                                                 opus-mt model   \n",
              "90          No                                                                 \n",
              "91          No                                                                 \n",
              "93                                                                  Statfish   \n",
              "95          No                                                                 \n",
              "99          No                                                                 \n",
              "101                                                                            \n",
              "102         No                                                                 \n",
              "104                                                                            \n",
              "106                                                                            \n",
              "108                                                                            \n",
              "109                                                                            \n",
              "\n",
              "                                                                                   purposes  \n",
              "2                                                                                            \n",
              "3                                                                                            \n",
              "4                                                                                            \n",
              "5                                                                                            \n",
              "7                                                                                            \n",
              "9                                                                                            \n",
              "10                                                                           Internal tools  \n",
              "12                                                    Text classification for B2B customers  \n",
              "14                                                             Training and inference setup  \n",
              "19                                                                                           \n",
              "20                                                                                           \n",
              "22                                                                                           \n",
              "25                                                                                           \n",
              "26                                                                           Prompt testing  \n",
              "28                                                                                           \n",
              "29   Interacting with multiple LLM providers/APIs, standardized prompt generation framework  \n",
              "34                                                                               Embeddings  \n",
              "37                                                                                           \n",
              "41                                                                                           \n",
              "44                                                                                           \n",
              "45                                                                        Output validation  \n",
              "46                                                                                      POC  \n",
              "49                                                                                           \n",
              "50                                                           Operational Delivery Framework  \n",
              "52                                                                               Production  \n",
              "53                                                                         VSCode Extension  \n",
              "56                                                                        Open source tools  \n",
              "60                                                                              Bot serving  \n",
              "61               Numerical data processing, Commentary generation, Visual report generation  \n",
              "65                                                                                           \n",
              "66                                                              Chatbot, Question Answering  \n",
              "68                                                                High scale server on K8's  \n",
              "69                                                                         Additional tools  \n",
              "70                                                                     Database connections  \n",
              "71                                                                               Deployment  \n",
              "72                                                                                           \n",
              "76                                                                                           \n",
              "79                                                                          Code generation  \n",
              "81                                                                              Translation  \n",
              "84                                                                                           \n",
              "87                                                                                           \n",
              "88                                                                        Code verification  \n",
              "89                                                                              Translation  \n",
              "90                                                                                           \n",
              "91                                                               Feature store architecture  \n",
              "93                                                         Model development and production  \n",
              "95                                                                                           \n",
              "99                                                                                           \n",
              "101                                                                             Integration  \n",
              "102                                                                                          \n",
              "104                                                                          Secret project  \n",
              "106                                                                             In progress  \n",
              "108                                                                                 Chatbot  \n",
              "109                              Development environment, pipelining and deployment tooling  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-935ed1f1-d4cd-463a-bcab-89a44e3e0f05\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Have you integrated or built any internal tools to support LLMs in your org? If so what?</th>\n",
              "      <th>integrated</th>\n",
              "      <th>solutions</th>\n",
              "      <th>purposes</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>https://chat.mantisnlp.com, internal annotation tools</td>\n",
              "      <td>No</td>\n",
              "      <td>MantisNLP, internal annotation tools</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Not yet</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>yes, confidential</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Confidential</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Just a bunch of shitty scripts I need to refactor in the future.</td>\n",
              "      <td>No</td>\n",
              "      <td>Refactoring of scripts</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>oi</td>\n",
              "      <td>No</td>\n",
              "      <td>OI</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>Yes, we're using OpenAI for a couple of internal tools.</td>\n",
              "      <td>Yes</td>\n",
              "      <td>OpenAI</td>\n",
              "      <td>Internal tools</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>One of our use cases is to build text classification models for each of our customers (we are a B2B company). Achieved this using LLMs and some engineering</td>\n",
              "      <td>Yes</td>\n",
              "      <td>LLMs</td>\n",
              "      <td>Text classification for B2B customers</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>Yes, training setup as well as inference setup</td>\n",
              "      <td>Yes</td>\n",
              "      <td>LLMs</td>\n",
              "      <td>Training and inference setup</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>Not much, just some prompt storage and an automated retries to deal with different failure cases.</td>\n",
              "      <td>No</td>\n",
              "      <td>Prompt storage, automated retries</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>Not yet</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>Not (yet).</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>dbt pre-processing to get the text right for the prompts and test it</td>\n",
              "      <td>No</td>\n",
              "      <td>dbt pre-processing</td>\n",
              "      <td>Prompt testing</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>Modular python library for interacting with multiple LLM providers/APIs and a standardized prompt generation framework.</td>\n",
              "      <td>No</td>\n",
              "      <td>Modular python library</td>\n",
              "      <td>Interacting with multiple LLM providers/APIs, standardized prompt generation framework</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>yes</td>\n",
              "      <td>Yes</td>\n",
              "      <td>BERT</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>built internal</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>Langchain, OpenAI Playground, ChatGPT, PromptLayer, Weaviate</td>\n",
              "      <td>No</td>\n",
              "      <td>Langchain, OpenAI Playground, ChatGPT, PromptLayer, Weaviate</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>We have some output validation methods and templates, basically regex matching prompts.</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td>Output validation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>In POC at the moment</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td>POC</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>No just used langchain and OpenAI</td>\n",
              "      <td>No</td>\n",
              "      <td>Langchain, OpenAI</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>I'm unable to utilise LLM's currently due to data security restrictions, however as a OpenAI beta tester, around 2 years ago I created a business Operational Delivery Framework for a £500M nuclear project. I then transferred the learning framework to £5B project.</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td>Operational Delivery Framework</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>Not yet but its in production. Using langchain.</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Langchain</td>\n",
              "      <td>Production</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53</th>\n",
              "      <td>no</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td>VSCode Extension</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>VSCode Extension</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td>Open source tools</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>60</th>\n",
              "      <td>No, we rely on open source tools</td>\n",
              "      <td>No</td>\n",
              "      <td>Langchain</td>\n",
              "      <td>Bot serving</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>Langchain. Will serve the bot from a flask server, being set up for the bot, into our existing java platform</td>\n",
              "      <td>Yes</td>\n",
              "      <td>GPT3.5, PyTorch, AWS Lambda, Docker (ECR/ECS)</td>\n",
              "      <td>Numerical data processing, Commentary generation, Visual report generation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>Yes.\\nResponsible for the design, building, testing and deployment of a GPT-3.5 application that processes numerical data and free-response text in tabular form to generates commentary and analysis for a visual report. Researchers are only required to upload a zip folder(1-click).\\nTechnologies: AWS lambda, Docker (ECR/ECS), GPT3.5, PyTorch.</td>\n",
              "      <td>No</td>\n",
              "      <td>HF Inference Endpoints</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66</th>\n",
              "      <td>HF Inference Endpoints</td>\n",
              "      <td>Yes</td>\n",
              "      <td>RASA, Haystack</td>\n",
              "      <td>Chatbot, Question Answering</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>just used RASA for chatbot creation and haystack for open domain question answering</td>\n",
              "      <td>Yes</td>\n",
              "      <td></td>\n",
              "      <td>High scale server on K8's</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>69</th>\n",
              "      <td>yes - high scale server on K8's</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td>Additional tools</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>70</th>\n",
              "      <td>No additional tools</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td>Database connections</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>71</th>\n",
              "      <td>Database connections</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Open source LLM</td>\n",
              "      <td>Deployment</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>72</th>\n",
              "      <td>No</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Model store</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>76</th>\n",
              "      <td>not at the moment in my department</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>79</th>\n",
              "      <td>Yes, several</td>\n",
              "      <td>Yes</td>\n",
              "      <td>chatgpt api</td>\n",
              "      <td>Code generation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>81</th>\n",
              "      <td>Yes</td>\n",
              "      <td>Yes</td>\n",
              "      <td>opus-mt model</td>\n",
              "      <td>Translation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>84</th>\n",
              "      <td>Not yet</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87</th>\n",
              "      <td>-</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>88</th>\n",
              "      <td>Yes -- our product, Mito, is a spreadsheet that generates Python code as you edit it. We have integrated the chatgpt api into Mito so users can generate code using the LLM. Then, they can use the Mito spreadsheet to verify that the code it generated is correct.</td>\n",
              "      <td>Yes</td>\n",
              "      <td>chatgpt api</td>\n",
              "      <td>Code verification</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>89</th>\n",
              "      <td>Yes we have used opus-mt model in containerised environment for translation</td>\n",
              "      <td>Yes</td>\n",
              "      <td>opus-mt model</td>\n",
              "      <td>Translation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>90</th>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>91</th>\n",
              "      <td>No, but have plans to do so! I will be involved in the feature store architecture.</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td>Feature store architecture</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>93</th>\n",
              "      <td>We have a platform to develop/run models in production called Statfish</td>\n",
              "      <td></td>\n",
              "      <td>Statfish</td>\n",
              "      <td>Model development and production</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>95</th>\n",
              "      <td>nope</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>99</th>\n",
              "      <td>no.</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>101</th>\n",
              "      <td>integrate</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Integration</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>102</th>\n",
              "      <td>no custom tools</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>Yeah, but it's super secret until we try and sell it</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Secret project</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>106</th>\n",
              "      <td>In progress</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>In progress</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>108</th>\n",
              "      <td>chat bot</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Chatbot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>development environment, pipelining and deployment tooling</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Development environment, pipelining and deployment tooling</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-935ed1f1-d4cd-463a-bcab-89a44e3e0f05')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-935ed1f1-d4cd-463a-bcab-89a44e3e0f05 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-935ed1f1-d4cd-463a-bcab-89a44e3e0f05');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q5_question = questions[5]\n",
        "q5_instructions = \"\"\"I received answers from an MLOps survey about Large Language Models (LLMs).\n",
        "Your task is to extract the challenges mentioned in each answer.\n",
        "You should use high-level keywords whenever possible and avoid synonyms and acronyms.\n",
        "If no challenges are mentioned, you should use an empty string as a placeholder.\n",
        "You should output these information in the JSON lines format.\n",
        "You should generate one JSON line per answer.\n",
        "Here's an example of an answer:\n",
        "- Size in all regards. Also training time, response time and that multi gpu debugging is weird.\n",
        "- Slow. They give bad answers when you set the confidence requirements too high\n",
        "- Work with 3 AI providers at the same time\n",
        "Here's an example of the expected output:\n",
        "{\"challenges\": \"Performance,Latency,Debugging\"}\n",
        "{\"challenges\": \"Performance,Evaluation\"}\n",
        "{\"challenges\": \"Interoperability\"}\n",
        "Answers:\"\"\"\n",
        "q5_inputs = results.iloc[:, 5]\n",
        "print(f'Question 5:', q5_question)\n",
        "q5_reviews = associate(model, q5_instructions, q5_inputs)\n",
        "q5_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "tdNSBhEC9GeS",
        "outputId": "ae3c0824-fca6-4d02-a9ea-c7cc464f9a54"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 5: What are some of the main challenges you have encountered thus far when building with LLMs\n",
            "- Inputs: 110\n",
            "- batch: 0 -> 20\n",
            "- tokens: 602\n",
            "Warning! Got: 19, expected: 20\n",
            "- batch: 1 -> 20\n",
            "- tokens: 655\n",
            "- batch: 2 -> 19\n",
            "- tokens: 590\n",
            "Warning! Got: 17, expected: 19\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                                                                              What are some of the main challenges you have encountered thus far when building with LLMs  \\\n",
              "2                                                                                                                                                                           Current infrastructure not designed for such massive models, having to implement workarounds for quick fixes   \n",
              "3                                                                                                                                                                                                  out of date info, hallucinations, cost, difficulty in deploying on own infrastructure   \n",
              "4                                                                                                                                                                                                                                                                                not yet   \n",
              "5                                                                                                                                                                                                                                                                            reliability   \n",
              "7                                                                                                                                                                                          Size in all regards. Also training time, response time and that multi gpu debugging is weird.   \n",
              "9                                                                                                                                                                                                                                                                                    lik   \n",
              "11                                                                                                                                                                                                                                     Prompt debugging! We need a new form of debugger.   \n",
              "14                                                                                                                                                                                                                                                           Latency and deployment time   \n",
              "19                                                                                  Hallucinations are a big one. In general, problems are hard to debug without careful manual review, so we're only using them when the amount of results generated is small enough to review by hand.   \n",
              "20                                                                                                                                                                                                                                                                      Still developing   \n",
              "22                                                                                                                                                                                                                                        It’s a mix of taming a beast and using alchemy   \n",
              "25                                                                                                                                                                                                                               Hardware requirements and AI accelerators compatibility   \n",
              "26                                                                                                                                                                                                                                                   explainability, expense, embeddings   \n",
              "29                                                                                                                                                                                          Consistency of formatting in freeform LLM text generation. Subtleties in prompt engineering.   \n",
              "30                                                                                                                                                                                                                                                                       Reproducibility   \n",
              "34                                                                                                                                                                                                                                                                               latency   \n",
              "37                                                                                                                                                                                                                                                                      reducing latency   \n",
              "38                                                                                                                                                                                                                                                                 Infrastructure costs.   \n",
              "39                                                                                                                                                                                                                                                                     Analyzing results   \n",
              "44                                                                                                                                                                                                                                                                 Rapidly changing APIs   \n",
              "45                                                                                                                                                                                                        Input length limitation and having to work around it with non-optimal methods.   \n",
              "46                                                                                                                                                                                                                                                          Vectorizing CRUD and ci / CD   \n",
              "48                                                                                                                                                                                                                           Compute performance, fine tuning challenges, explainability   \n",
              "49                                                                                                                                                                                                                                     Prompt engineering; handling very large documents   \n",
              "50                                                                                                                                                                                                                                                      Environment dependency conflicts   \n",
              "52   The main challenges are using the different tools from langchain to optimise for the use case. There are a lot of things to consider. Also, we use ts and angular front end so not having all the tools that are available in python required us to have python as the backend now.   \n",
              "53                                                                                                                                                                                                                                                         deploy models into production   \n",
              "56                                                                                                                                                                                                                                             Work with 3 AI providers at the same time   \n",
              "57                                                                                                                                                                                                                                           Embedding with appropriate data effectively   \n",
              "58                                                                                                                                                                                                                                                       Managing training data labeling   \n",
              "60                                                                                                                                                                                                                                    Explainability, Testing, Getting ground truth data   \n",
              "61                                                                                                                                                                                       Langchain LLM Parse error. Turbo not using multiple tools for some reason, when davinci-003 did   \n",
              "65                                                                                 control of the output for our usecase. Reports need to understand the client's intent and take information from briefing calls and key erquriements. These can help guide the output, but now always.   \n",
              "66                                                                                                                                                                                                                                                                            Cloud Cost   \n",
              "68                                                                                                                                                                          currently used only pre-trained models, biggest challenge was to obtain data for specific domain fine tuning   \n",
              "69                                                                                                                                                                                                                                                               K8's caches... TESTING.   \n",
              "71                                                                                                                                                                                                                                                            Context length limitations   \n",
              "72                                                                                                                                                                                                                                                            Dealing with vector stores   \n",
              "73                                                                                                                                                                                                                                                              Data privacy, big models   \n",
              "74                                                                                                                                                                                                                                                  Memory requirements during inference   \n",
              "79                                                                                                                                                                                                                                                                The need to use Python   \n",
              "81                                                                                                                                                                                                                                                                       Computing power   \n",
              "84                                                                                                                                                        Sparse representations (e.g. tfidf) are easier to work with for our use-cases, because they survive simple aggregation better.   \n",
              "87                                                                                                                                                                                                                                                                                     -   \n",
              "88                                                                                                  We are not yet sure how large enterprises are going to adopt LLMs. Our large enterprise customers are currently not using the Mito AI feature because there is too much uncertainty.   \n",
              "89                                                                                                                                                            It is pretty hard to work with this models as they need lot of resources to run which significantly increase the cost also   \n",
              "90                                                                                                                                               Getting it to write a query was relatively easy. Getting it to write a query that would actually be correct AND execute is much harder.   \n",
              "91                                                                                                                                                                                                                                                The unpredictable nature of responses.   \n",
              "93                                                                                                                                                                                                                                                           Accuracy and hallucinations   \n",
              "95                                                                                                                                                                                                                                               View of accuracy and quality of results   \n",
              "98                                                                                                                                                                                                                                                         supplement/support vs replace   \n",
              "99                                                                                                                                                                                                                                                          Not sure what's the standard   \n",
              "101                                                                                                                                                                                                                                                                          rate limit,   \n",
              "102                                                                                                                                                                                                                                                                           scaling :)   \n",
              "104                                                                                                                                                                                                        Slow. They give bad answers when you set the confidence requirements too high   \n",
              "107                                                                                                                                                                                                                                                                           Compliance   \n",
              "\n",
              "                                                             challenges  \n",
              "2                                            Infrastructure,Workarounds  \n",
              "3                         Accuracy,Outdated Information,Cost,Deployment  \n",
              "4                                                                        \n",
              "5                                                           Reliability  \n",
              "7                                 Size,Training,Response Time,Debugging  \n",
              "9                                                                        \n",
              "11                                                            Debugging  \n",
              "14                                                   Latency,Deployment  \n",
              "19                                             Debugging,Hallucinations  \n",
              "20                                                          Development  \n",
              "22                                               Hardware,Compatibility  \n",
              "25                                    Explainability,Expense,Embeddings  \n",
              "26                                        Formatting,Prompt Engineering  \n",
              "29                                                      Reproducibility  \n",
              "30                                                              Latency  \n",
              "34                                                              Latency  \n",
              "37                                                 Infrastructure Costs  \n",
              "38                                                      Result Analysis  \n",
              "39   Automation,Scaling,Transfer Learning,Tuning,Reinforcement Learning  \n",
              "44                                                          API changes  \n",
              "45                                              Input length limitation  \n",
              "46                                                    CI/CD,Performance  \n",
              "48                               Performance,Explainability,Fine-tuning  \n",
              "49                          Prompt engineering,Handling large documents  \n",
              "50                                     Environment dependency conflicts  \n",
              "52                                        Optimization,Interoperability  \n",
              "53                                                           Deployment  \n",
              "56                                                     Interoperability  \n",
              "57                                                       Data embedding  \n",
              "58                                                        Data labeling  \n",
              "60                                 Explainability,Testing,Data labeling  \n",
              "61                                                          Parse error  \n",
              "65                                                       Output control  \n",
              "66                                                           Cloud cost  \n",
              "68                                                     Data acquisition  \n",
              "69                                                              Testing  \n",
              "71                                            Context length limitation  \n",
              "72                                                        Vector stores  \n",
              "73                                              Data privacy,Model size  \n",
              "74                                              Memory,Inference,Python  \n",
              "79                                                      Computing Power  \n",
              "81                                   Sparse Representations,Aggregation  \n",
              "84                                                                       \n",
              "87                                      Enterprise Adoption,Uncertainty  \n",
              "88                                           Resource Requirements,Cost  \n",
              "89                                     Query Writing,Accuracy,Execution  \n",
              "90                                                     Unpredictability  \n",
              "91                                      Accuracy,Hallucinations,Quality  \n",
              "93                                                     Accuracy,Quality  \n",
              "95                                       Supplement,Support,Replacement  \n",
              "98                                                      Standardization  \n",
              "99                                                        Rate Limiting  \n",
              "101                                                         Scalability  \n",
              "102                                        Slow,Confidence Requirements  \n",
              "104                                                          Compliance  \n",
              "107               Scalability,Immature Stack,Benchmarks,Risk Assessment  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9fa1d4cc-dc63-4526-9e75-6890d5c4fd5e\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>What are some of the main challenges you have encountered thus far when building with LLMs</th>\n",
              "      <th>challenges</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Current infrastructure not designed for such massive models, having to implement workarounds for quick fixes</td>\n",
              "      <td>Infrastructure,Workarounds</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>out of date info, hallucinations, cost, difficulty in deploying on own infrastructure</td>\n",
              "      <td>Accuracy,Outdated Information,Cost,Deployment</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>not yet</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>reliability</td>\n",
              "      <td>Reliability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Size in all regards. Also training time, response time and that multi gpu debugging is weird.</td>\n",
              "      <td>Size,Training,Response Time,Debugging</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>lik</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>Prompt debugging! We need a new form of debugger.</td>\n",
              "      <td>Debugging</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>Latency and deployment time</td>\n",
              "      <td>Latency,Deployment</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>Hallucinations are a big one. In general, problems are hard to debug without careful manual review, so we're only using them when the amount of results generated is small enough to review by hand.</td>\n",
              "      <td>Debugging,Hallucinations</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>Still developing</td>\n",
              "      <td>Development</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>It’s a mix of taming a beast and using alchemy</td>\n",
              "      <td>Hardware,Compatibility</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>Hardware requirements and AI accelerators compatibility</td>\n",
              "      <td>Explainability,Expense,Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>explainability, expense, embeddings</td>\n",
              "      <td>Formatting,Prompt Engineering</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>Consistency of formatting in freeform LLM text generation. Subtleties in prompt engineering.</td>\n",
              "      <td>Reproducibility</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>Reproducibility</td>\n",
              "      <td>Latency</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>latency</td>\n",
              "      <td>Latency</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>reducing latency</td>\n",
              "      <td>Infrastructure Costs</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>Infrastructure costs.</td>\n",
              "      <td>Result Analysis</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>Analyzing results</td>\n",
              "      <td>Automation,Scaling,Transfer Learning,Tuning,Reinforcement Learning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>Rapidly changing APIs</td>\n",
              "      <td>API changes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>Input length limitation and having to work around it with non-optimal methods.</td>\n",
              "      <td>Input length limitation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>Vectorizing CRUD and ci / CD</td>\n",
              "      <td>CI/CD,Performance</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>48</th>\n",
              "      <td>Compute performance, fine tuning challenges, explainability</td>\n",
              "      <td>Performance,Explainability,Fine-tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>Prompt engineering; handling very large documents</td>\n",
              "      <td>Prompt engineering,Handling large documents</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>Environment dependency conflicts</td>\n",
              "      <td>Environment dependency conflicts</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>The main challenges are using the different tools from langchain to optimise for the use case. There are a lot of things to consider. Also, we use ts and angular front end so not having all the tools that are available in python required us to have python as the backend now.</td>\n",
              "      <td>Optimization,Interoperability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53</th>\n",
              "      <td>deploy models into production</td>\n",
              "      <td>Deployment</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>Work with 3 AI providers at the same time</td>\n",
              "      <td>Interoperability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57</th>\n",
              "      <td>Embedding with appropriate data effectively</td>\n",
              "      <td>Data embedding</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>58</th>\n",
              "      <td>Managing training data labeling</td>\n",
              "      <td>Data labeling</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>60</th>\n",
              "      <td>Explainability, Testing, Getting ground truth data</td>\n",
              "      <td>Explainability,Testing,Data labeling</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>Langchain LLM Parse error. Turbo not using multiple tools for some reason, when davinci-003 did</td>\n",
              "      <td>Parse error</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>control of the output for our usecase. Reports need to understand the client's intent and take information from briefing calls and key erquriements. These can help guide the output, but now always.</td>\n",
              "      <td>Output control</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66</th>\n",
              "      <td>Cloud Cost</td>\n",
              "      <td>Cloud cost</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>currently used only pre-trained models, biggest challenge was to obtain data for specific domain fine tuning</td>\n",
              "      <td>Data acquisition</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>69</th>\n",
              "      <td>K8's caches... TESTING.</td>\n",
              "      <td>Testing</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>71</th>\n",
              "      <td>Context length limitations</td>\n",
              "      <td>Context length limitation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>72</th>\n",
              "      <td>Dealing with vector stores</td>\n",
              "      <td>Vector stores</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>73</th>\n",
              "      <td>Data privacy, big models</td>\n",
              "      <td>Data privacy,Model size</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>74</th>\n",
              "      <td>Memory requirements during inference</td>\n",
              "      <td>Memory,Inference,Python</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>79</th>\n",
              "      <td>The need to use Python</td>\n",
              "      <td>Computing Power</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>81</th>\n",
              "      <td>Computing power</td>\n",
              "      <td>Sparse Representations,Aggregation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>84</th>\n",
              "      <td>Sparse representations (e.g. tfidf) are easier to work with for our use-cases, because they survive simple aggregation better.</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87</th>\n",
              "      <td>-</td>\n",
              "      <td>Enterprise Adoption,Uncertainty</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>88</th>\n",
              "      <td>We are not yet sure how large enterprises are going to adopt LLMs. Our large enterprise customers are currently not using the Mito AI feature because there is too much uncertainty.</td>\n",
              "      <td>Resource Requirements,Cost</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>89</th>\n",
              "      <td>It is pretty hard to work with this models as they need lot of resources to run which significantly increase the cost also</td>\n",
              "      <td>Query Writing,Accuracy,Execution</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>90</th>\n",
              "      <td>Getting it to write a query was relatively easy. Getting it to write a query that would actually be correct AND execute is much harder.</td>\n",
              "      <td>Unpredictability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>91</th>\n",
              "      <td>The unpredictable nature of responses.</td>\n",
              "      <td>Accuracy,Hallucinations,Quality</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>93</th>\n",
              "      <td>Accuracy and hallucinations</td>\n",
              "      <td>Accuracy,Quality</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>95</th>\n",
              "      <td>View of accuracy and quality of results</td>\n",
              "      <td>Supplement,Support,Replacement</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>supplement/support vs replace</td>\n",
              "      <td>Standardization</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>99</th>\n",
              "      <td>Not sure what's the standard</td>\n",
              "      <td>Rate Limiting</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>101</th>\n",
              "      <td>rate limit,</td>\n",
              "      <td>Scalability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>102</th>\n",
              "      <td>scaling :)</td>\n",
              "      <td>Slow,Confidence Requirements</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>Slow. They give bad answers when you set the confidence requirements too high</td>\n",
              "      <td>Compliance</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>107</th>\n",
              "      <td>Compliance</td>\n",
              "      <td>Scalability,Immature Stack,Benchmarks,Risk Assessment</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9fa1d4cc-dc63-4526-9e75-6890d5c4fd5e')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-9fa1d4cc-dc63-4526-9e75-6890d5c4fd5e button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-9fa1d4cc-dc63-4526-9e75-6890d5c4fd5e');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q6_question = questions[6]\n",
        "q6_instructions = \"\"\"I received answers from an MLOps survey about Large Language Models (LLMs).\n",
        "Your task is to extract the concerns mentioned in each answer.\n",
        "You should use high-level keywords whenever possible and avoid synonyms and acronyms.\n",
        "If no challenges are mentioned, you should use an empty string as a placeholder.\n",
        "You should output these information in the JSON lines format.\n",
        "You should generate one JSON line per answer.\n",
        "Here's an example of an answer:\n",
        "- The cost of openai API is too high\n",
        "- Safe and ethical usage of these systems\n",
        "- Can't rely on answers 100% of the time, No proper documentation\n",
        "Here's an example of the expected output:\n",
        "{\"concerns\": \"Costs\"}\n",
        "{\"concerns\": \"Safety,Ethics\"}\n",
        "{\"concerns\": \"Reliability,Documentation\"}\n",
        "Answers:\"\"\"\n",
        "q6_inputs = results.iloc[:, 6]\n",
        "print(f'Question 6:', q6_question)\n",
        "q6_reviews = associate(model, q6_instructions, q6_inputs)\n",
        "q6_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "EgA5CrOk9fdQ",
        "outputId": "cfe36f74-d5e5-4504-a090-db8192d958f6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 6: What are your main concerns with using LLMs in production?\n",
            "- Inputs: 110\n",
            "- batch: 0 -> 20\n",
            "- tokens: 585\n",
            "- batch: 1 -> 20\n",
            "- tokens: 707\n",
            "Warning! Got: 19, expected: 20\n",
            "- batch: 2 -> 20\n",
            "- tokens: 614\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                                                                                                                                                                                                                                                                                                                             What are your main concerns with using LLMs in production?  \\\n",
              "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Cost   \n",
              "3                                                                                                                                                                                                                                                                                                                                                                                                                                                  they could serve nonsense or out of date information   \n",
              "4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    cost of openai API   \n",
              "5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    cost of openai API   \n",
              "7                                                                                                                                                                                                                                                                                                                                                                 Whether the cost involved for us justifies the gains we get, considering we mainly make money from ads and not a really good product.   \n",
              "9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    il   \n",
              "10                                                                                                                                                                                                                                                                                                                                                                                                                       Cost. But our issues with productionisation is more general than just for LLMs   \n",
              "11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     They are erratic   \n",
              "14                                                                                                                                                                                                                                                                                                                                                                                                                                               Resources usage along with latency and deployment time   \n",
              "19                                                                                                                                                                                                                                                                                                                                                                                                                    Reliability, cost, injecting domain-specific language and understanding into them   \n",
              "20                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 Cost   \n",
              "22                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Consistency of the output quality   \n",
              "25                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Inference scalability   \n",
              "26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            see above   \n",
              "28                                                                                                                                                                                                                                                                                                                                                                                                                                                         Complacency and atrophy of critical thinking   \n",
              "29                                                                                                                                                                                                                                                                                                                                                                          Detecting truthfulness, hallucination. Output validation. Risk of monopolisation and centralised access to LLM as a service   \n",
              "30                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Hallucination and latency   \n",
              "34                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  latency, factuality   \n",
              "37                                                                                                                                                                                                                                                                                                                                                                                                                                                   not knowing the data used for the initial training   \n",
              "38                                                                                                                                                                                                                                                                                                                                                                                                                                                   Is what I;m building getting obsolete in 3 months,   \n",
              "39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             security   \n",
              "41                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Scaling   \n",
              "44                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Hallucinations misleading customers   \n",
              "45                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       Hallucinations   \n",
              "46                                                                                                                                                                                                                                                                                                                                                                                                                                                                     To get better at semantic search   \n",
              "49                                                                                                                                                                                                                                                                                                                                                                                                          Bad data/completions; runtime errors parsing completions (we use regex to parse each line )   \n",
              "50                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Safe and ethical usage   \n",
              "52                                                                                                                                                                                                                                                                                                                                                                                                                                         It’s how useful/valuable it will actually be for the client.   \n",
              "53                                                                                                                                                                                                                                                                                                                                                                                                                                                                      hight training & inference cost   \n",
              "56                                                                                                                                                                                                                                                                                                                                                                                                                                                                     The wrong respond from the model   \n",
              "57                                                                                                                                                                                                                                                                                                                                                         Being dependent on one company for the API so far. Having models of equivalent performance whose weights were public would be a great relief   \n",
              "58                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       explainability   \n",
              "60                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Trustworthiness   \n",
              "61                                                                                                                                                                                                                                                                                                                                                                                                         Need HIPAA compliance, so trying to get on Azure. Worried about the contract falling through   \n",
              "65                                                                                                                                                                                                                                                                                                                                                                                                                 Thta it will hallucinate something that we won;t pick up in the report editing phase   \n",
              "66                                                                                                                                                                                                                                                                                                                                                                                                                                                                              No proper documentation   \n",
              "68                                                                                                                                                                                                                                                                                                                                                                                 it is very important in my opinion to accurately monitor the kind of usage to prevent frauds or malicious intentions   \n",
              "69   cost, latency, truthfulnes - but also\\n\\nMaybe management in the sense of versioning and resolving / attributing issues?\\n\\nCopyright and intellectual property liability - who knows what it's been trained on and what it might spit out... What happens if it provides a Disney script for an advert? (I know the answer - nothing good).\\n\\nSafety : what if it tells someone to kill themselves? Or spits out a recipe to make smallpox? (I am being kind of silly here, but... not entirely)   \n",
              "70                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       Hallucinations   \n",
              "72                                                                                                                                                                                                                                                                                                                                                                                                                                        Keeping agent on track with system (OpenAI’s ChatGPT) or role   \n",
              "73                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Fake information   \n",
              "74                                                                                                                                                                                                                                                                                                                                                                                                                                            Slow start up times/it’s very different from regular code   \n",
              "76                                                                                                                                                                                                                                                                                                                                                                                                                                                                  costs under control and reliability   \n",
              "79                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       Cost, accuracy   \n",
              "84                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Explainability of embeddings.   \n",
              "87                                                                                                                                                                                                                                                                                                                                                                                                                                                                            bad output, costs etc,etc   \n",
              "89                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Need of resources   \n",
              "90                                                                                                                                                                                                                                                                                                                                                                                 My concerns at this point are primarily focused around getting the LLM to generate an executable query in all cases.   \n",
              "91                                                                                                                                                                                                                                                                                                                                                                                                                                                         That they give out results that are way off.   \n",
              "93                                                                                                                                                                                                                                                                                                                                                                                                                                                               Can't rely on answers 100% of the time   \n",
              "95                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Reliability   \n",
              "98                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            user risk   \n",
              "99                                                                                                                                                                                                                                                                                                                                                                                                                                                                      The output can be unpredictable   \n",
              "101                                                                                                                                                                                                                                                                                                                                                                                                                                                                      privacy, trustfulness, latency   \n",
              "102                                                                                                                                                                                                                                                                                                                                                                                                                                                             great for batch, terrible for real time   \n",
              "104                                                                                                                                                                                                                                                                                                                                                                                                                      Should we be paying for third party providers or train our own. Alpaca anyone?   \n",
              "107                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Compliance, security and legal.   \n",
              "108                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        data privacy   \n",
              "109                                                                                                                                                                                                                                                                                                                                                                                                                                evaluating efficacy and ROI; understanding power needs/CO2 emissions   \n",
              "\n",
              "                                                            concerns  \n",
              "2                                                               Cost  \n",
              "3                                                 Accuracy, Currency  \n",
              "4                                                               Cost  \n",
              "5                                                               Cost  \n",
              "7                                               Cost, Business Value  \n",
              "9                                                                     \n",
              "10                                           Cost, Productionization  \n",
              "11                                                       Reliability  \n",
              "14                               Resource Usage, Latency, Deployment  \n",
              "19                       Reliability, Cost, Domain-specific Language  \n",
              "20                                                              Cost  \n",
              "22                                                    Output Quality  \n",
              "25                                                       Scalability  \n",
              "26                                                                    \n",
              "28                                                 Critical Thinking  \n",
              "29                          Truthfulness, Validation, Monopolization  \n",
              "30                                            Hallucination, Latency  \n",
              "34                                               Latency, Factuality  \n",
              "37                                                 Data Transparency  \n",
              "38                                                      Obsolescence  \n",
              "39                                                          Security  \n",
              "41                                                           Scaling  \n",
              "44                               Hallucinations,Misleading customers  \n",
              "45                                                    Hallucinations  \n",
              "46                                                   Semantic search  \n",
              "49                                           Bad data,Runtime errors  \n",
              "50                                                     Safety,Ethics  \n",
              "52                                                  Value for client  \n",
              "53                                      Training cost,Inference cost  \n",
              "56                                               Incorrect responses  \n",
              "57                                         Dependency on one company  \n",
              "58                                                    Explainability  \n",
              "60                                                   Trustworthiness  \n",
              "61                         HIPAA compliance,Contract falling through  \n",
              "65                                                    Hallucinations  \n",
              "66                                                     Documentation  \n",
              "68                                 Usage monitoring,Fraud prevention  \n",
              "69   Cost,Latency,Truthfulness,Versioning,Issue resolution,Copyright  \n",
              "70             Safety,Hallucinations,Intellectual property liability  \n",
              "72                                       Keeping agent on track,Role  \n",
              "73                                                  Fake information  \n",
              "74                   Slow start up times,Different from regular code  \n",
              "76                                                 Costs,Reliability  \n",
              "79                                                    Costs,Accuracy  \n",
              "84                                      Explainability of embeddings  \n",
              "87                                                  Bad output,Costs  \n",
              "89                                                 Need of resources  \n",
              "90                                       Generating executable query  \n",
              "91                                                Inaccurate results  \n",
              "93                                                Unreliable answers  \n",
              "95                                                       Reliability  \n",
              "98                                                         User risk  \n",
              "99                                              Unpredictable output  \n",
              "101                                     Privacy,Trustfulness,Latency  \n",
              "102                           Great for batch,Terrible for real time  \n",
              "104                           Third party providers,Training our own  \n",
              "107                                        Compliance,Security,Legal  \n",
              "108                                                     Data privacy  \n",
              "109                Evaluating efficacy,ROI,Power needs,CO2 emissions  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-255993cd-05aa-4399-a65f-7858bda0ade2\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>What are your main concerns with using LLMs in production?</th>\n",
              "      <th>concerns</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Cost</td>\n",
              "      <td>Cost</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>they could serve nonsense or out of date information</td>\n",
              "      <td>Accuracy, Currency</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>cost of openai API</td>\n",
              "      <td>Cost</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>cost of openai API</td>\n",
              "      <td>Cost</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Whether the cost involved for us justifies the gains we get, considering we mainly make money from ads and not a really good product.</td>\n",
              "      <td>Cost, Business Value</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>il</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>Cost. But our issues with productionisation is more general than just for LLMs</td>\n",
              "      <td>Cost, Productionization</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>They are erratic</td>\n",
              "      <td>Reliability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>Resources usage along with latency and deployment time</td>\n",
              "      <td>Resource Usage, Latency, Deployment</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>Reliability, cost, injecting domain-specific language and understanding into them</td>\n",
              "      <td>Reliability, Cost, Domain-specific Language</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>Cost</td>\n",
              "      <td>Cost</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>Consistency of the output quality</td>\n",
              "      <td>Output Quality</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>Inference scalability</td>\n",
              "      <td>Scalability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>see above</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>Complacency and atrophy of critical thinking</td>\n",
              "      <td>Critical Thinking</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>Detecting truthfulness, hallucination. Output validation. Risk of monopolisation and centralised access to LLM as a service</td>\n",
              "      <td>Truthfulness, Validation, Monopolization</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>Hallucination and latency</td>\n",
              "      <td>Hallucination, Latency</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>latency, factuality</td>\n",
              "      <td>Latency, Factuality</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>not knowing the data used for the initial training</td>\n",
              "      <td>Data Transparency</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>Is what I;m building getting obsolete in 3 months,</td>\n",
              "      <td>Obsolescence</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>security</td>\n",
              "      <td>Security</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>Scaling</td>\n",
              "      <td>Scaling</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>Hallucinations misleading customers</td>\n",
              "      <td>Hallucinations,Misleading customers</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>Hallucinations</td>\n",
              "      <td>Hallucinations</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>To get better at semantic search</td>\n",
              "      <td>Semantic search</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>Bad data/completions; runtime errors parsing completions (we use regex to parse each line )</td>\n",
              "      <td>Bad data,Runtime errors</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>Safe and ethical usage</td>\n",
              "      <td>Safety,Ethics</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>It’s how useful/valuable it will actually be for the client.</td>\n",
              "      <td>Value for client</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53</th>\n",
              "      <td>hight training &amp; inference cost</td>\n",
              "      <td>Training cost,Inference cost</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>The wrong respond from the model</td>\n",
              "      <td>Incorrect responses</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57</th>\n",
              "      <td>Being dependent on one company for the API so far. Having models of equivalent performance whose weights were public would be a great relief</td>\n",
              "      <td>Dependency on one company</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>58</th>\n",
              "      <td>explainability</td>\n",
              "      <td>Explainability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>60</th>\n",
              "      <td>Trustworthiness</td>\n",
              "      <td>Trustworthiness</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>Need HIPAA compliance, so trying to get on Azure. Worried about the contract falling through</td>\n",
              "      <td>HIPAA compliance,Contract falling through</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>Thta it will hallucinate something that we won;t pick up in the report editing phase</td>\n",
              "      <td>Hallucinations</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66</th>\n",
              "      <td>No proper documentation</td>\n",
              "      <td>Documentation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>it is very important in my opinion to accurately monitor the kind of usage to prevent frauds or malicious intentions</td>\n",
              "      <td>Usage monitoring,Fraud prevention</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>69</th>\n",
              "      <td>cost, latency, truthfulnes - but also\\n\\nMaybe management in the sense of versioning and resolving / attributing issues?\\n\\nCopyright and intellectual property liability - who knows what it's been trained on and what it might spit out... What happens if it provides a Disney script for an advert? (I know the answer - nothing good).\\n\\nSafety : what if it tells someone to kill themselves? Or spits out a recipe to make smallpox? (I am being kind of silly here, but... not entirely)</td>\n",
              "      <td>Cost,Latency,Truthfulness,Versioning,Issue resolution,Copyright</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>70</th>\n",
              "      <td>Hallucinations</td>\n",
              "      <td>Safety,Hallucinations,Intellectual property liability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>72</th>\n",
              "      <td>Keeping agent on track with system (OpenAI’s ChatGPT) or role</td>\n",
              "      <td>Keeping agent on track,Role</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>73</th>\n",
              "      <td>Fake information</td>\n",
              "      <td>Fake information</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>74</th>\n",
              "      <td>Slow start up times/it’s very different from regular code</td>\n",
              "      <td>Slow start up times,Different from regular code</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>76</th>\n",
              "      <td>costs under control and reliability</td>\n",
              "      <td>Costs,Reliability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>79</th>\n",
              "      <td>Cost, accuracy</td>\n",
              "      <td>Costs,Accuracy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>84</th>\n",
              "      <td>Explainability of embeddings.</td>\n",
              "      <td>Explainability of embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87</th>\n",
              "      <td>bad output, costs etc,etc</td>\n",
              "      <td>Bad output,Costs</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>89</th>\n",
              "      <td>Need of resources</td>\n",
              "      <td>Need of resources</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>90</th>\n",
              "      <td>My concerns at this point are primarily focused around getting the LLM to generate an executable query in all cases.</td>\n",
              "      <td>Generating executable query</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>91</th>\n",
              "      <td>That they give out results that are way off.</td>\n",
              "      <td>Inaccurate results</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>93</th>\n",
              "      <td>Can't rely on answers 100% of the time</td>\n",
              "      <td>Unreliable answers</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>95</th>\n",
              "      <td>Reliability</td>\n",
              "      <td>Reliability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>user risk</td>\n",
              "      <td>User risk</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>99</th>\n",
              "      <td>The output can be unpredictable</td>\n",
              "      <td>Unpredictable output</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>101</th>\n",
              "      <td>privacy, trustfulness, latency</td>\n",
              "      <td>Privacy,Trustfulness,Latency</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>102</th>\n",
              "      <td>great for batch, terrible for real time</td>\n",
              "      <td>Great for batch,Terrible for real time</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>Should we be paying for third party providers or train our own. Alpaca anyone?</td>\n",
              "      <td>Third party providers,Training our own</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>107</th>\n",
              "      <td>Compliance, security and legal.</td>\n",
              "      <td>Compliance,Security,Legal</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>108</th>\n",
              "      <td>data privacy</td>\n",
              "      <td>Data privacy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>evaluating efficacy and ROI; understanding power needs/CO2 emissions</td>\n",
              "      <td>Evaluating efficacy,ROI,Power needs,CO2 emissions</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-255993cd-05aa-4399-a65f-7858bda0ade2')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-255993cd-05aa-4399-a65f-7858bda0ade2 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-255993cd-05aa-4399-a65f-7858bda0ade2');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q7_question = questions[7]\n",
        "q7_inputs = results.iloc[:, 7]\n",
        "print(f'Question 7:', q7_question)\n",
        "# nothing to do\n",
        "q7_reviews = q7_inputs.to_frame()\n",
        "q7_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "bXoACwieprvv",
        "outputId": "86c2385a-6e21-439b-f0ed-9cf383567481"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 7: how are you using LLMs?\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "            how are you using LLMs?\n",
              "2    Open source model (GPT-J, etc)\n",
              "3                       Open AI API\n",
              "4                       Open AI API\n",
              "5                    In house model\n",
              "7          Other model provider API\n",
              "9                       Open AI API\n",
              "10                      Open AI API\n",
              "11                      Open AI API\n",
              "12   Open source model (GPT-J, etc)\n",
              "14   Open source model (GPT-J, etc)\n",
              "19                      Open AI API\n",
              "20                      Open AI API\n",
              "22         Other model provider API\n",
              "25   Open source model (GPT-J, etc)\n",
              "26                      Open AI API\n",
              "28   Open source model (GPT-J, etc)\n",
              "29         Other model provider API\n",
              "30                      Open AI API\n",
              "34   Open source model (GPT-J, etc)\n",
              "37                   In house model\n",
              "38                   In house model\n",
              "39                   In house model\n",
              "41                      Open AI API\n",
              "44                      Open AI API\n",
              "45                      Open AI API\n",
              "46                      Open AI API\n",
              "48   Open source model (GPT-J, etc)\n",
              "49                      Open AI API\n",
              "50                      Open AI API\n",
              "52                      Open AI API\n",
              "53   Open source model (GPT-J, etc)\n",
              "56                      Open AI API\n",
              "57                      Open AI API\n",
              "58   Open source model (GPT-J, etc)\n",
              "60   Open source model (GPT-J, etc)\n",
              "61                      Open AI API\n",
              "65                      Open AI API\n",
              "66         Other model provider API\n",
              "68         Other model provider API\n",
              "69   Open source model (GPT-J, etc)\n",
              "70   Open source model (GPT-J, etc)\n",
              "71                      Open AI API\n",
              "72                      Open AI API\n",
              "73   Open source model (GPT-J, etc)\n",
              "74                   In house model\n",
              "76   Open source model (GPT-J, etc)\n",
              "79                      Open AI API\n",
              "81                      Open AI API\n",
              "84   Open source model (GPT-J, etc)\n",
              "87                      Open AI API\n",
              "88                      Open AI API\n",
              "89         Other model provider API\n",
              "90                      Open AI API\n",
              "91         Other model provider API\n",
              "93   Open source model (GPT-J, etc)\n",
              "95                      Open AI API\n",
              "98                   In house model\n",
              "99                      Open AI API\n",
              "101                     Open AI API\n",
              "102  Open source model (GPT-J, etc)\n",
              "104                     Open AI API\n",
              "106                     Open AI API\n",
              "107                     Open AI API\n",
              "108  Open source model (GPT-J, etc)\n",
              "109                  In house model"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b8be95f4-b675-4c3c-9aa7-0ccde15c26a1\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>how are you using LLMs?</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>In house model</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Other model provider API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>Other model provider API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>Other model provider API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>In house model</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>In house model</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>In house model</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>48</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>58</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>60</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66</th>\n",
              "      <td>Other model provider API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>Other model provider API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>69</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>70</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>71</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>72</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>73</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>74</th>\n",
              "      <td>In house model</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>76</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>79</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>81</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>84</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>88</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>89</th>\n",
              "      <td>Other model provider API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>90</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>91</th>\n",
              "      <td>Other model provider API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>93</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>95</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>In house model</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>99</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>101</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>102</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>106</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>107</th>\n",
              "      <td>Open AI API</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>108</th>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>In house model</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b8be95f4-b675-4c3c-9aa7-0ccde15c26a1')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-b8be95f4-b675-4c3c-9aa7-0ccde15c26a1 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-b8be95f4-b675-4c3c-9aa7-0ccde15c26a1');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q8_question = questions[8]\n",
        "q8_instructions = \"\"\"I received answers from an MLOps survey about Large Language Models (LLMs).\n",
        "Your task is to extract the tools mentioned in each answer.\n",
        "You should normalize the tool names whenever possible (e.g., case, separator).\n",
        "If no tools are mentioned, you should use an empty string as a placeholder.\n",
        "You should output these information in the JSON lines format.\n",
        "You should generate one JSON line per answer.\n",
        "Here's an example of an answer:\n",
        "- Pinecone, weaviate, langchain, hugging face\n",
        "- PyTorch huggingaface\n",
        "- Langchain/ts/csharpe/angular/k8s\n",
        "Here's an example of the expected output:\n",
        "{\"tools\": \"Pinecone, Weaviate, LangChain, Hugging Face\"}\n",
        "{\"tools\": \"PyTorch, Hugging Face\"}\n",
        "{\"tools\": \"LangChain, TypeScript, C#, Angular, Kubernetes\"}\n",
        "Answers:\"\"\"\n",
        "q8_inputs = results.iloc[:, 8]\n",
        "print(f'Question 8:', q8_question)\n",
        "q8_reviews = associate(model, q8_instructions, q8_inputs)\n",
        "q8_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "0--KYxYC_Ily",
        "outputId": "acd69056-5b86-4907-894a-4722323777e2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 8: What tools are you using with LLMs?\n",
            "- Inputs: 110\n",
            "- batch: 0 -> 20\n",
            "- tokens: 563\n",
            "- batch: 1 -> 20\n",
            "- tokens: 573\n",
            "- batch: 2 -> 9\n",
            "- tokens: 338\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                      What tools are you using with LLMs?  \\\n",
              "2                                                                                                                  Seldon   \n",
              "4                                                                  Looking at Pinecone, weaviate, langchain, hugging face   \n",
              "7                                                                         Hugginface, sagemaker, openapi api, tensorflow.   \n",
              "9                                                                                                                     loi   \n",
              "10                                                                                                              streamlit   \n",
              "22                                                                                                                    ./.   \n",
              "25                                                 PyTorch, AWS SageMaker, OpenAI's API, Midjourney, CLIP/BLIP/etc models   \n",
              "26                                                                                                               metaflow   \n",
              "28                                                                                        Microsoft Azure OpenAI services   \n",
              "29                                                                                                                 Python   \n",
              "30                                                                                                              HoneyHive   \n",
              "37                                                                                                        TensorFlow, TFX   \n",
              "38                                                                             In house Bert (from Huggingface) + Open AI   \n",
              "39                                                                                                    in house built tool   \n",
              "41                                                                                                   Lang chain, k8s, gcp   \n",
              "44                                                                                                            PromptLayer   \n",
              "45                                                                                         Langchain, Huggingface, Polars   \n",
              "46                                                                                     Langchain, pinecone, slack and aws   \n",
              "49                                                                                                              LangChain   \n",
              "50                                                                                                     Use case dependent   \n",
              "52                                                                                          Langchain/ts/csharpe/angular.   \n",
              "53                                                                                  transformers, triton inference server   \n",
              "56                                                                                            Code GPT https://codegpt.co   \n",
              "57                                                                                                   Gpt index, langchain   \n",
              "58                                                                                                          AWS Sagemaker   \n",
              "60                                                                          Milvus DB, FastAPI, Hugginface, Pytorch, etc.   \n",
              "61                                    Langchain. All my agent’s tools are custom-built. I hit our internal APIs with them   \n",
              "65                                                                                                       Torch and Pandas   \n",
              "68                                                                                                               Haystack   \n",
              "69   not sure what this means - we use k8's and cloud functions to run them, also interface to case management solutions.   \n",
              "70                                                                                                                UpTrain   \n",
              "71                                                                            Langchain, Llama Index, Deep lake, Weaviate   \n",
              "72                                                                                                              LangChain   \n",
              "73                                                                                                   PyTorch huggingaface   \n",
              "74                                                                                                           Hugging Face   \n",
              "79                                                                                                                 Zapier   \n",
              "87                                                                                                                      -   \n",
              "88                                                                                              Just the Mito spreadsheet   \n",
              "89                                                                                        Docker kubernetes aws resources   \n",
              "90                                                                                                  Apache Drill, Python.   \n",
              "91                                                                                                      Databricks, Azure   \n",
              "93                                                                                          Jupyter, kubernetes, haystack   \n",
              "95                                                                                                            Azure stack   \n",
              "98                                                                                                                     hf   \n",
              "101                                                                                                      vector databases   \n",
              "102                                                                                                        azure ml suite   \n",
              "104                                                                                             Nvidia NeMO, Azure OpenAI   \n",
              "108                                                                                                              kubeflow   \n",
              "109                                                                                              PyTorch + in-house tools   \n",
              "\n",
              "                                                  tools  \n",
              "2                                                Seldon  \n",
              "4           Pinecone, Weaviate, LangChain, Hugging Face  \n",
              "7          Hugging Face, SageMaker, OpenAPI, TensorFlow  \n",
              "9                                                   loi  \n",
              "10                                            Streamlit  \n",
              "22                                                       \n",
              "25   PyTorch, SageMaker, OpenAI, Midjourney, CLIP, BLIP  \n",
              "26                                             Metaflow  \n",
              "28                              Microsoft Azure, OpenAI  \n",
              "29                                               Python  \n",
              "30                                            HoneyHive  \n",
              "37                                      TensorFlow, TFX  \n",
              "38                                In house Bert, OpenAI  \n",
              "39                                  In house built tool  \n",
              "41                           LangChain, Kubernetes, GCP  \n",
              "44                                          PromptLayer  \n",
              "45                       LangChain, HuggingFace, Polars  \n",
              "46                      LangChain, Pinecone, Slack, AWS  \n",
              "49                                            LangChain  \n",
              "50                                                       \n",
              "52                   LangChain, TypeScript, C#, Angular  \n",
              "53                Transformers, Triton Inference Server  \n",
              "56                                             Code GPT  \n",
              "57                                 GPT Index, LangChain  \n",
              "58                                        AWS Sagemaker  \n",
              "60            Milvus DB, FastAPI, Hugging Face, PyTorch  \n",
              "61                                            LangChain  \n",
              "65                                        Torch, Pandas  \n",
              "68                                             Haystack  \n",
              "69                                                       \n",
              "70                                              UpTrain  \n",
              "71          LangChain, Llama Index, Deep Lake, Weaviate  \n",
              "72                                            LangChain  \n",
              "73                                PyTorch, Hugging Face  \n",
              "74                                         Hugging Face  \n",
              "79                                               Zapier  \n",
              "87                                                       \n",
              "88                                     Mito Spreadsheet  \n",
              "89                    Docker, Kubernetes, AWS Resources  \n",
              "90                                 Apache Drill, Python  \n",
              "91                                    Databricks, Azure  \n",
              "93                        Jupyter, Kubernetes, Haystack  \n",
              "95                                          Azure Stack  \n",
              "98                                         Hugging Face  \n",
              "101                                    Vector Databases  \n",
              "102                                      Azure ML Suite  \n",
              "104                           Nvidia NeMO, Azure OpenAI  \n",
              "108                                            Kubeflow  \n",
              "109                             PyTorch, In-house Tools  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-2d8100c5-2081-45ec-b991-ddb19ed4dfde\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>What tools are you using with LLMs?</th>\n",
              "      <th>tools</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Seldon</td>\n",
              "      <td>Seldon</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Looking at Pinecone, weaviate, langchain, hugging face</td>\n",
              "      <td>Pinecone, Weaviate, LangChain, Hugging Face</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Hugginface, sagemaker, openapi api, tensorflow.</td>\n",
              "      <td>Hugging Face, SageMaker, OpenAPI, TensorFlow</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>loi</td>\n",
              "      <td>loi</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>streamlit</td>\n",
              "      <td>Streamlit</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>./.</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>PyTorch, AWS SageMaker, OpenAI's API, Midjourney, CLIP/BLIP/etc models</td>\n",
              "      <td>PyTorch, SageMaker, OpenAI, Midjourney, CLIP, BLIP</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>metaflow</td>\n",
              "      <td>Metaflow</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>Microsoft Azure OpenAI services</td>\n",
              "      <td>Microsoft Azure, OpenAI</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>Python</td>\n",
              "      <td>Python</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>HoneyHive</td>\n",
              "      <td>HoneyHive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>TensorFlow, TFX</td>\n",
              "      <td>TensorFlow, TFX</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>In house Bert (from Huggingface) + Open AI</td>\n",
              "      <td>In house Bert, OpenAI</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>in house built tool</td>\n",
              "      <td>In house built tool</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>Lang chain, k8s, gcp</td>\n",
              "      <td>LangChain, Kubernetes, GCP</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>PromptLayer</td>\n",
              "      <td>PromptLayer</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>Langchain, Huggingface, Polars</td>\n",
              "      <td>LangChain, HuggingFace, Polars</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>Langchain, pinecone, slack and aws</td>\n",
              "      <td>LangChain, Pinecone, Slack, AWS</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>LangChain</td>\n",
              "      <td>LangChain</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>Use case dependent</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>Langchain/ts/csharpe/angular.</td>\n",
              "      <td>LangChain, TypeScript, C#, Angular</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53</th>\n",
              "      <td>transformers, triton inference server</td>\n",
              "      <td>Transformers, Triton Inference Server</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>Code GPT https://codegpt.co</td>\n",
              "      <td>Code GPT</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57</th>\n",
              "      <td>Gpt index, langchain</td>\n",
              "      <td>GPT Index, LangChain</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>58</th>\n",
              "      <td>AWS Sagemaker</td>\n",
              "      <td>AWS Sagemaker</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>60</th>\n",
              "      <td>Milvus DB, FastAPI, Hugginface, Pytorch, etc.</td>\n",
              "      <td>Milvus DB, FastAPI, Hugging Face, PyTorch</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>Langchain. All my agent’s tools are custom-built. I hit our internal APIs with them</td>\n",
              "      <td>LangChain</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>Torch and Pandas</td>\n",
              "      <td>Torch, Pandas</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>Haystack</td>\n",
              "      <td>Haystack</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>69</th>\n",
              "      <td>not sure what this means - we use k8's and cloud functions to run them, also interface to case management solutions.</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>70</th>\n",
              "      <td>UpTrain</td>\n",
              "      <td>UpTrain</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>71</th>\n",
              "      <td>Langchain, Llama Index, Deep lake, Weaviate</td>\n",
              "      <td>LangChain, Llama Index, Deep Lake, Weaviate</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>72</th>\n",
              "      <td>LangChain</td>\n",
              "      <td>LangChain</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>73</th>\n",
              "      <td>PyTorch huggingaface</td>\n",
              "      <td>PyTorch, Hugging Face</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>74</th>\n",
              "      <td>Hugging Face</td>\n",
              "      <td>Hugging Face</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>79</th>\n",
              "      <td>Zapier</td>\n",
              "      <td>Zapier</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87</th>\n",
              "      <td>-</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>88</th>\n",
              "      <td>Just the Mito spreadsheet</td>\n",
              "      <td>Mito Spreadsheet</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>89</th>\n",
              "      <td>Docker kubernetes aws resources</td>\n",
              "      <td>Docker, Kubernetes, AWS Resources</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>90</th>\n",
              "      <td>Apache Drill, Python.</td>\n",
              "      <td>Apache Drill, Python</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>91</th>\n",
              "      <td>Databricks, Azure</td>\n",
              "      <td>Databricks, Azure</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>93</th>\n",
              "      <td>Jupyter, kubernetes, haystack</td>\n",
              "      <td>Jupyter, Kubernetes, Haystack</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>95</th>\n",
              "      <td>Azure stack</td>\n",
              "      <td>Azure Stack</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>hf</td>\n",
              "      <td>Hugging Face</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>101</th>\n",
              "      <td>vector databases</td>\n",
              "      <td>Vector Databases</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>102</th>\n",
              "      <td>azure ml suite</td>\n",
              "      <td>Azure ML Suite</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>Nvidia NeMO, Azure OpenAI</td>\n",
              "      <td>Nvidia NeMO, Azure OpenAI</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>108</th>\n",
              "      <td>kubeflow</td>\n",
              "      <td>Kubeflow</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>PyTorch + in-house tools</td>\n",
              "      <td>PyTorch, In-house Tools</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2d8100c5-2081-45ec-b991-ddb19ed4dfde')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-2d8100c5-2081-45ec-b991-ddb19ed4dfde button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-2d8100c5-2081-45ec-b991-ddb19ed4dfde');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q9_question = questions[9]\n",
        "q9_instructions = \"\"\"I received answers from an MLOps survey about Large Language Models (LLMs).\n",
        "Your task is to extract topics mentioned in each answer.\n",
        "You should normalize the topics whenever possible (e.g., case, separator).\n",
        "If no topics are mentioned, you should use an empty string as a placeholder.\n",
        "You should output these information in the JSON lines format.\n",
        "You should generate one JSON line per answer.\n",
        "Here's an example of an answer:\n",
        "- Fine tunning LLMS\n",
        "- Inferenece for LLMs\n",
        "- Working with OSS LLMs\n",
        "- Finetuning and embeddings\n",
        "Here's an example of the expected output:\n",
        "{\"topics\": \"Fine Tuning\"}\n",
        "{\"topics\": \"Inference\"}\n",
        "{\"topics\": \"Open Source\"}\n",
        "{\"topics\": \"Fine-Tuning, Embeddings\"}\n",
        "Answers:\"\"\"\n",
        "q9_inputs = results.iloc[:, 9]\n",
        "print(f'Question 9:', q9_question)\n",
        "q9_reviews = associate(model, q9_instructions, q9_inputs)\n",
        "q9_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "Lz_cgWUhAxUl",
        "outputId": "1b052a0a-41a4-479d-aa23-6d65bb3b187d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 9: What areas are you most interested in learning more about?\n",
            "- Inputs: 110\n",
            "- batch: 0 -> 20\n",
            "- tokens: 463\n",
            "- batch: 1 -> 20\n",
            "- tokens: 482\n",
            "- batch: 2 -> 20\n",
            "- tokens: 481\n",
            "- batch: 3 -> 7\n",
            "- tokens: 274\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                            What areas are you most interested in learning more about?  \\\n",
              "2                                                                  Inferenece for LLMs   \n",
              "3                                                                Working with OSS LLMs   \n",
              "4                                                                           Embeddings   \n",
              "5                                                                Working with OSS LLMs   \n",
              "7                                                                Working with OSS LLMs   \n",
              "9                                                                Working with OSS LLMs   \n",
              "10                                                                 Inferenece for LLMs   \n",
              "11                                                               Working with OSS LLMs   \n",
              "12                                                                          Embeddings   \n",
              "14                                                                 Inferenece for LLMs   \n",
              "15                                                                 Inferenece for LLMs   \n",
              "19                                                                          Embeddings   \n",
              "20                                                                   Fine tunning LLMS   \n",
              "22                                                               Working with OSS LLMs   \n",
              "25                                                                 Inferenece for LLMs   \n",
              "26                                                                          Embeddings   \n",
              "28                                                                   Fine tunning LLMS   \n",
              "29                                                               Working with OSS LLMs   \n",
              "30                                                                   Fine tunning LLMS   \n",
              "31                                                               Working with OSS LLMs   \n",
              "34                                                                 Inferenece for LLMs   \n",
              "37                                                                 Inferenece for LLMs   \n",
              "38                                                                   Fine tunning LLMS   \n",
              "39                                                                   Fine tunning LLMS   \n",
              "41                                                      Learning from new/private data   \n",
              "44                                                                          Embeddings   \n",
              "45                                                                   Fine tunning LLMS   \n",
              "46                                                                          Embeddings   \n",
              "48                                                                 Inferenece for LLMs   \n",
              "49                                                                   Fine tunning LLMS   \n",
              "50                                                                   Fine tunning LLMS   \n",
              "52                                                                   Fine tunning LLMS   \n",
              "53                                                               Working with OSS LLMs   \n",
              "56                                                                          Embeddings   \n",
              "57                                                                          Embeddings   \n",
              "58                                                                   Fine tunning LLMS   \n",
              "60                                                               Working with OSS LLMs   \n",
              "61   I want deep knowledge on how to make the ReAct pattern work in various edge cases   \n",
              "65                                                 Prompt Engineering (Riley Goodside)   \n",
              "66                                                                   Fine tunning LLMS   \n",
              "68                                                                   Fine tunning LLMS   \n",
              "69                          calibration of llm output / calbration and testing of llms   \n",
              "70                                                                          Embeddings   \n",
              "71                                                                   Fine tunning LLMS   \n",
              "72                                                                          Embeddings   \n",
              "73                                                                 Inferenece for LLMs   \n",
              "74                                                                 Inferenece for LLMs   \n",
              "76                                                                 Inferenece for LLMs   \n",
              "79                                                           Finetuning and embeddings   \n",
              "81                                                                   Fine tunning LLMS   \n",
              "84                                                                          Embeddings   \n",
              "87                                                                   Fine tunning LLMS   \n",
              "88                                  Figuring out how to deploy LLMs for enterprise use   \n",
              "89                                                                 Inferenece for LLMs   \n",
              "90                                                                   Fine tunning LLMS   \n",
              "91                                                                          Embeddings   \n",
              "93                                                                 Inferenece for LLMs   \n",
              "95                                                                          Embeddings   \n",
              "98                                                                   Fine tunning LLMS   \n",
              "99                                                                 Inferenece for LLMs   \n",
              "101                                                                Inferenece for LLMs   \n",
              "102                                                                         Embeddings   \n",
              "104                                                                         Embeddings   \n",
              "106                                                                Inferenece for LLMs   \n",
              "107                                                                  Fine tunning LLMS   \n",
              "108                                                                  Fine tunning LLMS   \n",
              "109                                                                Inferenece for LLMs   \n",
              "\n",
              "                             topics  \n",
              "2                         Inference  \n",
              "3                       Open Source  \n",
              "4                        Embeddings  \n",
              "5                       Open Source  \n",
              "7                       Open Source  \n",
              "9                       Open Source  \n",
              "10                        Inference  \n",
              "11                      Open Source  \n",
              "12                       Embeddings  \n",
              "14                        Inference  \n",
              "15                        Inference  \n",
              "19                       Embeddings  \n",
              "20                      Fine-Tuning  \n",
              "22                      Open Source  \n",
              "25                        Inference  \n",
              "26                       Embeddings  \n",
              "28                      Fine-Tuning  \n",
              "29                      Open Source  \n",
              "30                      Fine-Tuning  \n",
              "31                      Open Source  \n",
              "34                        Inference  \n",
              "37                        Inference  \n",
              "38                      Fine-Tuning  \n",
              "39                      Fine-Tuning  \n",
              "41   Learning from New/Private Data  \n",
              "44                       Embeddings  \n",
              "45          Fine-Tuning, Embeddings  \n",
              "46                       Embeddings  \n",
              "48                        Inference  \n",
              "49                      Fine-Tuning  \n",
              "50                      Fine-Tuning  \n",
              "52                      Fine-Tuning  \n",
              "53                      Open Source  \n",
              "56                       Embeddings  \n",
              "57                       Embeddings  \n",
              "58                      Fine-Tuning  \n",
              "60                      Open Source  \n",
              "61        ReAct Pattern, Edge Cases  \n",
              "65               Prompt Engineering  \n",
              "66                      Fine-Tuning  \n",
              "68                      Fine-Tuning  \n",
              "69             Calibration, Testing  \n",
              "70                       Embeddings  \n",
              "71          Fine-Tuning, Embeddings  \n",
              "72                       Embeddings  \n",
              "73                        Inference  \n",
              "74                        Inference  \n",
              "76                        Inference  \n",
              "79          Fine-Tuning, Embeddings  \n",
              "81                      Fine-Tuning  \n",
              "84                       Embeddings  \n",
              "87                      Fine-Tuning  \n",
              "88                       Deployment  \n",
              "89                        Inference  \n",
              "90                      Fine-Tuning  \n",
              "91                       Embeddings  \n",
              "93                        Inference  \n",
              "95                       Embeddings  \n",
              "98                      Fine-Tuning  \n",
              "99                        Inference  \n",
              "101                       Inference  \n",
              "102                      Embeddings  \n",
              "104                      Embeddings  \n",
              "106                       Inference  \n",
              "107                     Fine-Tuning  \n",
              "108                     Fine-Tuning  \n",
              "109                       Inference  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-6e3dcd0f-d8fa-4362-828d-3602b95105c6\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>What areas are you most interested in learning more about?</th>\n",
              "      <th>topics</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>Open Source</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>Open Source</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>Open Source</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>Open Source</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>Open Source</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>Open Source</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>Open Source</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>Open Source</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>Learning from new/private data</td>\n",
              "      <td>Learning from New/Private Data</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning, Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>48</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53</th>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>Open Source</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57</th>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>58</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>60</th>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>Open Source</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>I want deep knowledge on how to make the ReAct pattern work in various edge cases</td>\n",
              "      <td>ReAct Pattern, Edge Cases</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>Prompt Engineering (Riley Goodside)</td>\n",
              "      <td>Prompt Engineering</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>69</th>\n",
              "      <td>calibration of llm output / calbration and testing of llms</td>\n",
              "      <td>Calibration, Testing</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>70</th>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>71</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning, Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>72</th>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>73</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>74</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>76</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>79</th>\n",
              "      <td>Finetuning and embeddings</td>\n",
              "      <td>Fine-Tuning, Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>81</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>84</th>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>88</th>\n",
              "      <td>Figuring out how to deploy LLMs for enterprise use</td>\n",
              "      <td>Deployment</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>89</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>90</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>91</th>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>93</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>95</th>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>99</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>101</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>102</th>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>106</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>107</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>108</th>\n",
              "      <td>Fine tunning LLMS</td>\n",
              "      <td>Fine-Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6e3dcd0f-d8fa-4362-828d-3602b95105c6')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-6e3dcd0f-d8fa-4362-828d-3602b95105c6 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-6e3dcd0f-d8fa-4362-828d-3602b95105c6');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q10_question = questions[10]\n",
        "q10_instructions = \"\"\"I received answers from an MLOps survey about Large Language Models (LLMs).\n",
        "Your task is to summarize the approaches to deal with reliability mentioned in each answer with keywords.\n",
        "You should normalize the topics whenever possible (e.g., case, separator) and avoid synonyms and acronyms.\n",
        "If no approaches are mentioned, you should use an empty string as a placeholder.\n",
        "You should output these information in the JSON lines format.\n",
        "You should generate one JSON line per answer.\n",
        "Here's an example of an answer:\n",
        "- Not a concern. In-house\n",
        "- Feedback button or manual review depending on use case\n",
        "- We try to make the output as transparent as we can (using explainability techniques)\n",
        "- We have come up with a large list of proxy metrics that we found correlate with performance on our tasks - so we track those\n",
        "Here's an example of the expected output:\n",
        "{\"approaches\": \"Nothing\"}\n",
        "{\"approaches\": \"User Feedbacks, Manual Review\"}\n",
        "{\"approaches\": \"Model Explainability\"}\n",
        "{\"approaches\": \"Model Evaluation\"}\n",
        "Answers:\"\"\"\n",
        "q10_inputs = results.iloc[:, 10]\n",
        "print(f'Question 10:', q10_question)\n",
        "q10_reviews = associate(model, q10_instructions, q10_inputs)\n",
        "q10_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "LJXN7KH1DDgg",
        "outputId": "18c9e593-e4e8-4904-df3e-135e514457a0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 10: How do you deal with reliability of output?\n",
            "- Inputs: 110\n",
            "- batch: 0 -> 20\n",
            "- tokens: 648\n",
            "- batch: 1 -> 20\n",
            "- tokens: 667\n",
            "- batch: 2 -> 5\n",
            "- tokens: 301\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                                                     How do you deal with reliability of output?  \\\n",
              "3                                                                                                                                            we provide links to the context from which answers have been drawn.   \n",
              "4                                                                                                                                                                                                temperature = 0   \n",
              "7                                                                                                                   Ill refer to the picture of the xkcd guy \"just stir the model until the numbers look right\".   \n",
              "9                                                                                                                                                                                                             lo   \n",
              "10                                                                                                                                                All internal tools, reliability is not a major concern for now   \n",
              "11                                                                                                                                                                                       We have control systems   \n",
              "14                                                                                                                                                                                      Assessment at evaluation   \n",
              "19                                                                                                                       Automated retries for some simple & obvious failures, manual review for everything else   \n",
              "22                                                                                                                                                                                                          HITL   \n",
              "28                                                                                                                                                             Educate others on the limitations and proper use.   \n",
              "30                                                                                                                                        Multi-responses shown to user side by side, ghost writer style dropout   \n",
              "37                                                                                                                                                                                                           yes   \n",
              "38                                                                                                                               I am stuck with this right now - can I integrate these into the UI for AI Hero.   \n",
              "39                                                                                                                                                                                           Very manual so far.   \n",
              "41                                                                                                                                                                                       Not a concern. In-house   \n",
              "44                                                                                                Pre/post-processing, constitutional / self-auditing, customer feedback thumbs up/down, custom embedding matrix   \n",
              "45                                                                                                                                                             We implement checking with less stochastic tools.   \n",
              "49                                                                                                                                                                                We don’t do anything right now   \n",
              "50                                                                                                                                                          AI is replaced by HI(Human) for final check, always.   \n",
              "52                                                                                                                                                                                 We havent gotten to that yet.   \n",
              "56                                                                                                                                                                     Now it is just responds tickets in github   \n",
              "57                                                                                                                                      Prompt engineering, allow user to be specific, and of course re-generate   \n",
              "58                                                                                                                                                                     Test each new model candidate on test set   \n",
              "60                                         We try to make the output as transparent as we can (using explainability techniques) and not relying upon LLM for the end decision (these are used as a support tool)   \n",
              "61                                                                                       Will have friendly internal users using it first in the course of serving customers, and giving us per-message feedback   \n",
              "65                                                                                                                                                                                  Have editors comb through it   \n",
              "68   it depends from specific context and specific use cases, in my experience I rely on validation set and a validation tool that estimates accuracy of answers based on answers manually scrutinized by humans   \n",
              "69                                                                                                                                    badly. We do lots of testing but I constantly feel we are on shaky ground.   \n",
              "70                                                                                                                                                                               We are building a tool for that   \n",
              "71                                                                                                                                                                                                Error handling   \n",
              "73                                                                                                                                                                                      Generate multiple output   \n",
              "74                                                                                                                                                                     We’re doing classification not generation   \n",
              "79                                                                                                                                                                                             extensive testing   \n",
              "84                                                                                 We have come up with a large list of proxy metrics that we found correlate with performance on our tasks - so we track those.   \n",
              "87                                                                                                                                                                                                             -   \n",
              "88                                                                              We give users a spreadsheet interface to verify the output of the LLM. Spreadsheets are great for understanding changes to data!   \n",
              "91                                                                                                                                                                                                   Not well :D   \n",
              "93                                                                                                                                                                                       Working on that problem   \n",
              "95                                                                                                                                                                                              no framework yet   \n",
              "98                                                                                                                                                                                                 cross fingers   \n",
              "99                                                                                                                                                                                              Human evaluation   \n",
              "101                                                                                                                                                                                                         eval   \n",
              "102                                                                                                                                                                   human verification of sample of inferences   \n",
              "104                                                                                                                                                       Feedback button or manual review depending on use case   \n",
              "109                                                                                                                                                                                                user feedback   \n",
              "\n",
              "                                                                         approaches  \n",
              "3                                                                           Nothing  \n",
              "4                                                               Temperature Control  \n",
              "7                                                                     Manual Tuning  \n",
              "9                                                                           Nothing  \n",
              "10                                                         Internal Control Systems  \n",
              "11                                                                  Control Systems  \n",
              "14                                                                 Model Evaluation  \n",
              "19                                                 Automated Retries, Manual Review  \n",
              "22                                                         Human-in-the-Loop (HITL)  \n",
              "28                                          Education on Limitations and Proper Use  \n",
              "30                                                         Multi-Responses, Dropout  \n",
              "37                                                                          Nothing  \n",
              "38                                                                   UI Integration  \n",
              "39                                                                    Manual Review  \n",
              "41                                                                          Nothing  \n",
              "44   Pre/Post-Processing, Self-Auditing, Customer Feedback, Custom Embedding Matrix  \n",
              "45                                                        Stochastic Tools Checking  \n",
              "49                                                                          Nothing  \n",
              "50                                                         Human-in-the-Loop (HITL)  \n",
              "52                                                                          Nothing  \n",
              "56                                                                   GitHub Tickets  \n",
              "57                                  Prompt Engineering, User Feedback, Regeneration  \n",
              "58                                                                 Model Evaluation  \n",
              "60                                      Model Explainability, Human Decision Making  \n",
              "61                                                                    User Feedback  \n",
              "65                                                                    Manual Review  \n",
              "68                                  Validation Set, Validation Tool, Human Scrutiny  \n",
              "69                                                                Extensive Testing  \n",
              "70                                                                 Tool Development  \n",
              "71                                                                   Error Handling  \n",
              "73                                                       Multiple Output Generation  \n",
              "74                                 Model Evaluation, Task-Specific Metrics Tracking  \n",
              "79                                                                                   \n",
              "84                                         User Verification, Spreadsheet Interface  \n",
              "87                                                                                   \n",
              "88                                                           Working on the Problem  \n",
              "91                                                                                   \n",
              "93                                                                                   \n",
              "95                                                                                   \n",
              "98                                                                             Hope  \n",
              "99                                                                 Human Evaluation  \n",
              "101                                                                Model Evaluation  \n",
              "102                                                              Human Verification  \n",
              "104                                                   User Feedbacks, Manual Review  \n",
              "109                                                                  User Feedbacks  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-8e8a9920-c516-4674-a414-3da34441b424\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>How do you deal with reliability of output?</th>\n",
              "      <th>approaches</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>we provide links to the context from which answers have been drawn.</td>\n",
              "      <td>Nothing</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>temperature = 0</td>\n",
              "      <td>Temperature Control</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Ill refer to the picture of the xkcd guy \"just stir the model until the numbers look right\".</td>\n",
              "      <td>Manual Tuning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>lo</td>\n",
              "      <td>Nothing</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>All internal tools, reliability is not a major concern for now</td>\n",
              "      <td>Internal Control Systems</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>We have control systems</td>\n",
              "      <td>Control Systems</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>Assessment at evaluation</td>\n",
              "      <td>Model Evaluation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>Automated retries for some simple &amp; obvious failures, manual review for everything else</td>\n",
              "      <td>Automated Retries, Manual Review</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>HITL</td>\n",
              "      <td>Human-in-the-Loop (HITL)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>Educate others on the limitations and proper use.</td>\n",
              "      <td>Education on Limitations and Proper Use</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>Multi-responses shown to user side by side, ghost writer style dropout</td>\n",
              "      <td>Multi-Responses, Dropout</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>yes</td>\n",
              "      <td>Nothing</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>I am stuck with this right now - can I integrate these into the UI for AI Hero.</td>\n",
              "      <td>UI Integration</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>Very manual so far.</td>\n",
              "      <td>Manual Review</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>Not a concern. In-house</td>\n",
              "      <td>Nothing</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>Pre/post-processing, constitutional / self-auditing, customer feedback thumbs up/down, custom embedding matrix</td>\n",
              "      <td>Pre/Post-Processing, Self-Auditing, Customer Feedback, Custom Embedding Matrix</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>We implement checking with less stochastic tools.</td>\n",
              "      <td>Stochastic Tools Checking</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>We don’t do anything right now</td>\n",
              "      <td>Nothing</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>AI is replaced by HI(Human) for final check, always.</td>\n",
              "      <td>Human-in-the-Loop (HITL)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>We havent gotten to that yet.</td>\n",
              "      <td>Nothing</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>Now it is just responds tickets in github</td>\n",
              "      <td>GitHub Tickets</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57</th>\n",
              "      <td>Prompt engineering, allow user to be specific, and of course re-generate</td>\n",
              "      <td>Prompt Engineering, User Feedback, Regeneration</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>58</th>\n",
              "      <td>Test each new model candidate on test set</td>\n",
              "      <td>Model Evaluation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>60</th>\n",
              "      <td>We try to make the output as transparent as we can (using explainability techniques) and not relying upon LLM for the end decision (these are used as a support tool)</td>\n",
              "      <td>Model Explainability, Human Decision Making</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>Will have friendly internal users using it first in the course of serving customers, and giving us per-message feedback</td>\n",
              "      <td>User Feedback</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>Have editors comb through it</td>\n",
              "      <td>Manual Review</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>it depends from specific context and specific use cases, in my experience I rely on validation set and a validation tool that estimates accuracy of answers based on answers manually scrutinized by humans</td>\n",
              "      <td>Validation Set, Validation Tool, Human Scrutiny</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>69</th>\n",
              "      <td>badly. We do lots of testing but I constantly feel we are on shaky ground.</td>\n",
              "      <td>Extensive Testing</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>70</th>\n",
              "      <td>We are building a tool for that</td>\n",
              "      <td>Tool Development</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>71</th>\n",
              "      <td>Error handling</td>\n",
              "      <td>Error Handling</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>73</th>\n",
              "      <td>Generate multiple output</td>\n",
              "      <td>Multiple Output Generation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>74</th>\n",
              "      <td>We’re doing classification not generation</td>\n",
              "      <td>Model Evaluation, Task-Specific Metrics Tracking</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>79</th>\n",
              "      <td>extensive testing</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>84</th>\n",
              "      <td>We have come up with a large list of proxy metrics that we found correlate with performance on our tasks - so we track those.</td>\n",
              "      <td>User Verification, Spreadsheet Interface</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87</th>\n",
              "      <td>-</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>88</th>\n",
              "      <td>We give users a spreadsheet interface to verify the output of the LLM. Spreadsheets are great for understanding changes to data!</td>\n",
              "      <td>Working on the Problem</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>91</th>\n",
              "      <td>Not well :D</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>93</th>\n",
              "      <td>Working on that problem</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>95</th>\n",
              "      <td>no framework yet</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>cross fingers</td>\n",
              "      <td>Hope</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>99</th>\n",
              "      <td>Human evaluation</td>\n",
              "      <td>Human Evaluation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>101</th>\n",
              "      <td>eval</td>\n",
              "      <td>Model Evaluation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>102</th>\n",
              "      <td>human verification of sample of inferences</td>\n",
              "      <td>Human Verification</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>Feedback button or manual review depending on use case</td>\n",
              "      <td>User Feedbacks, Manual Review</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>user feedback</td>\n",
              "      <td>User Feedbacks</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8e8a9920-c516-4674-a414-3da34441b424')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-8e8a9920-c516-4674-a414-3da34441b424 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-8e8a9920-c516-4674-a414-3da34441b424');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q11_question = questions[11]\n",
        "q11_instructions = \"\"\"I received answers from an MLOps survey about Large Language Models (LLMs).\n",
        "Your task is to summarize the stories told in each answer with as few words as possible.\n",
        "If no stories are mentioned, you should use an empty string as a placeholder.\n",
        "You should output these information in the JSON lines format.\n",
        "You should generate one JSON line per answer.\n",
        "Here's an example of an answer:\n",
        "- The extension hace more than 200,000 downloads and it work great!\n",
        "- ChatGPT API output is different every time for the same input - makes it hard to create a reliable service.\n",
        "- They are amazing. But as mentioned, a chatbot alone is probably not enough to be useful, hence building out other capabilities will become important.\n",
        "Here's an example of the expected output:\n",
        "{\"stories\": \"LLM leads to great success\"}\n",
        "{\"stories\": \"Reliability with LLM is hard\"}\n",
        "{\"stories\": \"Integrations are important for chatbot\"}\n",
        "Answers:\"\"\"\n",
        "q11_inputs = results.iloc[:, 11]\n",
        "print(f'Question 11:', q11_question)\n",
        "q11_reviews = associate(model, q11_instructions, q11_inputs)\n",
        "q11_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "XC4ADFVPCHru",
        "outputId": "0fa70407-06c6-4f57-8a28-a9067f51c754"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 11: Any stories you have that are worth sharing about working with LLMs?\n",
            "- Inputs: 110\n",
            "- batch: 0 -> 20\n",
            "- tokens: 882\n",
            "Warning! Got: 18, expected: 20\n",
            "- batch: 1 -> 4\n",
            "- tokens: 366\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Any stories you have that are worth sharing about working with LLMs?  \\\n",
              "7                                                                                                                                                                                                                                                                                                                                                                                                                           I am not sure what ranks as a LLM. Anything bert and beyond? Anyway, we once used setfit for a production task but found that performance was really tricky to debug. The proof of concept was fine. Production was a complete mess. To this day I dont know why- underlying hardware?   \n",
              "9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               lo   \n",
              "22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        They take every Input equally serious which makes for some great laughs…   \n",
              "28                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      One of our citizen data scientists now (figuratively) \"worships\" ChatGPT Gods given its capabilities. LOL.   \n",
              "38                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ChatGPT API output is different every time for the same input - makes it hard to create a reliable service.   \n",
              "39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    confidential   \n",
              "41                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              No   \n",
              "45                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 Content language matters a lot!   \n",
              "50                                                                                                                                                                                                                                                                                                                                                                                                                                                                       Yes, the ODF I created was regarded as best in class by regulators and has been suggested to be implemented as new gold standard. I used GPT3 to advise on the design of the framework, and to write all the necessary code to create it.   \n",
              "52                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           They are amazing. But as mentioned, a chatbot alone is probably not enough to be useful, hence building out other capabilities will become important.   \n",
              "56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               The extension hace more than 200,000 downloads and it work great!   \n",
              "61                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     davinci-003 hallucinating using an existing tool without actually using it!   \n",
              "65                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     We aim to turn a report around from survey finishing to client inbox in 24 hour. So far we are at 48 hours.   \n",
              "68                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      not so far   \n",
              "69   We had a nice time selecting models, proving it would all work and using TSNE and PCA to reduce the dimensionality of embeddings and get compression for our vector lookup system... Very good and satisfying work, we got prizes and praising from all. Then camesome dreadful times making our vector lookup system work under load for long durations in a proper prod environment (where we couldn't touch it to do things like restart it or reload the index). I think this will be a big issue in the future - we need things that are as reliable as oracle for running the inferences, we do not have them - everything is glued together (I am guessing this is not true if you work for Google...)   \n",
              "73                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Not really   \n",
              "84                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             As far as embeddings are concerned, we haven't found a clear advantage of LLMs vs simpler models (tfidf) trained on a relevant dataset. Both require elaborate preprocessing of the embedded documents for example.   \n",
              "87                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               -   \n",
              "98                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                not that i can share cheaply, ha   \n",
              "102                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       we fine tune a bert sized model with our data and run the inference in batch. We experimented with quantization and distillation and its pretty impressive how much you can do with much smaller models.   \n",
              "104                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Microsoft's call summarization tool can capture a surprising degree of detail while hilariously completely missing the main point of the meeting   \n",
              "109                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       extremely power-hungry, hard to scale; networking bandwidth is as important as memory when parallelising   \n",
              "\n",
              "                                                                        stories  \n",
              "7                                                   Production issues with LLMs  \n",
              "9                                                                                \n",
              "22                                             LLMs provide entertainment value  \n",
              "28                   Citizen data scientist impressed with ChatGPT capabilities  \n",
              "38                                         Reliability with ChatGPT API is hard  \n",
              "39                                                                               \n",
              "41                                                                               \n",
              "45                                       Content language is important for LLMs  \n",
              "50                              GPT3 used to create best in class ODF framework  \n",
              "52                                       Integrations are important for chatbot  \n",
              "56                                                   LLM leads to great success  \n",
              "61                                       LLM hallucination without actual usage  \n",
              "65              Challenges with production environment for vector lookup system  \n",
              "68                                                                               \n",
              "69                               No clear advantage of LLMs over simpler models  \n",
              "73                                                                               \n",
              "84                                                                               \n",
              "87                                                  No experience with LLMs yet  \n",
              "98                                                                               \n",
              "102                    Fine-tuned BERT model with quantization and distillation  \n",
              "104   Microsoft's call summarization tool captures detail but misses main point  \n",
              "109  LLMs are power-hungry and hard to scale, networking bandwidth is important  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-d27dd227-4563-4f2d-b30c-ac0374daacf9\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Any stories you have that are worth sharing about working with LLMs?</th>\n",
              "      <th>stories</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>I am not sure what ranks as a LLM. Anything bert and beyond? Anyway, we once used setfit for a production task but found that performance was really tricky to debug. The proof of concept was fine. Production was a complete mess. To this day I dont know why- underlying hardware?</td>\n",
              "      <td>Production issues with LLMs</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>lo</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>They take every Input equally serious which makes for some great laughs…</td>\n",
              "      <td>LLMs provide entertainment value</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>One of our citizen data scientists now (figuratively) \"worships\" ChatGPT Gods given its capabilities. LOL.</td>\n",
              "      <td>Citizen data scientist impressed with ChatGPT capabilities</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>ChatGPT API output is different every time for the same input - makes it hard to create a reliable service.</td>\n",
              "      <td>Reliability with ChatGPT API is hard</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>confidential</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>Content language matters a lot!</td>\n",
              "      <td>Content language is important for LLMs</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>Yes, the ODF I created was regarded as best in class by regulators and has been suggested to be implemented as new gold standard. I used GPT3 to advise on the design of the framework, and to write all the necessary code to create it.</td>\n",
              "      <td>GPT3 used to create best in class ODF framework</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>They are amazing. But as mentioned, a chatbot alone is probably not enough to be useful, hence building out other capabilities will become important.</td>\n",
              "      <td>Integrations are important for chatbot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>The extension hace more than 200,000 downloads and it work great!</td>\n",
              "      <td>LLM leads to great success</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>davinci-003 hallucinating using an existing tool without actually using it!</td>\n",
              "      <td>LLM hallucination without actual usage</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>We aim to turn a report around from survey finishing to client inbox in 24 hour. So far we are at 48 hours.</td>\n",
              "      <td>Challenges with production environment for vector lookup system</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>not so far</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>69</th>\n",
              "      <td>We had a nice time selecting models, proving it would all work and using TSNE and PCA to reduce the dimensionality of embeddings and get compression for our vector lookup system... Very good and satisfying work, we got prizes and praising from all. Then camesome dreadful times making our vector lookup system work under load for long durations in a proper prod environment (where we couldn't touch it to do things like restart it or reload the index). I think this will be a big issue in the future - we need things that are as reliable as oracle for running the inferences, we do not have them - everything is glued together (I am guessing this is not true if you work for Google...)</td>\n",
              "      <td>No clear advantage of LLMs over simpler models</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>73</th>\n",
              "      <td>Not really</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>84</th>\n",
              "      <td>As far as embeddings are concerned, we haven't found a clear advantage of LLMs vs simpler models (tfidf) trained on a relevant dataset. Both require elaborate preprocessing of the embedded documents for example.</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87</th>\n",
              "      <td>-</td>\n",
              "      <td>No experience with LLMs yet</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>not that i can share cheaply, ha</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>102</th>\n",
              "      <td>we fine tune a bert sized model with our data and run the inference in batch. We experimented with quantization and distillation and its pretty impressive how much you can do with much smaller models.</td>\n",
              "      <td>Fine-tuned BERT model with quantization and distillation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>Microsoft's call summarization tool can capture a surprising degree of detail while hilariously completely missing the main point of the meeting</td>\n",
              "      <td>Microsoft's call summarization tool captures detail but misses main point</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>extremely power-hungry, hard to scale; networking bandwidth is as important as memory when parallelising</td>\n",
              "      <td>LLMs are power-hungry and hard to scale, networking bandwidth is important</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d27dd227-4563-4f2d-b30c-ac0374daacf9')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-d27dd227-4563-4f2d-b30c-ac0374daacf9 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-d27dd227-4563-4f2d-b30c-ac0374daacf9');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q12_question = questions[12]\n",
        "q12_instructions = \"\"\"I received answers from an MLOps survey about Large Language Models (LLMs).\n",
        "Your task is to summarize the questions found in each answer with as few words as possible.\n",
        "If no questions are mentioned, you should use an empty string as a placeholder.\n",
        "You should output these information in the JSON lines format.\n",
        "You should generate one JSON line per answer.\n",
        "Here's an example of an answer:\n",
        "- Best automated/scalable pipeline for tuning,, transfer learning (augmentation), reinforcement feedback\n",
        "- Growing LLM features iteratively & coherently vs Alphabet's \"put LLM in all the things!\"\n",
        "- What are some of the best practices for dealing with LLMs or foundation models?\n",
        "Here's an example of the expected output:\n",
        "{\"questions\": \"What is the best pipeline architecture for LLM?\"}\n",
        "{\"questions\": \"How to grow LLM features iteratively and coherently?\"}\n",
        "{\"questions\": \"What are the best practices for deadline with LLMs and foundation models?\"}\n",
        "Answers:\"\"\"\n",
        "q12_inputs = results.iloc[:, 12]\n",
        "print(f'Question 12:', q12_question)\n",
        "q12_reviews = associate(model, q12_instructions, q12_inputs)\n",
        "q12_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "oyRQh4PwCNtY",
        "outputId": "4fafa7c8-4841-4731-999c-25e263d5e91f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 12: Any questions you have for the community about LLM in production?\n",
            "- Inputs: 110\n",
            "- batch: 0 -> 20\n",
            "- tokens: 946\n",
            "Warning! Got: 19, expected: 20\n",
            "- batch: 1 -> 4\n",
            "- tokens: 380\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                                                        Any questions you have for the community about LLM in production?  \\\n",
              "7    I mainly wonder how everyone is monitoring these. ALSO. For my company GDPR is a big deal. My DPO shoots down our ideas more often than not or limits it to using the last three months of data. How do other people deal with this?   \n",
              "9                                                                                                                                                                                                                                      lo   \n",
              "19                                                         Where are others drawing the line between training models locally and using LLMs? Have others had success with using LLMs to bootstrap training data for smaller local models?   \n",
              "25                                                                  Any opinions regarding the newer AI accelerator chips? Many people fighting for GPUs, but not necessarily mentioning AWS Inferentia 2 chips, TPUs, Habana Gaudi, etc.   \n",
              "28                                                                                                                                 None at the moment given our priorities are mostly with computer vision and time-series data analysis.   \n",
              "30                                                                                                                                                                       How are you thinking about tracking user feedback in production?   \n",
              "38                                                                                                                                                                                     How are you managing training/deployment of these?   \n",
              "39                                                                                                                                                                                                                                     no   \n",
              "41                                                                                                                                 Best automated/scalable pipeline for tuning,, transfer learning (augmentation), reinforcement feedback   \n",
              "44                                                      How might you avoid siloing all of your embeddings when different teams need different tools without building it all in house yet all would benefit from the combined collection?   \n",
              "46                                                                                                                              How to better deploy python apps using aws and operationalizing it.\\nHow to better manage vector indexes.   \n",
              "50                                                                                                                                     Nope, no questions, just a great big thankyou, and a virtual high five for all the awesome peeps 🙏   \n",
              "61                                                                                                                            How should I split my initial prompt up between turbo’s System prompt and the first Human prompt I give it?   \n",
              "68                                                                                                      it would be interesting a discussion about how to control the output and in general the kind of usage the system has been exposed   \n",
              "70                                                                                                                                                                   How is everyone thinking about fine-tuning LLMs for their use-cases?   \n",
              "72                                                                                                                                                        What are some of the best practices for dealing with LLMs or foundation models?   \n",
              "73                                                                                                                                                                                                           Chatgpt for domain knowledge   \n",
              "74                                                                                                                                                                                                 Curious to hear about other use cases!   \n",
              "84                                                                                                                                                     I'd love to connect with others who use LLMs embeddings and talk in more detail :)   \n",
              "95                                                                                                                                                                                                 How we manage quality of LLM outputs ?   \n",
              "98                                                                                                                                               Growing LLM features iteratively & coherently vs Alphabet's \"put LLM in all the things!\"   \n",
              "104                                                                                                                                               Best practices to prevent leakage for client facing models that employ online learning?   \n",
              "109                                                                                  is anyone measuring LLM inference power consumption (and CO2 emissions)? what are the observed ranges of watts/inference of the various model sizes?   \n",
              "\n",
              "                                                                                                                                              questions  \n",
              "7                                                                                      How do other people monitor LLMs and deal with GDPR limitations?  \n",
              "9                                                                                                                                                        \n",
              "19              Where is the line drawn between training models locally and using LLMs? Can LLMs be used to bootstrap training data for smaller models?  \n",
              "25                                               What are opinions on newer AI accelerator chips like AWS Inferentia 2 chips, TPUs, Habana Gaudi, etc.?  \n",
              "28                                                                                                            Any questions or opinions regarding LLMs?  \n",
              "30                                                                                                            How to track user feedback in production?  \n",
              "38                                                                                                           How to manage training/deployment of LLMs?  \n",
              "39                                                                                                                                                       \n",
              "41                                                What is the best pipeline architecture for LLM tuning, transfer learning, and reinforcement feedback?  \n",
              "44                                          How to avoid siloing embeddings when different teams need different tools without building it all in house?  \n",
              "46                                                                                       How to deploy python apps using AWS and manage vector indexes?  \n",
              "50                                                                                                                                                       \n",
              "61                                                                How to split initial prompt between turbo's System prompt and the first Human prompt?  \n",
              "68                                                                                                         How to control the output and usage of LLMs?  \n",
              "70                                                                                                        How to fine-tune LLMs for specific use-cases?  \n",
              "72                                                                              What are the best practices for dealing with LLMs or foundation models?  \n",
              "73                                                                                                             How to use Chatgpt for domain knowledge?  \n",
              "74                                                                                                                   What are other use cases for LLMs?  \n",
              "84                                                                                              Looking to connect with others who use LLMs embeddings.  \n",
              "95                                                                                                                How to manage quality of LLM outputs?  \n",
              "98                                                                                                 How to grow LLM features iteratively and coherently?  \n",
              "104                                                       What are the best practices to prevent leakage for client facing models with online learning?  \n",
              "109  Is anyone measuring LLM inference power consumption and CO2 emissions? What are the observed ranges of watts/inference of the various model sizes?  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-6f964438-266c-42a7-b792-878082977a39\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Any questions you have for the community about LLM in production?</th>\n",
              "      <th>questions</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>I mainly wonder how everyone is monitoring these. ALSO. For my company GDPR is a big deal. My DPO shoots down our ideas more often than not or limits it to using the last three months of data. How do other people deal with this?</td>\n",
              "      <td>How do other people monitor LLMs and deal with GDPR limitations?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>lo</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>Where are others drawing the line between training models locally and using LLMs? Have others had success with using LLMs to bootstrap training data for smaller local models?</td>\n",
              "      <td>Where is the line drawn between training models locally and using LLMs? Can LLMs be used to bootstrap training data for smaller models?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>Any opinions regarding the newer AI accelerator chips? Many people fighting for GPUs, but not necessarily mentioning AWS Inferentia 2 chips, TPUs, Habana Gaudi, etc.</td>\n",
              "      <td>What are opinions on newer AI accelerator chips like AWS Inferentia 2 chips, TPUs, Habana Gaudi, etc.?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>None at the moment given our priorities are mostly with computer vision and time-series data analysis.</td>\n",
              "      <td>Any questions or opinions regarding LLMs?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>How are you thinking about tracking user feedback in production?</td>\n",
              "      <td>How to track user feedback in production?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>How are you managing training/deployment of these?</td>\n",
              "      <td>How to manage training/deployment of LLMs?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>no</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>Best automated/scalable pipeline for tuning,, transfer learning (augmentation), reinforcement feedback</td>\n",
              "      <td>What is the best pipeline architecture for LLM tuning, transfer learning, and reinforcement feedback?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>How might you avoid siloing all of your embeddings when different teams need different tools without building it all in house yet all would benefit from the combined collection?</td>\n",
              "      <td>How to avoid siloing embeddings when different teams need different tools without building it all in house?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>How to better deploy python apps using aws and operationalizing it.\\nHow to better manage vector indexes.</td>\n",
              "      <td>How to deploy python apps using AWS and manage vector indexes?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>Nope, no questions, just a great big thankyou, and a virtual high five for all the awesome peeps 🙏</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>How should I split my initial prompt up between turbo’s System prompt and the first Human prompt I give it?</td>\n",
              "      <td>How to split initial prompt between turbo's System prompt and the first Human prompt?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>it would be interesting a discussion about how to control the output and in general the kind of usage the system has been exposed</td>\n",
              "      <td>How to control the output and usage of LLMs?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>70</th>\n",
              "      <td>How is everyone thinking about fine-tuning LLMs for their use-cases?</td>\n",
              "      <td>How to fine-tune LLMs for specific use-cases?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>72</th>\n",
              "      <td>What are some of the best practices for dealing with LLMs or foundation models?</td>\n",
              "      <td>What are the best practices for dealing with LLMs or foundation models?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>73</th>\n",
              "      <td>Chatgpt for domain knowledge</td>\n",
              "      <td>How to use Chatgpt for domain knowledge?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>74</th>\n",
              "      <td>Curious to hear about other use cases!</td>\n",
              "      <td>What are other use cases for LLMs?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>84</th>\n",
              "      <td>I'd love to connect with others who use LLMs embeddings and talk in more detail :)</td>\n",
              "      <td>Looking to connect with others who use LLMs embeddings.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>95</th>\n",
              "      <td>How we manage quality of LLM outputs ?</td>\n",
              "      <td>How to manage quality of LLM outputs?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>Growing LLM features iteratively &amp; coherently vs Alphabet's \"put LLM in all the things!\"</td>\n",
              "      <td>How to grow LLM features iteratively and coherently?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>Best practices to prevent leakage for client facing models that employ online learning?</td>\n",
              "      <td>What are the best practices to prevent leakage for client facing models with online learning?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>is anyone measuring LLM inference power consumption (and CO2 emissions)? what are the observed ranges of watts/inference of the various model sizes?</td>\n",
              "      <td>Is anyone measuring LLM inference power consumption and CO2 emissions? What are the observed ranges of watts/inference of the various model sizes?</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6f964438-266c-42a7-b792-878082977a39')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-6f964438-266c-42a7-b792-878082977a39 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-6f964438-266c-42a7-b792-878082977a39');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q13_question = questions[13]\n",
        "q13_instructions = \"\"\"I received answers from an MLOps survey about Large Language Models (LLMs).\n",
        "Your task is to extract the reasons found in each answer for not using LLMs.\n",
        "If no reasons are mentioned, you should use an empty string as a placeholder.\n",
        "You should output these information in the JSON lines format.\n",
        "You should generate one JSON line per answer.\n",
        "Here's an example of an answer:\n",
        "- No business need yet\n",
        "- Data privacy concerns\n",
        "- inference cost vs. impact on revenue\n",
        "- It's only just now becoming feasible with the ChatGPT API. Before it was too expensive and quality was not good enough.\t\n",
        "Here's an example of the expected output:\n",
        "{\"reasons\": \"Lack of Use Case\"}\n",
        "{\"reasons\": \"Data Privacy\"}\n",
        "{\"reasons\": \"Return of Investment\"}\n",
        "{\"reasons\": \"Cost, Quality\"}\n",
        "Answers:\"\"\"\n",
        "q13_inputs = results.iloc[:, 13]\n",
        "print(f'Question 13:', q13_question)\n",
        "q13_reviews = associate(model, q13_instructions, q13_inputs)\n",
        "q13_reviews.dropna()"
      ],
      "metadata": {
        "id": "2cekCDiZGji3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "a1a03cf1-1c8d-406c-e6ce-68e7f262580e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 13: What is the main reason for not using LLMs in your org?\n",
            "- Inputs: 110\n",
            "- batch: 0 -> 20\n",
            "- tokens: 796\n",
            "Warning! Got: 17, expected: 20\n",
            "- batch: 1 -> 20\n",
            "- tokens: 736\n",
            "- batch: 2 -> 1\n",
            "- tokens: 199\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                                                                                                                                                                                                                                                                                                         What is the main reason for not using LLMs in your org?  \\\n",
              "1                                                                                                                                                                                                                                                                                                                                                                                                                                                    Production takes investment   \n",
              "6                                                                                                                                                                                                                                                                                                                                                        It's only just now becoming feasible with the ChatGPT API. Before it was too expensive and quality was not good enough.   \n",
              "8                                                                                                                                                                                                                                                                                                                                                                                                                                                                No applications   \n",
              "16                                                                                                                                                                                          Two main reasons: 1. Very compute & power hungry 2. Cannot be relied upon as it will very confidently return incorrect information or in “existent facts” which is problematic for use in my industry.\\n\\nEg it took us less than 3mn to get BioGPT to tell us vaccines cause autism   \n",
              "17                                                                                            We use medium-sized language models (e.g. RoBERTa-base) as the backbone for our deployed NLP models because they can perform inference on CPU, which is cheap. We don't use LLMs (e.g. ChatGPT, GPT-3) because the APIs are expensive, and fine-tuning them to perform NLP tasks other than language generation (e.g. NER, open-set intent classification) is not straightforward.   \n",
              "18                                                                                                                                                                                                                                                                                                                                                                                                                                                      Not much usecases for us   \n",
              "21                                                                                                                                                                                                                                                                                                                                                                                                                                                  Not applicable to the domain   \n",
              "23                                                                                                                                                                                                                                                                                                                                                                                                                              Production LLM lifecycle and use cases not clear   \n",
              "24                                                                                                                                                                                                                                                                                                                                                                                                                                                         Cost / right use-case   \n",
              "27                                                                                                                                                                                                                                                                                                                                                                                                                                                             Exploration stage   \n",
              "32                                                                                                                                                                                                                                                                                                                                                                                                                                                          No business need yet   \n",
              "33                                                                                                                                                                                                                                                                                                                                                                                                                                                                We're learning   \n",
              "35                                                                                                                                                                                                                                                                                                                                                                                                                                                       Cost, Latency, Accuracy   \n",
              "36                                                                                                                                                                                                                                                                                                                                                                                                                                                      not really in our domain   \n",
              "40   1. gcm diagflow is good for complete basic dialogs, 2. halucinations 3.a. how to make answers be focused on what my company is doing rather than general answers, 3.b. how to keep that evolving as we learn more about the customers 3.c. how to prevent a practical joker who's conversing with my llm to inject some practical joke into it 4. if im relying on openai-s llm, i need to always get a reply what happens if they're busy, whats the proverbial 'failover'   \n",
              "42                                                                                                                                                                                                                                                                                                                                                                                                                                                                   No need atm   \n",
              "43                                                                                                                                                                                                                                                                                                                                                                                                                                       we'll be using at a startup I'm part of   \n",
              "55                                                                                                                                                                                                                                                                                                                                                                                                                                             Because I am in education sector.   \n",
              "59                                                                                                                                                                                                                                                                                                                                                                                                                                          inference cost vs. impact on revenue   \n",
              "62                                                                                                                                                                                                                                                                                                                                                                                               No Use case, being a fintech startup, besides helping with processing documents   \n",
              "63                                                                                                                                                                                                                                                                                                                                                                                                                                                             No clear use case   \n",
              "64                                                                                                                                                                                                                                                                                                                                                                                                                                                                           Not   \n",
              "67                                                                                                                                                                                                                                                                    We don't have a need for generating unstructured data, we work in small languages, our text is very domain specific, LLMs are very compute intensive -> expensive to run and take a lot of work to set up.   \n",
              "75                                                                                                                                                                                                                                                                                                                                                                                                                                                                   No need yet   \n",
              "77                                                                                                                                                                                                                                                                                                                                                                                                     Currently assessing the risk and potential applications, performing POCs.   \n",
              "78                                                                                                                                                                                                                                                                                                                                                                                             Wee are exploring the space but don't feel they are ready for production use yeet   \n",
              "80                                                                                                                                                                                                                                                                                                                                                                                                                                                                             $   \n",
              "82                                                                                                                                                                                                                                                                                                                                                                             MLOps and DataScience rather young discipline. We are currently working on an LLM use case though   \n",
              "83                                                                                                                                                                                                                                                                                                                                                                              Domain specific challenges for what we do. Digging into how we could train LLM to fit the needs.   \n",
              "85                                                                                                                                                                                                                                                                                                                                                                                                                         it isn't in focus on my leaders to use LLMs right now   \n",
              "86                                                                                                                                                                                                                                                                                                                                                                                                                                             No current need, but also tooling   \n",
              "92                                                                                                                                                                                                                                                                                                                                                                                                                                                             Legal said not to   \n",
              "94                                                                                                                                                                                                                                                                                                                                                                                                 currently using classic ML and I plan to introduce LLM capabilities to my org   \n",
              "96                                                                                                                                                                                                                                                                                                                                                                                                                           Lack of applicable business use cases and resources   \n",
              "97                                                                                                                                                                                                                                                                                                                                                                                                                                         Expensive, no ROI, complex, dangerous   \n",
              "100                                                                                                                                                                                                                                                                                                                                                                                                                                            No clear use case for our product   \n",
              "103                                                                                                                                                                                                                                                                                                                                                                              LLMs and their hallucinations just are not good for our current use cases (healthcare industry)   \n",
              "105                                                                                                                                                                                                                                                                                                                                                                                                                                                               Less expertise   \n",
              "\n",
              "                                                                                 reasons  \n",
              "1                                                                        Production Cost  \n",
              "6                                                                          Cost, Quality  \n",
              "8                                                                       Lack of Use Case  \n",
              "16                                                                  Compute, Reliability  \n",
              "17                                         Cost, API Limitations, Fine-tuning Difficulty  \n",
              "18                                                                      Lack of Use Case  \n",
              "21                                                                        Not Applicable  \n",
              "23                                                          Production Lifecycle Unclear  \n",
              "24                                                                  Cost, Learning Stage  \n",
              "27                                                               Cost, Latency, Accuracy  \n",
              "32                                                                         Not in Domain  \n",
              "33                        Dialog Management, Answer Relevance, Practical Jokes, Failover  \n",
              "35                                                                      No Business Need  \n",
              "36                                                                   Startup Use Planned  \n",
              "40                                                                          Data Privacy  \n",
              "42                                               Infrastructure Evaluation, Traceability  \n",
              "43                                                                      Lack of Use Case  \n",
              "55                                                                      Education sector  \n",
              "59                                                                  Return of Investment  \n",
              "62                                     Lack of Use Case, except for processing documents  \n",
              "63                                                                      Lack of Use Case  \n",
              "64                                                                                        \n",
              "67              Domain specific language, compute intensive, expensive to run and set up  \n",
              "75                                                                           No need yet  \n",
              "77                            Assessing risk and potential applications, performing POCs  \n",
              "78                                                      Not ready for production use yet  \n",
              "80                                                                                  Cost  \n",
              "82   MLOps and DataScience rather young discipline, currently working on an LLM use case  \n",
              "83               Domain specific challenges, exploring how to train LLM to fit the needs  \n",
              "85                                         Not in focus of leaders to use LLMs right now  \n",
              "86                                                      No current need, lack of tooling  \n",
              "92                                                                    Legal restrictions  \n",
              "94                                Plan to introduce LLM capabilities to the organization  \n",
              "96                                   Lack of applicable business use cases and resources  \n",
              "97                                                 Expensive, no ROI, complex, dangerous  \n",
              "100                                                                     Lack of Use Case  \n",
              "103                                Not good for current use cases in healthcare industry  \n",
              "105                                                                       Less expertise  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-89ed41f3-0e9b-4149-82e5-21352bf11320\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>What is the main reason for not using LLMs in your org?</th>\n",
              "      <th>reasons</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Production takes investment</td>\n",
              "      <td>Production Cost</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>It's only just now becoming feasible with the ChatGPT API. Before it was too expensive and quality was not good enough.</td>\n",
              "      <td>Cost, Quality</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>No applications</td>\n",
              "      <td>Lack of Use Case</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>Two main reasons: 1. Very compute &amp; power hungry 2. Cannot be relied upon as it will very confidently return incorrect information or in “existent facts” which is problematic for use in my industry.\\n\\nEg it took us less than 3mn to get BioGPT to tell us vaccines cause autism</td>\n",
              "      <td>Compute, Reliability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>We use medium-sized language models (e.g. RoBERTa-base) as the backbone for our deployed NLP models because they can perform inference on CPU, which is cheap. We don't use LLMs (e.g. ChatGPT, GPT-3) because the APIs are expensive, and fine-tuning them to perform NLP tasks other than language generation (e.g. NER, open-set intent classification) is not straightforward.</td>\n",
              "      <td>Cost, API Limitations, Fine-tuning Difficulty</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>Not much usecases for us</td>\n",
              "      <td>Lack of Use Case</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>Not applicable to the domain</td>\n",
              "      <td>Not Applicable</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>Production LLM lifecycle and use cases not clear</td>\n",
              "      <td>Production Lifecycle Unclear</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>Cost / right use-case</td>\n",
              "      <td>Cost, Learning Stage</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>Exploration stage</td>\n",
              "      <td>Cost, Latency, Accuracy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>No business need yet</td>\n",
              "      <td>Not in Domain</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>33</th>\n",
              "      <td>We're learning</td>\n",
              "      <td>Dialog Management, Answer Relevance, Practical Jokes, Failover</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>Cost, Latency, Accuracy</td>\n",
              "      <td>No Business Need</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>not really in our domain</td>\n",
              "      <td>Startup Use Planned</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>1. gcm diagflow is good for complete basic dialogs, 2. halucinations 3.a. how to make answers be focused on what my company is doing rather than general answers, 3.b. how to keep that evolving as we learn more about the customers 3.c. how to prevent a practical joker who's conversing with my llm to inject some practical joke into it 4. if im relying on openai-s llm, i need to always get a reply what happens if they're busy, whats the proverbial 'failover'</td>\n",
              "      <td>Data Privacy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>42</th>\n",
              "      <td>No need atm</td>\n",
              "      <td>Infrastructure Evaluation, Traceability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43</th>\n",
              "      <td>we'll be using at a startup I'm part of</td>\n",
              "      <td>Lack of Use Case</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>55</th>\n",
              "      <td>Because I am in education sector.</td>\n",
              "      <td>Education sector</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>59</th>\n",
              "      <td>inference cost vs. impact on revenue</td>\n",
              "      <td>Return of Investment</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>62</th>\n",
              "      <td>No Use case, being a fintech startup, besides helping with processing documents</td>\n",
              "      <td>Lack of Use Case, except for processing documents</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>63</th>\n",
              "      <td>No clear use case</td>\n",
              "      <td>Lack of Use Case</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>64</th>\n",
              "      <td>Not</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>67</th>\n",
              "      <td>We don't have a need for generating unstructured data, we work in small languages, our text is very domain specific, LLMs are very compute intensive -&gt; expensive to run and take a lot of work to set up.</td>\n",
              "      <td>Domain specific language, compute intensive, expensive to run and set up</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75</th>\n",
              "      <td>No need yet</td>\n",
              "      <td>No need yet</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>77</th>\n",
              "      <td>Currently assessing the risk and potential applications, performing POCs.</td>\n",
              "      <td>Assessing risk and potential applications, performing POCs</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>78</th>\n",
              "      <td>Wee are exploring the space but don't feel they are ready for production use yeet</td>\n",
              "      <td>Not ready for production use yet</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>80</th>\n",
              "      <td>$</td>\n",
              "      <td>Cost</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>82</th>\n",
              "      <td>MLOps and DataScience rather young discipline. We are currently working on an LLM use case though</td>\n",
              "      <td>MLOps and DataScience rather young discipline, currently working on an LLM use case</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>83</th>\n",
              "      <td>Domain specific challenges for what we do. Digging into how we could train LLM to fit the needs.</td>\n",
              "      <td>Domain specific challenges, exploring how to train LLM to fit the needs</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>85</th>\n",
              "      <td>it isn't in focus on my leaders to use LLMs right now</td>\n",
              "      <td>Not in focus of leaders to use LLMs right now</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>86</th>\n",
              "      <td>No current need, but also tooling</td>\n",
              "      <td>No current need, lack of tooling</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>92</th>\n",
              "      <td>Legal said not to</td>\n",
              "      <td>Legal restrictions</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>94</th>\n",
              "      <td>currently using classic ML and I plan to introduce LLM capabilities to my org</td>\n",
              "      <td>Plan to introduce LLM capabilities to the organization</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>96</th>\n",
              "      <td>Lack of applicable business use cases and resources</td>\n",
              "      <td>Lack of applicable business use cases and resources</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>97</th>\n",
              "      <td>Expensive, no ROI, complex, dangerous</td>\n",
              "      <td>Expensive, no ROI, complex, dangerous</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>100</th>\n",
              "      <td>No clear use case for our product</td>\n",
              "      <td>Lack of Use Case</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>103</th>\n",
              "      <td>LLMs and their hallucinations just are not good for our current use cases (healthcare industry)</td>\n",
              "      <td>Not good for current use cases in healthcare industry</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>105</th>\n",
              "      <td>Less expertise</td>\n",
              "      <td>Less expertise</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-89ed41f3-0e9b-4149-82e5-21352bf11320')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-89ed41f3-0e9b-4149-82e5-21352bf11320 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-89ed41f3-0e9b-4149-82e5-21352bf11320');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q14_question = questions[14]\n",
        "q14_instructions = \"\"\"I received answers from an MLOps survey about Large Language Models (LLMs).\n",
        "Your task is to extract the challenges found in each answer for using LLMs in production.\n",
        "If no reasons are mentioned, you should use an empty string as a placeholder.\n",
        "You should output these information in the JSON lines format.\n",
        "You should generate one JSON line per answer.\n",
        "Here's an example of an answer:\n",
        "- ROI of more expensive models\n",
        "- Is it reliable? How do we know it's reliable?\n",
        "- how to get it to use my company's specific knowledge\n",
        "- How to reduce inference cost can be high, hallucinations\n",
        "Here's an example of the expected output:\n",
        "{\"challenges\": \"ROI\"}\n",
        "{\"challenges\": \"Reliabiligy\"}\n",
        "{\"challenges\": \"Domain Specific Knowledge\"}\n",
        "{\"challenges\": \"Inference Cost, Hallucinations\"}\n",
        "Answers:\"\"\"\n",
        "q14_inputs = results.iloc[:, 14]\n",
        "print(f'Question 14:', q14_question)\n",
        "q14_reviews = associate(model, q14_instructions, q14_inputs)\n",
        "q14_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "ST1diY8dDTfQ",
        "outputId": "c8d37453-b6cf-4656-a331-5cbd4bd5baba"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 14: What are some key questions you face when it comes to using LLM in prod?\n",
            "- Inputs: 110\n",
            "- batch: 0 -> 20\n",
            "- tokens: 698\n",
            "- batch: 1 -> 13\n",
            "- tokens: 529\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                                                                                                                                                                 What are some key questions you face when it comes to using LLM in prod?  \\\n",
              "1                                                                                                                                                                                                                                                                                                                            Cost to maintain the service   \n",
              "6                                                                                                                                                                                                                                                                                                                              How to make them reliable.   \n",
              "8                                                                                                                                                                                                                                                                                                                            ROI of more expensive models   \n",
              "16                                                                                                                                                 What uses cases does it have for our business? how much gain do we get from using an LLM vs a simpler language model where we can control fact and outputs? How much will it cost (to deploy, and run)   \n",
              "17   How can we ensure the same version of the model is being used? How can we supply a random seed to get deterministic results? How can we minimize LLM API usage costs? How can we fine-tune a LLM on a niche task without having to replicate the weights of the LLM? How can we deploy our own copy of an open-source LLM without breaking the bank?   \n",
              "21                                                                                                                                                                                                                                                                                                                                                Not yet   \n",
              "23                                                                                                                                                                                                                                                                                                                                        Data governance   \n",
              "27                                                                                                                                                                                                                                                                             Putting company data in the right structure and updating the model with it   \n",
              "35                                                                                                                                                                                                                                                                                                                       Cost can be high, hallucinations   \n",
              "40                                                                                                                                                                                                                                                                                                   how to get it to use my company's specific knowledge   \n",
              "42                                                                                                                                                                                                                                                                                                                                            Reliability   \n",
              "43                                                                                                                                                                                                                                                                                                     cost and fit for purpose. Actual market validation   \n",
              "47                                                                                                                                                                                                                                                                                                 Difficulty or ease of maintenance; costs; data privacy   \n",
              "51                                                                                                                                                                                                                                                                                              Retraining, tackle hallucinations and avoid wrong results   \n",
              "54                                                                                                                                                                                                                                                                                                   cost of maintenance, lack of institutional knowledge   \n",
              "55                                                                                                                                                                                                                                                                                                    when you use LLM in production it can fail any time   \n",
              "59                                                                                                                                                                                                                                                                                                                           how to reduce inference cost   \n",
              "62                                                                                                                                                                                                                                                                                                                                    why waste the money   \n",
              "63                                                                                                                                                                                                                                                                                                                        Running them, fine tuning, cost   \n",
              "67                                                                                                                                                                                                                                            Cost, latency, actual performance in the languages / domains we work with (real languages, not programming)   \n",
              "75                                                                                                                                                                                                                                                                                                                                      I don't need them   \n",
              "77                                                                                                                                                                                          What are the potential risks, where are the benefits worth the costs for enterprise grade integration, how do we make these things secure, how do we monitor?   \n",
              "78                                                                                                                                                                                                                                            How to pin them to a domain, how to verify the veracity of teh outcome and how to best chain them with APIs   \n",
              "80                                                                                                                                                                                                                                                                                                                                                      $   \n",
              "82                                                                                                                                                                                                                                                                                                                               Runtime, Drift detection   \n",
              "83                                                                                                                                                                                                                                                       What is it? There is just the huge gap in understanding about analyzing text vs generating text.   \n",
              "85                                                                                                                                                                                                  Two key questions I do myself on LLM in prod are: 1 - How is the best strategy to train or fine-tune a LLM ? 2 - How to manage infrastructure costs ?   \n",
              "86                                                                                                                                                                                                                                                                                                    Reliability, safety, certainty, hallucinations, etc   \n",
              "92                                                                                                                                                                                                                                                                                             How do we keep costs down? How do we validate the outputs?   \n",
              "94                                                                                                                                                                                                                                                                                                        cost of training and inference, time, gpu, ....   \n",
              "96                                                                                                                                                                                                                                                                                                                                 How do you serve them?   \n",
              "97                                                                                                                                                                                                                                                                                                          Is it reliable? How do we know it's reliable?   \n",
              "105                                                                                                                                                                                                                                                                                                                             Large size models latency   \n",
              "\n",
              "                                                                      challenges  \n",
              "1                                                            Cost of Maintenance  \n",
              "6                                                                    Reliability  \n",
              "8                                                                            ROI  \n",
              "16                                                      Business Use Cases, Cost  \n",
              "17   Version Control, Random Seed, API Usage Costs, Fine-tuning, Deployment Cost  \n",
              "21                                                                                \n",
              "23                                                               Data Governance  \n",
              "27                                                              Data Preparation  \n",
              "35                                                Inference Cost, Hallucinations  \n",
              "40                                                     Domain Specific Knowledge  \n",
              "42                                                                   Reliability  \n",
              "43                                                       Cost, Market Validation  \n",
              "47                                              Maintenance, Costs, Data Privacy  \n",
              "51                                          Retraining, Hallucinations, Accuracy  \n",
              "54                                  Cost of Maintenance, Institutional Knowledge  \n",
              "55                                                                   Reliability  \n",
              "59                                                                Inference Cost  \n",
              "62                                                                          Cost  \n",
              "63                                                    Running, Fine-tuning, Cost  \n",
              "67                                  Cost, Latency, Performance in Real Languages  \n",
              "75                                                                                \n",
              "77                                                     Security, Monitoring, ROI  \n",
              "78              Domain Specific Knowledge, Outcome Verification, API Integration  \n",
              "80                                                                          Cost  \n",
              "82                                                      Runtime, Drift Detection  \n",
              "83                                   Understanding, Analyzing vs Generating Text  \n",
              "85                                       Training Strategy, Infrastructure Costs  \n",
              "86                                Reliability, Safety, Certainty, Hallucinations  \n",
              "92                                                       Cost, Output Validation  \n",
              "94                                                Cost, Training, Inference, GPU  \n",
              "96                                                                 Model Serving  \n",
              "97                                                                   Reliability  \n",
              "105                                                                      Latency  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f0888bf0-2a7f-426f-bd2b-9b66c40bac57\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>What are some key questions you face when it comes to using LLM in prod?</th>\n",
              "      <th>challenges</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Cost to maintain the service</td>\n",
              "      <td>Cost of Maintenance</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>How to make them reliable.</td>\n",
              "      <td>Reliability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>ROI of more expensive models</td>\n",
              "      <td>ROI</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>What uses cases does it have for our business? how much gain do we get from using an LLM vs a simpler language model where we can control fact and outputs? How much will it cost (to deploy, and run)</td>\n",
              "      <td>Business Use Cases, Cost</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>How can we ensure the same version of the model is being used? How can we supply a random seed to get deterministic results? How can we minimize LLM API usage costs? How can we fine-tune a LLM on a niche task without having to replicate the weights of the LLM? How can we deploy our own copy of an open-source LLM without breaking the bank?</td>\n",
              "      <td>Version Control, Random Seed, API Usage Costs, Fine-tuning, Deployment Cost</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>Not yet</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>Data governance</td>\n",
              "      <td>Data Governance</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>Putting company data in the right structure and updating the model with it</td>\n",
              "      <td>Data Preparation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>Cost can be high, hallucinations</td>\n",
              "      <td>Inference Cost, Hallucinations</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>how to get it to use my company's specific knowledge</td>\n",
              "      <td>Domain Specific Knowledge</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>42</th>\n",
              "      <td>Reliability</td>\n",
              "      <td>Reliability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43</th>\n",
              "      <td>cost and fit for purpose. Actual market validation</td>\n",
              "      <td>Cost, Market Validation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>47</th>\n",
              "      <td>Difficulty or ease of maintenance; costs; data privacy</td>\n",
              "      <td>Maintenance, Costs, Data Privacy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>51</th>\n",
              "      <td>Retraining, tackle hallucinations and avoid wrong results</td>\n",
              "      <td>Retraining, Hallucinations, Accuracy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>54</th>\n",
              "      <td>cost of maintenance, lack of institutional knowledge</td>\n",
              "      <td>Cost of Maintenance, Institutional Knowledge</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>55</th>\n",
              "      <td>when you use LLM in production it can fail any time</td>\n",
              "      <td>Reliability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>59</th>\n",
              "      <td>how to reduce inference cost</td>\n",
              "      <td>Inference Cost</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>62</th>\n",
              "      <td>why waste the money</td>\n",
              "      <td>Cost</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>63</th>\n",
              "      <td>Running them, fine tuning, cost</td>\n",
              "      <td>Running, Fine-tuning, Cost</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>67</th>\n",
              "      <td>Cost, latency, actual performance in the languages / domains we work with (real languages, not programming)</td>\n",
              "      <td>Cost, Latency, Performance in Real Languages</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75</th>\n",
              "      <td>I don't need them</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>77</th>\n",
              "      <td>What are the potential risks, where are the benefits worth the costs for enterprise grade integration, how do we make these things secure, how do we monitor?</td>\n",
              "      <td>Security, Monitoring, ROI</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>78</th>\n",
              "      <td>How to pin them to a domain, how to verify the veracity of teh outcome and how to best chain them with APIs</td>\n",
              "      <td>Domain Specific Knowledge, Outcome Verification, API Integration</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>80</th>\n",
              "      <td>$</td>\n",
              "      <td>Cost</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>82</th>\n",
              "      <td>Runtime, Drift detection</td>\n",
              "      <td>Runtime, Drift Detection</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>83</th>\n",
              "      <td>What is it? There is just the huge gap in understanding about analyzing text vs generating text.</td>\n",
              "      <td>Understanding, Analyzing vs Generating Text</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>85</th>\n",
              "      <td>Two key questions I do myself on LLM in prod are: 1 - How is the best strategy to train or fine-tune a LLM ? 2 - How to manage infrastructure costs ?</td>\n",
              "      <td>Training Strategy, Infrastructure Costs</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>86</th>\n",
              "      <td>Reliability, safety, certainty, hallucinations, etc</td>\n",
              "      <td>Reliability, Safety, Certainty, Hallucinations</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>92</th>\n",
              "      <td>How do we keep costs down? How do we validate the outputs?</td>\n",
              "      <td>Cost, Output Validation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>94</th>\n",
              "      <td>cost of training and inference, time, gpu, ....</td>\n",
              "      <td>Cost, Training, Inference, GPU</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>96</th>\n",
              "      <td>How do you serve them?</td>\n",
              "      <td>Model Serving</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>97</th>\n",
              "      <td>Is it reliable? How do we know it's reliable?</td>\n",
              "      <td>Reliability</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>105</th>\n",
              "      <td>Large size models latency</td>\n",
              "      <td>Latency</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f0888bf0-2a7f-426f-bd2b-9b66c40bac57')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-f0888bf0-2a7f-426f-bd2b-9b66c40bac57 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-f0888bf0-2a7f-426f-bd2b-9b66c40bac57');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q15_question = questions[15]\n",
        "q15_instructions = \"\"\"I received answers from an MLOps survey about Large Language Models (LLMs).\n",
        "Your task is to extract the use cases found in each answer for using LLMs in production.\n",
        "If no use cases are mentioned, you should use an empty string as a placeholder.\n",
        "You should also extract if the user replied positively or negatively with \"Yes\" or \"No\".\n",
        "You should output these information in the JSON lines format.\n",
        "You should generate one JSON line per answer.\n",
        "Here's an example of an answer:\n",
        "- no, closest we've got to dealing with language is fuzzy lookups\n",
        "- We are still exploring, eg for support in diagnosis of plant diseases.\n",
        "- I have not, but I know many of the devs want to; there are some other reasonable use cases too.\n",
        "- Nope. Tried some BERT for classification but that hardly counts as an LLM. And I've made models as big as BERT I think.\n",
        "Here's an example of the expected output:\n",
        "{\"tried\": \"No\", \"cases\": \"Fuzzy Lookups\"}\n",
        "{\"tried\": \"Yes\", \"cases\": \"Plan Diseases\"}\n",
        "{\"tried\": \"No\", \"cases\": \"\"}\n",
        "{\"tried\": \"Yes\", \"cases\": \"Classification\"}\n",
        "Answers:\"\"\"\n",
        "q15_inputs = results.iloc[:, 15]\n",
        "print(f'Question 15:', q15_question)\n",
        "q15_reviews = associate(model, q15_instructions, q15_inputs)\n",
        "q15_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "JjeYWWU6F0Qp",
        "outputId": "80589e79-b9c2-4bf2-aac9-d516786af6a0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 15: have you tried LLMs for different use cases in your org?\n",
            "- Inputs: 110\n",
            "- batch: 0 -> 20\n",
            "- tokens: 657\n",
            "- batch: 1 -> 11\n",
            "- tokens: 498\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                          have you tried LLMs for different use cases in your org?  \\\n",
              "1                                                                                                                                                                                              Yes   \n",
              "6                                                                                                                           We are still exploring, eg for support in diagnosis of plant diseases.   \n",
              "8                                                                                                                                                                                              Yes   \n",
              "16                                                                                                                                                            We abandoned the idea pretty quickly   \n",
              "17                                                                                                                                                                                             Yes   \n",
              "23                                                                                                                                                                                              no   \n",
              "27                                                                                                                                                                                             yes   \n",
              "36                                                                                                                                no - closest we've got to dealing with language is fuzzy lookups   \n",
              "40                                                                                                                                                                       not thoroughly enough yet   \n",
              "43                                                                                                                                                          yes. I've used them for classification   \n",
              "47                                                                                                                                                                                         Not yet   \n",
              "51                                                                                                                                                                                              No   \n",
              "54                                                                                                                                                                                              No   \n",
              "55                                                                                                                                                                                              NO   \n",
              "62                                                                                                                                                                                              no   \n",
              "63                                                                                                                                                                                            Nope   \n",
              "67                                                                         Nope. Tried some BERT for classification but that hardly counts as an LLM. And I've made models as big as BERT I think.   \n",
              "75                                                                                                                                                                                  Yes, too heavy   \n",
              "77                                                                                                                                                                     In the process of doing so.   \n",
              "78                                                                                                                              We are exploring customer facing and colleague facing applications   \n",
              "80                                                                                                                                                                                              No   \n",
              "82                                                                                                                                                                                              No   \n",
              "83   We were using s-BERT in our primary recsys model. Recent modeling efforts have reset to simpler systems that are performing better. Still using it in a 2nd model for a clustering challenge.   \n",
              "85                                                                                                                                                                                         not yet   \n",
              "86                                                                                                                                                                                              No   \n",
              "92                                                                                                                                                                                            nope   \n",
              "94                                                                                                                                                                                              no   \n",
              "96                                                                                                                                                                            Only for a developer   \n",
              "97                                                                                                                                                                                             Yes   \n",
              "100                                                                                                                                                                                             No   \n",
              "103                                                                                                I have not, but I know many of the devs want to; there are some other reasonable use cases too.   \n",
              "\n",
              "                tried                                       cases  \n",
              "1                 Yes                                              \n",
              "6                  No                               Plan Diseases  \n",
              "8                 Yes                                              \n",
              "16                 No                                              \n",
              "17                Yes                                              \n",
              "23                 No                                              \n",
              "27                Yes                                              \n",
              "36                 No                               Fuzzy Lookups  \n",
              "40                 No                                              \n",
              "43                Yes                              Classification  \n",
              "47                 No                                              \n",
              "51                 No                                              \n",
              "54                 No                                              \n",
              "55                 NO                                              \n",
              "62                 no                                              \n",
              "63               Nope                                              \n",
              "67               Nope                              Classification  \n",
              "75                Yes                                   Too Heavy  \n",
              "77     In the process                                              \n",
              "78   We are exploring  Customer and Colleague Facing Applications  \n",
              "80                 No                                              \n",
              "82                 No                                              \n",
              "83                Yes           Recommendation System, Clustering  \n",
              "85                 No                                              \n",
              "86                 No                                              \n",
              "92                 No                                              \n",
              "94                 No                                              \n",
              "96                 No                                 Development  \n",
              "97                Yes                                              \n",
              "100                No                                              \n",
              "103                No                Development, Other Use Cases  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-7f1bd965-44ce-43c6-a2f0-42ef1a82a0a9\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>have you tried LLMs for different use cases in your org?</th>\n",
              "      <th>tried</th>\n",
              "      <th>cases</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Yes</td>\n",
              "      <td>Yes</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>We are still exploring, eg for support in diagnosis of plant diseases.</td>\n",
              "      <td>No</td>\n",
              "      <td>Plan Diseases</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>Yes</td>\n",
              "      <td>Yes</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>We abandoned the idea pretty quickly</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>Yes</td>\n",
              "      <td>Yes</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>no</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>yes</td>\n",
              "      <td>Yes</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>no - closest we've got to dealing with language is fuzzy lookups</td>\n",
              "      <td>No</td>\n",
              "      <td>Fuzzy Lookups</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>not thoroughly enough yet</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43</th>\n",
              "      <td>yes. I've used them for classification</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Classification</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>47</th>\n",
              "      <td>Not yet</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>51</th>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>54</th>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>55</th>\n",
              "      <td>NO</td>\n",
              "      <td>NO</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>62</th>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>63</th>\n",
              "      <td>Nope</td>\n",
              "      <td>Nope</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>67</th>\n",
              "      <td>Nope. Tried some BERT for classification but that hardly counts as an LLM. And I've made models as big as BERT I think.</td>\n",
              "      <td>Nope</td>\n",
              "      <td>Classification</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75</th>\n",
              "      <td>Yes, too heavy</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Too Heavy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>77</th>\n",
              "      <td>In the process of doing so.</td>\n",
              "      <td>In the process</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>78</th>\n",
              "      <td>We are exploring customer facing and colleague facing applications</td>\n",
              "      <td>We are exploring</td>\n",
              "      <td>Customer and Colleague Facing Applications</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>80</th>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>82</th>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>83</th>\n",
              "      <td>We were using s-BERT in our primary recsys model. Recent modeling efforts have reset to simpler systems that are performing better. Still using it in a 2nd model for a clustering challenge.</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Recommendation System, Clustering</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>85</th>\n",
              "      <td>not yet</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>86</th>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>92</th>\n",
              "      <td>nope</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>94</th>\n",
              "      <td>no</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>96</th>\n",
              "      <td>Only for a developer</td>\n",
              "      <td>No</td>\n",
              "      <td>Development</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>97</th>\n",
              "      <td>Yes</td>\n",
              "      <td>Yes</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>100</th>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>103</th>\n",
              "      <td>I have not, but I know many of the devs want to; there are some other reasonable use cases too.</td>\n",
              "      <td>No</td>\n",
              "      <td>Development, Other Use Cases</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7f1bd965-44ce-43c6-a2f0-42ef1a82a0a9')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-7f1bd965-44ce-43c6-a2f0-42ef1a82a0a9 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-7f1bd965-44ce-43c6-a2f0-42ef1a82a0a9');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "q16_question = questions[16]\n",
        "q16_instructions = \"\"\"I received answers from an MLOps survey about Large Language Models (LLMs).\n",
        "Your task is to extract and summarize the problems in using LLMs with keywords.\n",
        "If no problems are mentioned, you should use an empty string as a placeholder.\n",
        "You should output these information in the JSON lines format.\n",
        "You should generate one JSON line per answer.\n",
        "Here's an example of an answer:\n",
        "- Too heavy\n",
        "- Expensive and the business use case was poorly defined. Just a shiny toy\n",
        "- We got terrible suggestions from the LLM that were false information and would be detrimental to the business\n",
        "Here's an example of the expected output:\n",
        "{\"problems\": \"Model Size\"}\n",
        "{\"problems\": \"Cost, Use Case\"}\n",
        "{\"problems\": \"Hallucinations, Business RIsks\"}\n",
        "Answers:\"\"\"\n",
        "q16_inputs = results.iloc[:, 16]\n",
        "print(f'Question 16:', q16_question)\n",
        "q16_reviews = associate(model, q16_instructions, q16_inputs)\n",
        "q16_reviews.dropna()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 687
        },
        "id": "ZDAGZSSnxHDd",
        "outputId": "d881f55d-75eb-4af9-a6ad-b9c4d36f9c51"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question 16: If yes, why did it not work out?\n",
            "- Inputs: 110\n",
            "- batch: 0 -> 15\n",
            "- tokens: 500\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                                                                     If yes, why did it not work out?  \\\n",
              "1                                                                                                                                                                                                        Date quality   \n",
              "6   So far it's not reliable enough and we don't have spent enough time preparing special content for it. Like rules it should use during diagnosis. The data ChatGPT was trained on seems a bit hit and miss for us.   \n",
              "8                                                                                                                                                                                                              No ROI   \n",
              "16                                                                                                      We got terrible suggestions from the LLM that were false information and would be detrimental to the business   \n",
              "17                                                        We evaluated a LLM (GPT-3) against RoBERTa fine-tuned on our small task-specific dataset, and our fine-tuned model got much higher accuracy than GPT-3 did.   \n",
              "27                                                                                                                                                                        getting fine tune right and LLMops pipeline   \n",
              "40                                                                                                                                                                                                still working on it   \n",
              "43                                                                                                                                                                                 they are in prod for that use case   \n",
              "47                                                                                                                                                                                                                 na   \n",
              "75                                                                                                                                                                                                          Too heavy   \n",
              "77                                                                                                                                                                                              We’re being thorough.   \n",
              "78                                                                                                                                              Concerns about sticking to a domain and about veracity if the answers   \n",
              "80                                                                                                                                                                                                                 No   \n",
              "85                                                                                                                                                                                                                  .   \n",
              "97                                                                                                                                           Expensive and the business use case was poorly defined. Just a shiny toy   \n",
              "\n",
              "                                          problems  \n",
              "1                                     Data Quality  \n",
              "6   Reliability, Data Quality, ROI, Business Risks  \n",
              "8            Accuracy, Fine-tuning, MLOps Pipeline  \n",
              "16                    Domain-specificity, Veracity  \n",
              "17                                  Cost, Use Case  \n",
              "27                                      Model Size  \n",
              "40                    Domain-specificity, Veracity  \n",
              "43                                            None  \n",
              "47                                            None  \n",
              "75                                      Model Size  \n",
              "77                                    Thoroughness  \n",
              "78                    Domain-specificity, Veracity  \n",
              "80                                            None  \n",
              "85                                            None  \n",
              "97                                                  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-6a91a9b4-7ba1-4488-a7e5-c23a1ead92b6\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>If yes, why did it not work out?</th>\n",
              "      <th>problems</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Date quality</td>\n",
              "      <td>Data Quality</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>So far it's not reliable enough and we don't have spent enough time preparing special content for it. Like rules it should use during diagnosis. The data ChatGPT was trained on seems a bit hit and miss for us.</td>\n",
              "      <td>Reliability, Data Quality, ROI, Business Risks</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>No ROI</td>\n",
              "      <td>Accuracy, Fine-tuning, MLOps Pipeline</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>We got terrible suggestions from the LLM that were false information and would be detrimental to the business</td>\n",
              "      <td>Domain-specificity, Veracity</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>We evaluated a LLM (GPT-3) against RoBERTa fine-tuned on our small task-specific dataset, and our fine-tuned model got much higher accuracy than GPT-3 did.</td>\n",
              "      <td>Cost, Use Case</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>getting fine tune right and LLMops pipeline</td>\n",
              "      <td>Model Size</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>still working on it</td>\n",
              "      <td>Domain-specificity, Veracity</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43</th>\n",
              "      <td>they are in prod for that use case</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>47</th>\n",
              "      <td>na</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75</th>\n",
              "      <td>Too heavy</td>\n",
              "      <td>Model Size</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>77</th>\n",
              "      <td>We’re being thorough.</td>\n",
              "      <td>Thoroughness</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>78</th>\n",
              "      <td>Concerns about sticking to a domain and about veracity if the answers</td>\n",
              "      <td>Domain-specificity, Veracity</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>80</th>\n",
              "      <td>No</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>85</th>\n",
              "      <td>.</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>97</th>\n",
              "      <td>Expensive and the business use case was poorly defined. Just a shiny toy</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6a91a9b4-7ba1-4488-a7e5-c23a1ead92b6')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-6a91a9b4-7ba1-4488-a7e5-c23a1ead92b6 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-6a91a9b4-7ba1-4488-a7e5-c23a1ead92b6');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# REVIEWS"
      ],
      "metadata": {
        "id": "UCzYO1n30E-i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "reviews = pd.concat([\n",
        "    q0_reviews,\n",
        "    q1_reviews,\n",
        "    q2_reviews,\n",
        "    q3_reviews,\n",
        "    q4_reviews,\n",
        "    q5_reviews,\n",
        "    q6_reviews,\n",
        "    q7_reviews,\n",
        "    q8_reviews,\n",
        "    q9_reviews,\n",
        "    q10_reviews,\n",
        "    q11_reviews,\n",
        "    q12_reviews,\n",
        "    q13_reviews,\n",
        "    q14_reviews,\n",
        "    q15_reviews,\n",
        "    q16_reviews,\n",
        "], axis='columns')\n",
        "print('Shape:', reviews.shape)\n",
        "reviews.head(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "K2sNEgeh0Esh",
        "outputId": "2c28b3b5-f8e9-4a1e-ff4c-59ec4424ae81"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Shape: (110, 37)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  What is your position/title at your company?                      title  \\\n",
              "0                                          NaN                        NaN   \n",
              "1                               Data scientist             Data Scientist   \n",
              "2                                  ML Engineer  Machine Learning Engineer   \n",
              "3                                      Founder                    Founder   \n",
              "4                                        Owner                      Owner   \n",
              "5                   Senior Principal Scientist        Principal Scientist   \n",
              "6                     Head of Machine Learning   Head of Machine Learning   \n",
              "7             Senior Machine Learning Engineer  Machine Learning Engineer   \n",
              "8                              Head of Product                              \n",
              "9                                          NaN                        NaN   \n",
              "\n",
              "  position    other How big is your organization? (number of employees)  \\\n",
              "0      NaN      NaN                                                 NaN   \n",
              "1                                                                1,000+   \n",
              "2                                                                50-500   \n",
              "3                                                                  1-10   \n",
              "4                                                                  1-10   \n",
              "5   Senior                                                       1,000+   \n",
              "6                                                                50-500   \n",
              "7   Senior                                                       1,000+   \n",
              "8     Head  Product                                               10-50   \n",
              "9      NaN      NaN                                              50-500   \n",
              "\n",
              "  Are you using LLM at in your organization?  \\\n",
              "0                                        NaN   \n",
              "1                                         No   \n",
              "2                                        Yes   \n",
              "3                                        Yes   \n",
              "4                                        Yes   \n",
              "5                                        Yes   \n",
              "6                                         No   \n",
              "7                                        Yes   \n",
              "8                                         No   \n",
              "9                                        Yes   \n",
              "\n",
              "         What is your use case/use cases?            fields  \\\n",
              "0                                     NaN               NaN   \n",
              "1                                     NaN               NaN   \n",
              "2                      Text summarisation                     \n",
              "3  Data annotation; summarization; search                     \n",
              "4    Content production and brainstorming  Content Creation   \n",
              "5                              Healthcare        Healthcare   \n",
              "6                                     NaN               NaN   \n",
              "7             Search, classification, NER                     \n",
              "8                                     NaN               NaN   \n",
              "9                                     NaN               NaN   \n",
              "\n",
              "                                                   tasks  \\\n",
              "0                                                    NaN   \n",
              "1                                                    NaN   \n",
              "2                                     Text Summarization   \n",
              "3            Data Annotation, Text Summarization, Search   \n",
              "4                                                          \n",
              "5                                                          \n",
              "6                                                    NaN   \n",
              "7  Search, Text Classification, Named Entity Recognition   \n",
              "8                                                    NaN   \n",
              "9                                                    NaN   \n",
              "\n",
              "  Have you integrated or built any internal tools to support LLMs in your org? If so what?  \\\n",
              "0                                                                                      NaN   \n",
              "1                                                                                      NaN   \n",
              "2                                                                                       No   \n",
              "3                                    https://chat.mantisnlp.com, internal annotation tools   \n",
              "4                                                                                  Not yet   \n",
              "5                                                                        yes, confidential   \n",
              "6                                                                                      NaN   \n",
              "7                         Just a bunch of shitty scripts I need to refactor in the future.   \n",
              "8                                                                                      NaN   \n",
              "9                                                                                       oi   \n",
              "\n",
              "  integrated                             solutions purposes  \\\n",
              "0        NaN                                   NaN      NaN   \n",
              "1        NaN                                   NaN      NaN   \n",
              "2         No                                                  \n",
              "3         No  MantisNLP, internal annotation tools            \n",
              "4         No                                                  \n",
              "5        Yes                          Confidential            \n",
              "6        NaN                                   NaN      NaN   \n",
              "7         No                Refactoring of scripts            \n",
              "8        NaN                                   NaN      NaN   \n",
              "9         No                                    OI            \n",
              "\n",
              "                     What are some of the main challenges you have encountered thus far when building with LLMs  \\\n",
              "0                                                                                                           NaN   \n",
              "1                                                                                                           NaN   \n",
              "2  Current infrastructure not designed for such massive models, having to implement workarounds for quick fixes   \n",
              "3                         out of date info, hallucinations, cost, difficulty in deploying on own infrastructure   \n",
              "4                                                                                                       not yet   \n",
              "5                                                                                                   reliability   \n",
              "6                                                                                                           NaN   \n",
              "7                 Size in all regards. Also training time, response time and that multi gpu debugging is weird.   \n",
              "8                                                                                                           NaN   \n",
              "9                                                                                                           lik   \n",
              "\n",
              "                                      challenges  \\\n",
              "0                                            NaN   \n",
              "1                                            NaN   \n",
              "2                     Infrastructure,Workarounds   \n",
              "3  Accuracy,Outdated Information,Cost,Deployment   \n",
              "4                                                  \n",
              "5                                    Reliability   \n",
              "6                                            NaN   \n",
              "7          Size,Training,Response Time,Debugging   \n",
              "8                                            NaN   \n",
              "9                                                  \n",
              "\n",
              "                                                                              What are your main concerns with using LLMs in production?  \\\n",
              "0                                                                                                                                    NaN   \n",
              "1                                                                                                                                    NaN   \n",
              "2                                                                                                                                   Cost   \n",
              "3                                                                                   they could serve nonsense or out of date information   \n",
              "4                                                                                                                     cost of openai API   \n",
              "5                                                                                                                     cost of openai API   \n",
              "6                                                                                                                                    NaN   \n",
              "7  Whether the cost involved for us justifies the gains we get, considering we mainly make money from ads and not a really good product.   \n",
              "8                                                                                                                                    NaN   \n",
              "9                                                                                                                                     il   \n",
              "\n",
              "               concerns         how are you using LLMs?  \\\n",
              "0                   NaN                             NaN   \n",
              "1                   NaN                             NaN   \n",
              "2                  Cost  Open source model (GPT-J, etc)   \n",
              "3    Accuracy, Currency                     Open AI API   \n",
              "4                  Cost                     Open AI API   \n",
              "5                  Cost                  In house model   \n",
              "6                   NaN                             NaN   \n",
              "7  Cost, Business Value        Other model provider API   \n",
              "8                   NaN                             NaN   \n",
              "9                                           Open AI API   \n",
              "\n",
              "                      What tools are you using with LLMs?  \\\n",
              "0                                                     NaN   \n",
              "1                                                     NaN   \n",
              "2                                                  Seldon   \n",
              "3                                                     NaN   \n",
              "4  Looking at Pinecone, weaviate, langchain, hugging face   \n",
              "5                                                     NaN   \n",
              "6                                                     NaN   \n",
              "7         Hugginface, sagemaker, openapi api, tensorflow.   \n",
              "8                                                     NaN   \n",
              "9                                                     loi   \n",
              "\n",
              "                                          tools  \\\n",
              "0                                           NaN   \n",
              "1                                           NaN   \n",
              "2                                        Seldon   \n",
              "3                                           NaN   \n",
              "4   Pinecone, Weaviate, LangChain, Hugging Face   \n",
              "5                                           NaN   \n",
              "6                                           NaN   \n",
              "7  Hugging Face, SageMaker, OpenAPI, TensorFlow   \n",
              "8                                           NaN   \n",
              "9                                           loi   \n",
              "\n",
              "  What areas are you most interested in learning more about?       topics  \\\n",
              "0                                                        NaN          NaN   \n",
              "1                                                        NaN          NaN   \n",
              "2                                        Inferenece for LLMs    Inference   \n",
              "3                                      Working with OSS LLMs  Open Source   \n",
              "4                                                 Embeddings   Embeddings   \n",
              "5                                      Working with OSS LLMs  Open Source   \n",
              "6                                                        NaN          NaN   \n",
              "7                                      Working with OSS LLMs  Open Source   \n",
              "8                                                        NaN          NaN   \n",
              "9                                      Working with OSS LLMs  Open Source   \n",
              "\n",
              "                                                    How do you deal with reliability of output?  \\\n",
              "0                                                                                           NaN   \n",
              "1                                                                                           NaN   \n",
              "2                                                                                           NaN   \n",
              "3                           we provide links to the context from which answers have been drawn.   \n",
              "4                                                                               temperature = 0   \n",
              "5                                                                                           NaN   \n",
              "6                                                                                           NaN   \n",
              "7  Ill refer to the picture of the xkcd guy \"just stir the model until the numbers look right\".   \n",
              "8                                                                                           NaN   \n",
              "9                                                                                            lo   \n",
              "\n",
              "            approaches  \\\n",
              "0                  NaN   \n",
              "1                  NaN   \n",
              "2                  NaN   \n",
              "3              Nothing   \n",
              "4  Temperature Control   \n",
              "5                  NaN   \n",
              "6                  NaN   \n",
              "7        Manual Tuning   \n",
              "8                  NaN   \n",
              "9              Nothing   \n",
              "\n",
              "                                                                                                                                                                                                                     Any stories you have that are worth sharing about working with LLMs?  \\\n",
              "0                                                                                                                                                                                                                                                                                     NaN   \n",
              "1                                                                                                                                                                                                                                                                                     NaN   \n",
              "2                                                                                                                                                                                                                                                                                     NaN   \n",
              "3                                                                                                                                                                                                                                                                                     NaN   \n",
              "4                                                                                                                                                                                                                                                                                     NaN   \n",
              "5                                                                                                                                                                                                                                                                                     NaN   \n",
              "6                                                                                                                                                                                                                                                                                     NaN   \n",
              "7  I am not sure what ranks as a LLM. Anything bert and beyond? Anyway, we once used setfit for a production task but found that performance was really tricky to debug. The proof of concept was fine. Production was a complete mess. To this day I dont know why- underlying hardware?   \n",
              "8                                                                                                                                                                                                                                                                                     NaN   \n",
              "9                                                                                                                                                                                                                                                                                      lo   \n",
              "\n",
              "                       stories  \\\n",
              "0                          NaN   \n",
              "1                          NaN   \n",
              "2                          NaN   \n",
              "3                          NaN   \n",
              "4                          NaN   \n",
              "5                          NaN   \n",
              "6                          NaN   \n",
              "7  Production issues with LLMs   \n",
              "8                          NaN   \n",
              "9                                \n",
              "\n",
              "                                                                                                                                                                      Any questions you have for the community about LLM in production?  \\\n",
              "0                                                                                                                                                                                                                                   NaN   \n",
              "1                                                                                                                                                                                                                                   NaN   \n",
              "2                                                                                                                                                                                                                                   NaN   \n",
              "3                                                                                                                                                                                                                                   NaN   \n",
              "4                                                                                                                                                                                                                                   NaN   \n",
              "5                                                                                                                                                                                                                                   NaN   \n",
              "6                                                                                                                                                                                                                                   NaN   \n",
              "7  I mainly wonder how everyone is monitoring these. ALSO. For my company GDPR is a big deal. My DPO shoots down our ideas more often than not or limits it to using the last three months of data. How do other people deal with this?   \n",
              "8                                                                                                                                                                                                                                   NaN   \n",
              "9                                                                                                                                                                                                                                    lo   \n",
              "\n",
              "                                                          questions  \\\n",
              "0                                                               NaN   \n",
              "1                                                               NaN   \n",
              "2                                                               NaN   \n",
              "3                                                               NaN   \n",
              "4                                                               NaN   \n",
              "5                                                               NaN   \n",
              "6                                                               NaN   \n",
              "7  How do other people monitor LLMs and deal with GDPR limitations?   \n",
              "8                                                               NaN   \n",
              "9                                                                     \n",
              "\n",
              "                                                                   What is the main reason for not using LLMs in your org?  \\\n",
              "0                                                                                                                      NaN   \n",
              "1                                                                                              Production takes investment   \n",
              "2                                                                                                                      NaN   \n",
              "3                                                                                                                      NaN   \n",
              "4                                                                                                                      NaN   \n",
              "5                                                                                                                      NaN   \n",
              "6  It's only just now becoming feasible with the ChatGPT API. Before it was too expensive and quality was not good enough.   \n",
              "7                                                                                                                      NaN   \n",
              "8                                                                                                          No applications   \n",
              "9                                                                                                                      NaN   \n",
              "\n",
              "            reasons  \\\n",
              "0               NaN   \n",
              "1   Production Cost   \n",
              "2               NaN   \n",
              "3               NaN   \n",
              "4               NaN   \n",
              "5               NaN   \n",
              "6     Cost, Quality   \n",
              "7               NaN   \n",
              "8  Lack of Use Case   \n",
              "9               NaN   \n",
              "\n",
              "  What are some key questions you face when it comes to using LLM in prod?  \\\n",
              "0                                                                      NaN   \n",
              "1                                             Cost to maintain the service   \n",
              "2                                                                      NaN   \n",
              "3                                                                      NaN   \n",
              "4                                                                      NaN   \n",
              "5                                                                      NaN   \n",
              "6                                               How to make them reliable.   \n",
              "7                                                                      NaN   \n",
              "8                                             ROI of more expensive models   \n",
              "9                                                                      NaN   \n",
              "\n",
              "            challenges  \\\n",
              "0                  NaN   \n",
              "1  Cost of Maintenance   \n",
              "2                  NaN   \n",
              "3                  NaN   \n",
              "4                  NaN   \n",
              "5                  NaN   \n",
              "6          Reliability   \n",
              "7                  NaN   \n",
              "8                  ROI   \n",
              "9                  NaN   \n",
              "\n",
              "                 have you tried LLMs for different use cases in your org?  \\\n",
              "0                                                                     NaN   \n",
              "1                                                                     Yes   \n",
              "2                                                                     NaN   \n",
              "3                                                                     NaN   \n",
              "4                                                                     NaN   \n",
              "5                                                                     NaN   \n",
              "6  We are still exploring, eg for support in diagnosis of plant diseases.   \n",
              "7                                                                     NaN   \n",
              "8                                                                     Yes   \n",
              "9                                                                     NaN   \n",
              "\n",
              "  tried          cases  \\\n",
              "0   NaN            NaN   \n",
              "1   Yes                  \n",
              "2   NaN            NaN   \n",
              "3   NaN            NaN   \n",
              "4   NaN            NaN   \n",
              "5   NaN            NaN   \n",
              "6    No  Plan Diseases   \n",
              "7   NaN            NaN   \n",
              "8   Yes                  \n",
              "9   NaN            NaN   \n",
              "\n",
              "                                                                                                                                                                                    If yes, why did it not work out?  \\\n",
              "0                                                                                                                                                                                                                NaN   \n",
              "1                                                                                                                                                                                                       Date quality   \n",
              "2                                                                                                                                                                                                                NaN   \n",
              "3                                                                                                                                                                                                                NaN   \n",
              "4                                                                                                                                                                                                                NaN   \n",
              "5                                                                                                                                                                                                                NaN   \n",
              "6  So far it's not reliable enough and we don't have spent enough time preparing special content for it. Like rules it should use during diagnosis. The data ChatGPT was trained on seems a bit hit and miss for us.   \n",
              "7                                                                                                                                                                                                                NaN   \n",
              "8                                                                                                                                                                                                             No ROI   \n",
              "9                                                                                                                                                                                                                NaN   \n",
              "\n",
              "                                         problems  \n",
              "0                                             NaN  \n",
              "1                                    Data Quality  \n",
              "2                                             NaN  \n",
              "3                                             NaN  \n",
              "4                                             NaN  \n",
              "5                                             NaN  \n",
              "6  Reliability, Data Quality, ROI, Business Risks  \n",
              "7                                             NaN  \n",
              "8           Accuracy, Fine-tuning, MLOps Pipeline  \n",
              "9                                             NaN  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-8d5316c7-c627-45d0-9a9e-c6461ab472eb\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>What is your position/title at your company?</th>\n",
              "      <th>title</th>\n",
              "      <th>position</th>\n",
              "      <th>other</th>\n",
              "      <th>How big is your organization? (number of employees)</th>\n",
              "      <th>Are you using LLM at in your organization?</th>\n",
              "      <th>What is your use case/use cases?</th>\n",
              "      <th>fields</th>\n",
              "      <th>tasks</th>\n",
              "      <th>Have you integrated or built any internal tools to support LLMs in your org? If so what?</th>\n",
              "      <th>integrated</th>\n",
              "      <th>solutions</th>\n",
              "      <th>purposes</th>\n",
              "      <th>What are some of the main challenges you have encountered thus far when building with LLMs</th>\n",
              "      <th>challenges</th>\n",
              "      <th>What are your main concerns with using LLMs in production?</th>\n",
              "      <th>concerns</th>\n",
              "      <th>how are you using LLMs?</th>\n",
              "      <th>What tools are you using with LLMs?</th>\n",
              "      <th>tools</th>\n",
              "      <th>What areas are you most interested in learning more about?</th>\n",
              "      <th>topics</th>\n",
              "      <th>How do you deal with reliability of output?</th>\n",
              "      <th>approaches</th>\n",
              "      <th>Any stories you have that are worth sharing about working with LLMs?</th>\n",
              "      <th>stories</th>\n",
              "      <th>Any questions you have for the community about LLM in production?</th>\n",
              "      <th>questions</th>\n",
              "      <th>What is the main reason for not using LLMs in your org?</th>\n",
              "      <th>reasons</th>\n",
              "      <th>What are some key questions you face when it comes to using LLM in prod?</th>\n",
              "      <th>challenges</th>\n",
              "      <th>have you tried LLMs for different use cases in your org?</th>\n",
              "      <th>tried</th>\n",
              "      <th>cases</th>\n",
              "      <th>If yes, why did it not work out?</th>\n",
              "      <th>problems</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Data scientist</td>\n",
              "      <td>Data Scientist</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>1,000+</td>\n",
              "      <td>No</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Production takes investment</td>\n",
              "      <td>Production Cost</td>\n",
              "      <td>Cost to maintain the service</td>\n",
              "      <td>Cost of Maintenance</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Yes</td>\n",
              "      <td></td>\n",
              "      <td>Date quality</td>\n",
              "      <td>Data Quality</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ML Engineer</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>50-500</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Text summarisation</td>\n",
              "      <td></td>\n",
              "      <td>Text Summarization</td>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Current infrastructure not designed for such massive models, having to implement workarounds for quick fixes</td>\n",
              "      <td>Infrastructure,Workarounds</td>\n",
              "      <td>Cost</td>\n",
              "      <td>Cost</td>\n",
              "      <td>Open source model (GPT-J, etc)</td>\n",
              "      <td>Seldon</td>\n",
              "      <td>Seldon</td>\n",
              "      <td>Inferenece for LLMs</td>\n",
              "      <td>Inference</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Founder</td>\n",
              "      <td>Founder</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>1-10</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Data annotation; summarization; search</td>\n",
              "      <td></td>\n",
              "      <td>Data Annotation, Text Summarization, Search</td>\n",
              "      <td>https://chat.mantisnlp.com, internal annotation tools</td>\n",
              "      <td>No</td>\n",
              "      <td>MantisNLP, internal annotation tools</td>\n",
              "      <td></td>\n",
              "      <td>out of date info, hallucinations, cost, difficulty in deploying on own infrastructure</td>\n",
              "      <td>Accuracy,Outdated Information,Cost,Deployment</td>\n",
              "      <td>they could serve nonsense or out of date information</td>\n",
              "      <td>Accuracy, Currency</td>\n",
              "      <td>Open AI API</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>Open Source</td>\n",
              "      <td>we provide links to the context from which answers have been drawn.</td>\n",
              "      <td>Nothing</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Owner</td>\n",
              "      <td>Owner</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>1-10</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Content production and brainstorming</td>\n",
              "      <td>Content Creation</td>\n",
              "      <td></td>\n",
              "      <td>Not yet</td>\n",
              "      <td>No</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>not yet</td>\n",
              "      <td></td>\n",
              "      <td>cost of openai API</td>\n",
              "      <td>Cost</td>\n",
              "      <td>Open AI API</td>\n",
              "      <td>Looking at Pinecone, weaviate, langchain, hugging face</td>\n",
              "      <td>Pinecone, Weaviate, LangChain, Hugging Face</td>\n",
              "      <td>Embeddings</td>\n",
              "      <td>Embeddings</td>\n",
              "      <td>temperature = 0</td>\n",
              "      <td>Temperature Control</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>Senior Principal Scientist</td>\n",
              "      <td>Principal Scientist</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "      <td>1,000+</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Healthcare</td>\n",
              "      <td>Healthcare</td>\n",
              "      <td></td>\n",
              "      <td>yes, confidential</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Confidential</td>\n",
              "      <td></td>\n",
              "      <td>reliability</td>\n",
              "      <td>Reliability</td>\n",
              "      <td>cost of openai API</td>\n",
              "      <td>Cost</td>\n",
              "      <td>In house model</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>Open Source</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>Head of Machine Learning</td>\n",
              "      <td>Head of Machine Learning</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>50-500</td>\n",
              "      <td>No</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>It's only just now becoming feasible with the ChatGPT API. Before it was too expensive and quality was not good enough.</td>\n",
              "      <td>Cost, Quality</td>\n",
              "      <td>How to make them reliable.</td>\n",
              "      <td>Reliability</td>\n",
              "      <td>We are still exploring, eg for support in diagnosis of plant diseases.</td>\n",
              "      <td>No</td>\n",
              "      <td>Plan Diseases</td>\n",
              "      <td>So far it's not reliable enough and we don't have spent enough time preparing special content for it. Like rules it should use during diagnosis. The data ChatGPT was trained on seems a bit hit and miss for us.</td>\n",
              "      <td>Reliability, Data Quality, ROI, Business Risks</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Senior Machine Learning Engineer</td>\n",
              "      <td>Machine Learning Engineer</td>\n",
              "      <td>Senior</td>\n",
              "      <td></td>\n",
              "      <td>1,000+</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Search, classification, NER</td>\n",
              "      <td></td>\n",
              "      <td>Search, Text Classification, Named Entity Recognition</td>\n",
              "      <td>Just a bunch of shitty scripts I need to refactor in the future.</td>\n",
              "      <td>No</td>\n",
              "      <td>Refactoring of scripts</td>\n",
              "      <td></td>\n",
              "      <td>Size in all regards. Also training time, response time and that multi gpu debugging is weird.</td>\n",
              "      <td>Size,Training,Response Time,Debugging</td>\n",
              "      <td>Whether the cost involved for us justifies the gains we get, considering we mainly make money from ads and not a really good product.</td>\n",
              "      <td>Cost, Business Value</td>\n",
              "      <td>Other model provider API</td>\n",
              "      <td>Hugginface, sagemaker, openapi api, tensorflow.</td>\n",
              "      <td>Hugging Face, SageMaker, OpenAPI, TensorFlow</td>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>Open Source</td>\n",
              "      <td>Ill refer to the picture of the xkcd guy \"just stir the model until the numbers look right\".</td>\n",
              "      <td>Manual Tuning</td>\n",
              "      <td>I am not sure what ranks as a LLM. Anything bert and beyond? Anyway, we once used setfit for a production task but found that performance was really tricky to debug. The proof of concept was fine. Production was a complete mess. To this day I dont know why- underlying hardware?</td>\n",
              "      <td>Production issues with LLMs</td>\n",
              "      <td>I mainly wonder how everyone is monitoring these. ALSO. For my company GDPR is a big deal. My DPO shoots down our ideas more often than not or limits it to using the last three months of data. How do other people deal with this?</td>\n",
              "      <td>How do other people monitor LLMs and deal with GDPR limitations?</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>Head of Product</td>\n",
              "      <td></td>\n",
              "      <td>Head</td>\n",
              "      <td>Product</td>\n",
              "      <td>10-50</td>\n",
              "      <td>No</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>No applications</td>\n",
              "      <td>Lack of Use Case</td>\n",
              "      <td>ROI of more expensive models</td>\n",
              "      <td>ROI</td>\n",
              "      <td>Yes</td>\n",
              "      <td>Yes</td>\n",
              "      <td></td>\n",
              "      <td>No ROI</td>\n",
              "      <td>Accuracy, Fine-tuning, MLOps Pipeline</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>50-500</td>\n",
              "      <td>Yes</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>oi</td>\n",
              "      <td>No</td>\n",
              "      <td>OI</td>\n",
              "      <td></td>\n",
              "      <td>lik</td>\n",
              "      <td></td>\n",
              "      <td>il</td>\n",
              "      <td></td>\n",
              "      <td>Open AI API</td>\n",
              "      <td>loi</td>\n",
              "      <td>loi</td>\n",
              "      <td>Working with OSS LLMs</td>\n",
              "      <td>Open Source</td>\n",
              "      <td>lo</td>\n",
              "      <td>Nothing</td>\n",
              "      <td>lo</td>\n",
              "      <td></td>\n",
              "      <td>lo</td>\n",
              "      <td></td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8d5316c7-c627-45d0-9a9e-c6461ab472eb')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-8d5316c7-c627-45d0-9a9e-c6461ab472eb button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-8d5316c7-c627-45d0-9a9e-c6461ab472eb');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "timestamp = str(datetime.now())\n",
        "csv_path = f'reviews_{timestamp}.csv'\n",
        "xlsx_path = f'reviews_{timestamp}.xlsx'\n",
        "reviews.to_csv(csv_path, quoting=csv.QUOTE_ALL)\n",
        "reviews.to_excel(xlsx_path) # easier to import"
      ],
      "metadata": {
        "id": "lKu4DJEtytwq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# files.download(csv_path) \n",
        "# files.download(xlsx_path) "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "a3Ucss5_0j-J",
        "outputId": "ea897028-c568-453b-bcf7-b8478faf2611"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_09890950-f4a4-4940-80e3-fe8b8f8becac\", \"reviews_2023-05-07 06:31:13.685946.csv\", 59473)"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_cef92bca-1618-4576-9a95-62cf27ad6c67\", \"reviews_2023-05-07 06:31:13.685946.xlsx\", 38645)"
            ]
          },
          "metadata": {}
        }
      ]
    }
  ]
}