AI Dungeon 2 implementation questions / discussion

[da-source]

I've stumbled upon a blog by post by one of the creators of AI Dungeon, in which he explains how they have used Cortex for their scalable model serving (https://medium.com/@aidungeon/how-we-scaled-ai-dungeon-2-to-support-over-1-000-000-users-d207d5623de9). This inspired me to try to deploy my own large finetuned GPT-2 models using Cortex but I've been facing difficulties implementing this.

AI Dungeon displays GPT2 outputs in realtime, something I've had the most difficulty implementing so far, since Cortex doesn't yet support websockets to stream the output. How have the creators manage to make AI Dungeon so interactive using Cortex and is it possible to replicate those results?

[deliahu]

@da-source The most straightforward way to accomplish this with Cortex is to generate only one word (or a small number of words) at a time, and to make multiple prediction requests.

For example, if you're generating text and the initial input is "word1, word2, word3", you could send a request with "word1, word2, word3" to generate a single word. When the response comes back with "word4", you can show that to the user, and make another request to your model with "word1, word2, word3, word4", repeating this process one word (or a few words) at a time. Would that work for you / your model?

[da-source]

@da-source The most straightforward way to accomplish this with Cortex is to generate only one word (or a small number of words) at a time, and to make multiple prediction requests.

For example, if you're generating text and the initial input is "word1, word2, word3", you could send a request with "word1, word2, word3" to generate a single word. When the response comes back with "word4", you can show that to the user, and make another request to your model with "word1, word2, word3, word4", repeating this process one word (or a few words) at a time. Would that work for you / your model?

I’ll try that. Meanwhile how can I configure my cluster to be spot-instance only? I can’t find cluster.yaml file mentioned in documentation.

[deliahu]

@da-source here is the documentation for cluster configuration, and here is the documentation for the spot_config section. Keep in mind that if you set on_demand_backup to false, you won't be guaranteed to get an instance from AWS, since spot instances are not guaranteed to always be available.

[deliahu]

@da-source I'm closing this issue due to inactivity, feel free to reach out if you have additional questions.