Notice: Following responsible disclosure procedures, the issues here have been brought up to the developers and have been fixed before this report was published. Similarly, all private data collected has been deleted.
On April 18th, I discovered a vulnerability in the AI Dungeon GraphQL API that allowed unpublished adventures, unpublished scenarios, and unpublished posts to be leaked. These resources could be read in bulk, at a rate of approximately 1000 requests per minute. Unfortunately, this is, in fact, the second time I have discovered this exact vulnerability. The first time, the issue was reported and fixed, but after finding it again, I can see that simply reporting the issue was a mistake.
Rather, I am using this report as leverage. By making not only the devs, but the users aware of these critical issues, it will hopefully incentivize the AI Dungeon team to ensure that vulnerabilities like these never see the light of day. By leading by example with this report, I hope that others in the community will have something to emulate going forward, and that this sort of collaboration with the community will result in a much more secure product for all. I truly believe this to be the case because, thanks to this report, things are already changing for the better! (#Security Improvements Resulting From This Report)
Nevertheless, the issue still stands that the vulnerability did, in fact, exist. Initially, this report had some anonymized, aggregated data to help demonstrate the severity of this vulnerability. The data was meant for the community, and showed just how sensitive adventure data actually is. It was also meant to prompt an investigation into breaches of user data on Latitude's servers, which I do not believe has been conducted in the past. This is because, between the game's release and June 15th 2020, adventures and scenarios quite simply had... zero api security. Anyone could read anything, yet no announcements regarding data breaches were made. As you can see, security issues have been around for a while. And as I found out, between then and now, things haven't gotten much better.
Now that this event is catching wider media attention, it seems that people are misinterpreting the point of this disclosure, presumably due to a lack of context. I might reinstate the data included in this report, but only if I have time to rewrite it with more nuance to help others outside the loop understand what this is about. Until that time, I will keep this disclosure to its current technical format.
All non-deleted resources, created from Dec 16th, 2019 onwards, were affected by this vulnerability.
The following goes into the technical details of the vulnerability itself. At a high level, the upvote api endpoint takes in an id and a resource type. The vulnerability consists of calling the upvote endpoint with ids identifying private resources, and getting full private resources back through a quirk of how graphql functions.
Here is the endpoint being called (all code snippets here were generated from parsing the introspection results - hence why you should disable introspection in production...!):
Votable voteContent(input:ContentResponseInput)As you can see, the return type of voteContent is Votable. Here is its definition:
Interface Votable implemented by Adventure, Comment, Post, Scenario {
id: ID
publicId: String
totalUpvotes: Int
userVote: String
}At this point, you might already be seeing how this comes together. The voteContent mutation returns a Votable, which is an interface implemented by Adventures, Comments, Posts, and Scenarios. And, depending on the ContentResponseInput, that's exactly what you get back - one of those four resource types.
Now, let's take a look at the ContentResponseInput object:
Input_object ContentResponseInput {
contentId: String
contentType: String
data: JSONObject
id: String
responseType: String
responseValue: String
}ContentResponseInput is entirely generic, which is great for code reuse, but makes it so that all the resource types are affected by the same vulnerability!
So, how exactly is this endpoint a vulnerability?
Well, consider the following ContentResponseInput:
{
"variables": {
"input": {
"contentId": "1",
"contentType": "adventure",
"responseType": "vote",
"responseValue": "novote"
}
},
"query": "mutation ($input:ContentResponseInput) { voteContent(input:$input) { __typename } }"
}The mutation above actually returned a response from the AI Dungeon API. Hence why I say that all adventures were at risk. From adventure id #1 all the way to adventure id #72500000. Still - you might be asking - if the response type is a Votable, how are adventures being leaked? The answer is because Votable is actually an interface. A real concrete object type is powering the response. And, in this case, it would be an Adventure, as printed by __typename. This means that any fields of an Adventure (aka: all of them!) - could be queried for in the response.
Normally, however, GraphQL disallows this - only Votable interface fields can be queried for. So how were adventure fields fetched? GraphQL has the concept of fragments.
Take a look at the following:
Object Adventure {
actions: [Action]
id: ID
user: User
title: String
// other fields omitted for clarity
}Calling ... voteContent(input:$input) { actions } returns an error - actions is not a field of the Votable interface. However, by defining the following:
fragment MyVulnerability on Adventure { actions { text } id user { username } title }...and including it:
mutation($input:ContentResponseInput) { voteContent(input:$input) { ...MyVulnerability } }voteContent will return all fields in the fragment. This is perhaps a strange quirk of GraphQL, but nevertheless is a feature, and very important to take note of. I don't know if this functionality has an exact name, but I propose calling this 'downcasting', as you're essentially performing an explicit downcast on an interface into a more specific type, at which point you're able to query the downcasted fields.
Ironically, however, even if this quirk didn't exist, I would still consider the voteContent endpoint a security vulnerability. It allows a user to associate a public uuid with an autoincrementing id, and is callable on unpublished resources, which greatly increases the attack surface for those resources.
Now you know why voting is disabled on the site!
Interestingly, this vulnerability was actually present in multiple endpoints.
// here's our good friend ContentResponseInput again
Savable saveContent(input:ContentResponseInput)
Interface Savable implemented by Adventure, Post, Scenario {
id: ID
isSaved: Boolean
publicId: String
}Saving of certain resources is currently disabled because of this vulnerability.
// A different vulnerability, still using ContentResponseInput
ContentResponse createContentResponse(input:ContentResponseInput)
Object ContentResponse{
commentText: String
contentId: String
contentType: String
createdAt: DateTime
data: JSONObject
id: ID
responseType: String
responseValue: String
user: User
userId: String
username: String
}This generic endpoint is also disabled. In this case, downcasting didn't need to be used - we simply got a resource response back wholesale.
Disabling this endpoint means that typing in feedback currently doesn't work on the site. This is somewhat unfortunate, because there doesn't seem to be any indication to the end user that their feedback didn't go through.
// Anything 'Commentable' is also disabled - you could create a comment on any resource, and get the full resource back
Commentable createComment(input:CommentInput)
Interface Commentable implemented by Adventure, Comment, Post, Scenario {
allowComments: Boolean
comments: [Comment]
id: ID
publicId: String
title: String
totalComments: Int
userId: String
}
Input_object CommentInput{
commentText: String
contentType: String
id: String
publicId: String
}Anything Commentable was vulnerable as well - hence createComment being disabled.
Interface Searchable implemented by Adventure, Post, Scenario { ... }Finally, Searchable has the exact same potential vulnerability going on. However, by its very nature the Searchable endpoints return, well, searchable data, so this is less of an issue.
[Searchable] deletedContent
[Searchable] search(input:SearchInput)
[Searchable] featured(isHoliday:Boolean)
[Searchable] revampedFeatured(isHoliday:Boolean)Interestingly, for Searchable, downcasting can actually be used to good effect for the feature it was meant to support - getting more specific details out of a result set without having to make multiple secondary queries!
Nevertheless, as you can see, a tremendous amount of endpoints all had the possibility of leaking every adventure, scenario, and post, and they've all been disabled because of it.
There are a number of confounding factors that made this vulnerability much worse than it otherwise might have been. Some of these have already been fixed after being brought up to the devs (great work!), but I will still go over them, as I think they're important lessons to keep in mind for anyone wanting to practice good graphql api security, or api security in general.
In my opinion, the biggest four factors at the time of the vulnerability were:
- Using autoincrementing ids (if adventure #20 exists, adventure #21 is likely to exist, as are #22-#30 etc.)
- Not having reasonable rate limits in place
- Not having anomaly detection alerts
- Keeping introspection enabled in production
Now, let's go over what each one means.
Autoincrementing ids are, in my opinion, by far the biggest issue. They allow someone to read all resources, simply by starting from 1 and counting upwards. Had these not been used, a secondary vulnerability would have needed to be discovered alongside the vote vulnerability in order to exploit either one. Otherwise, there would be no way to figure out what the private adventure ids are, even if they could be read through a vulnerability. I recommend deprecating and removing autoincrementing ids completely, as soon as possible. After which point leaking and publishing a non uuid id should be treated as a security issue just by itself.
Also note - autoincrementing ids allow anyone to trivially figure out roughly how many of each resource exists. For AI Dungeon, (as of April 19th) these would be:
- ~1B actions
- ~50M adventures
- ~800K scenarios
- ~250K comments - 10% on posts, 25% as nested comments, 50% on scenarios, 5% on adventures, 10% on "story" posts
- ~20K posts
Not having reasonable rate limits - this one is a mitigating factor, but could have still helped quite a bit by making the hurdle of reading thousands of resources much more difficult. It is not normal user behavior to upvote 1000 adventures a minute. Setting the maximum upvote mutations to, say, 5 requests max per second, 50 requests max per minute, and 500 requests max per hour, would have slowed down the data gathering tremendously.
Of course, these sorts of limits can be overcome - especially with a free product like AI Dungeon. Hundreds of machines could be rented to turn those 500 requests max per hour into tens of thousands. These sorts of attacks require much more sophistication and coordination to pull off, however, making them much less likely. Not impossible, though, which means that these should be mitigated as well. In my opinion, the best way to deal with something like that is through high-threshold anomaly alerting. If the vote endpoint is suddenly getting hammered with 100x its normal traffic, a high-priority alert - not a warning - should be issued, and a developer should be paged.
Finally, introspection should be turned off in production for GraphQL. The AI Dungeon API is a fast-changing, unpublished API. There is no reason to allow users to be able to query all queries, mutations, subscriptions, objects, and enums in one query to __schema.
Again, this can be somewhat circumvented. The endpoints a client uses would, of course, be present in the client code, and users can simply look at requests being made in their browser. However, private, deprecated, and otherwise hidden functionality would not be exposed through this method. Security through obscurity should not be relied upon, but disabling introspection increases the barrier of entry for more advanced attacks.
Speaking of introspection, the following was the exact query made to the AI Dungeon GraphQL API to get the list of endpoints and resources back:
The response was parsed using some hacky javascript. However, the results are something I dare say are presentable. As you can see, the script resolves required variables, arguments, default values, and objects.
Hacky javascript to parse introspection results
Object Mutation
Achievement achieve(achievementId:String)
ActionError addAction(input:ActionInput)
Adventure addAdventure(scenarioId:String, prompt:String, memory:String)
Adventure addCharacter(input:CharacterInput)
Boolean addDeviceToken(token:String, platform:String)
// 100 or so mutations not shown
This is a serious vulnerability that was allowed to happen due to a number of conflating issues. As part of responsible disclosure, this issue, and a handful of others, have been brought up to the devs and have been fixed before this report was published. Hopefully, this report will serve as an inflection point for the AI Dungeon team, causing them to reprioritize their efforts on security related issues.
The team has already made some security changes per my recommendations, such as disabling introspection to their GraphQL API, and are planning to schedule additional security penetration testing sessions to discover bugs and vulnerabilities before they become a problem. More automated testing resulting from this report was mentioned as well, which I am very happy to see. More testing leads to higher quality software, allowing devs to refactor safely, encouraging them to think critically about edge cases, and motivating best practices such as code reuse.
The half dozen or so vulnerable endpoints have been disabled as well, of course.
I am a site reliability engineer with a couple years of experience, and enjoy making things work. If you would like some consultation done, whether it's around graphql, security, infrastructure, scaling, or anything else, reach out. I'd love to work with you!
As something fun, I helped write the AI Dungeon Discord bot.