I've had more time to flesh this out and I've got a clear picture of what I want in my head now. Here is my revised idea:
We do not want multi-file pastes. We don't want to have anything remotely related to the word 'file' here. Files don't make sense for pastes. Pastes are not files, they are small pieces of files. This isn't always the case, but calling them 'files' by default is weird.
The real problem with multifile pastes is that it complicates everything, especially the API. See the gist API for an example. We want refheap to simple and powerful.
Scenario time. User (or tracked anonymous ip) pastes his code. User has an error message and needs to paste that too. These pastes are directly related to each other and it is verbose and complicated to send two different, seemingly unrelated pastes to his peers on IRC. What is User to do? User can link these pastes (perhaps User could also create the second paste and link it at the same time via a new button on the first pate page. We'd definitely want to make sure they can be edited and linked after the fact though)!
Why is this better than multifile pastes? First of all, we accomplish exactly the same thing and we don't have to call these things 'files' to do it. Multi-file paste is primarily a way to display a bunch of related data in one place without combining it in a single paste (and thus limiting it to a single lexer). We get that, but we also have the added benefit of all of these related pastes being completely standalone pastes as well as being obviously linked and easily visible in the same place. And the API stays just as simple as before. When you fetch a paste with the API, you get a key back containing all of the linked paste IDs and the client can, if he so desires, fetch them. Even better is that this requires almost no backend paste logic changes. This should be able to be added without changing the way pastes are stored or how they are created.
I really like this.
I've been thinking about a compromise since we don't really want to support multiple files in a single paste (because we're trying hard to avoid the whole "pastes are files" concept). What about this: (logged-in and perhaps tracked anonymous IPs) users can create multiple pastes and link them together. Basically, you could have references to pastes from other pastes.
Perhaps we could also display linked pastes on the paste pages somehow? I envision an "expand linked pastes" that shows the other pastes and is collapsible and collapsed by default.
I think that, if we do it right, it'll result in something cleaner than multiple file pastes without significant added complexity. Each paste will be standalone -- they will simply be 'linked' to other pastes so that you can have several pastes that are all related easily accessible. It also wont complicate the API since we can just return the linked paste IDs and any client can fetch those linked pastes if it wants them.
@amcnamara What do you think of this idea? I literally just shat it out in the last 10 minutes, so if it's crazy then tell me so.
Updated with my current thoughts. I've went ahead and marked this high priority because I don't think there is a good reason to not do this. It wont even be that hard.
Linking pastes doesn't sound like it would be necessarily intuitive, from either the side of how to do it or how to traverse them; and it would add more links to the semi-crowded paste links that are already there.
How about this: buckets, or a more friendly name -- heaps.
From your account page or API you can create a new 'heap', and drag/add pastes into it; they'll display just like the all-pastes page (summaries with more links) and all under one place, and have a single link point -- also as a bonus we can make them forkable so a single fork operation would bring the heap and fork all of its contents under your user account. This would also require no change to the current paste logic.
You might have mentioned this above, but I'm super tired and probably read it wrong.
I think both of these ideas should be consolidated. My idea of all pastes being equal here was a little underthought. We do need some concept of what was the 'root' paste and how the pastes connect and in what order. I like your idea more, to be honest. My problem with it is that I absolutely want users to be able to easily see the related pastes on the paste page they are linked to. If a user is linked to a 'heap' page, they just get a bunch of truncated pastes. Actually getting to this heap page from a member of the heap is a step I don't want.
User can create a paste. This new paste page has a button to create a linked paste. You can either link an existing paste or create a new linked paste. The linked paste, either new or existing, will now have a link somewhere back to the original paste. This does not go in the upper right corner unless we do some dropdown magic or something. We do not display all pastes within a linked hierarchy in each paste. The only place where all linked pastes are displayed are on the paste page from which they were linked, this way we have a root paste and we don't have a crazy twisted hierarchy with forks and stuff.
This is not unintuitive, as a matter of fact I think it is elegant in its simplicity. Here is how I see this being used:
Does that seem unintuitive? I don't think it does. It's much less confusing than my original idea and has all of the benefits of your heap idea without the cruft of having a heavy new 'concept' and none of the complexity of 'heap' pages.
Definitely open to modifications of this idea.
Contributing some thoughts:
When the user clicks "link", they should be offered (in a fancy UI, obviously) the choice to create a new paste, along with a list of their X most recent pastes to choose from, with the option of manually entering an ID being a tiny button in the corner or something.