Generated Detective: A NaNoGenMo Comic

[atduskgreg]

Presenting Generated Detective #1:

http://gregborenstein.com/comics/generated_detective/1/

For NaNoGenMo this year, I'm working on generating comics. As a starting point, I pulled a bunch of old detective novels off of this list on Project Gutenberg. I wrote a script that searches the text of those books for short sentences that match a series of words. These words act as a kind of script to shape the story. For this first comic here's the script I used:

[:question, :murderer, :witness, :saw, :scene, :killer, :weapon, :clue, :accuse, :reveal]

So, I end up with a random short-ish sentence that matches each of those words.

Then I pull out interesting phrases from each of these sentences (currently by-hand). I ran these sentences through Flickr search to pull down one for each. Finally I run each of these images through a Processing sketch I wrote that attempts to reproduce the look of one of those "manga camera" apps. It breaks the image down into three colors: white, black, and gray. The gray areas it fills in with 45 degree lines. It also uses my OpenCV for Processing library to find contours in the image and add those on top for a hand-drawn line look. The sketch also renders the original sentence in a comic font and automatically scales a box around it as a caption.

Then I put these altogether into a webpage and you get the result you see above.

Moving forward, I'm thinking about how to automate more of this (right now it's a somewhat hand-held pile of hacks). I'm also wondering if I should stick with the detective theme or maybe experiment with some different genres. I think the noir-quality of the detective theme fits well with the way these images look. So maybe I'll stick with that and try to see how much variety/continuity/etc I can wedge into this process.

One feature I plan on adding is speech balloons. Since I'm already using OpenCV it should be pretty easy to do face detection and put the lines from the books into those if I detect a face. I'd also like to experiment with panel layouts other than this totally linear flow (though that can be hard because it's hard to control the dimensions of the images that come back from Flickr search).

Finally, a question here is: what counts as 50,000 words in a comic? Surely the images count for something too. A standard issue of a comic is 20-24 pages long. How many images does a web comic need to be equivalently long? Is there a panel count I'm going for?

[atduskgreg]

Just stuck the code here: https://github.com/atduskgreg/GeneratedDetective

It's a big ole mess at the moment.

[lee2sman]

this is really cool! nice work.

[hugovk]

Wow, looks great!

I think text panels at the bottom works really well with the style.

Another idea is to specifically pull images from Flickr Commons. Partly due to them being copyright-free, most are older photos so may suit a noir-style better. For example, here's results for policeman and veiled woman.

Finally, a question here is: what counts as 50,000 words in a comic? Surely the images count for something too. A standard issue of a comic is 20-24 pages long. How many images does a web comic need to be equivalently long? Is there a panel count I'm going for?

This has been asked elsewhere in some other issues. A jokey answer is: if a picture tells a thousand words, 50 images should do it. The more serious answer is: success and failure are really up to you, the participant.

[lee2sman]

Not the best answer, but i've been experimenting with converting images to ascii, so therefore you'd be able to run a wordcount.

[enkiv2]

This is really cool!

On Thu Nov 06 2014 at 3:35:43 AM Lee T notifications@github.com wrote:

Not the best answer, but i've been experimenting with converting images to
ascii, so therefore you'd be able to run a wordcount.

—
Reply to this email directly or view it on GitHub
#70 (comment)
.

[atduskgreg]

Thanks, everybody! @hugovk Flickr Commons is a good call, thanks for suggesting it.

[atduskgreg]

@lee2sman: I don't think that's quite the aesthetic I want to go for. However, I did once help someone use my OpenCV for Processing library to do exactly that: https://twitter.com/guan/status/451879480282603522/photo/1

[MichaelPaulukonis]

Detectives work well with comics. But you know what was also big?

Romance comics.

Although I shudder to think what an automated engine would produce from today's internet....

On a serious note, science fiction would work well. Not sure how the image-sourcing would fly.

---

The generated comic is amazing. There's a cartoonist this remind me of, but the name is escaping me right now.

[atduskgreg]

@MichaelPaulukonis: thanks! Yeah I thought of doing some other genres. Maybe I should mash up the detective text with space or romance texts. Generated Detective: To the Moon, Generated Detective: In Love, etc. @dariusk pointed out that the results kind of look like Edie Campbell's work from From Hell: http://www.comicsreporter.com/images/uploads/From_Hell_Chapter_2_Page_13.jpg I've tried to emulate that style even more exactly in the past: https://www.flickr.com/photos/unavoidablegrain/8501953252/in/photolist-dXhJuU-2Si24j maybe I should try that here...

[ikarth]

I have no idea how you'd manage to algorithmically digest them for input, but there are large databases of public domain comic books (including lots of romance and detective ones) availible online: http://digitalcomicmuseum.com/

Probably wouldn't work with the really neat photo processing you have going on, but might provide some inspiration.

[lee2sman]

@atduskgreg i'd love to see the code (couldn't find guap's repository/account on github). here's my video to ascii. it's a bit simpler, but only outputs images as png's, not txt files. would love to see an example of how to do that. i have a stranger version here:
https://github.com/lee2sman/Selfie-Textual-Poem-Cam

[atduskgreg]

This was my tweak to Guan's code that generated text:

https://gist.github.com/atduskgreg/c0b901d5c9c4201b6985

I'm pretty sure all I changed there was some of the OpenCv image processing to get a better result rather than any of the text-generation which was all Guan's.

His code fed the generated text to an actual typewriter over serial (and presumably some amount of gadgetry on the other end to do the actual typing), but you can see how he's putting together the text. You'd just want to output it to a file instead.

[lee2sman]

thanks, i'll see if i can figure it out. appreciated. i got the (crazy?) idea of outputting a "novel" of ascii selfies for another NaNoGenMo contribution!

[atduskgreg]

Ok, generated a second "issue" of this:

http://gregborenstein.com/comics/generated_detective/2/

I tried what @hugovk suggested and used the Flickr Commons search for these. I feel like the result was better tied in together and more period but maybe a little bit more boring? There's just less dynamic and compositional range in the images.

Also, curious to hear what people think about the length? Are they maybe a little too long?

A friend compared the result to the Wild Palm comic strip which I'd never seen before but is totally a good comparison, I think (at least visually): http://www.spd.org/images/blog/Details%20Wild%20Palms.jpg

[hugovk]

I tried what @hugovk suggested and used the Flickr Commons search for these. I feel like the result was better tied in together and more period but maybe a little bit more boring? There's just less dynamic and compositional range in the images.

I agree with both points.

[enkiv2]

You could do a lot worse than emulating Wild Palms. (Next step: generate
the TV show instead of the comic)

On Fri Nov 07 2014 at 1:59:00 AM Hugo notifications@github.com wrote:

I feel like the result was better tied in together and more period but
maybe a little bit more boring? There's just less dynamic and compositional
range in the images.

I agree with both points.

—
Reply to this email directly or view it on GitHub
#70 (comment)
.

[atduskgreg]

Just generated Issue 3: http://gregborenstein.com/comics/generated_detective/3/

For this one I downloaded a bunch of sci-fi books from Gutenberg and gave my script generator a 50/50 chance of picking a line from scifi book or a detective book on each panel. I'm pretty excited about how this came out. It doesn't register as sci-fi at all but it added in some more macabre elements that picked up some striking imagery.

Also, I decided to make the comic a little shorter in response to some feedback I'd gotten from showing it to people in person. What do you guys think of this length? It feels a little thin and short to me, but maybe it's good to leave people wanting more.

[atduskgreg]

Oh! I almost forgot. I also added speech bubble generation for lines that come back as quotes. If I get a line that is a quote then I wrote code that looks for faces in the image that comes back and positions a speech bubble with the line in the right-ish spot relative to that. So that's something new as well.

[enkiv2]

This is really cool. In fact, I'd read a webcomic like this even not
knowing it was computer-generated.

On Fri Nov 07 2014 at 7:14:03 PM Greg Borenstein notifications@github.com
wrote:

Oh! I almost forgot. I also added speech bubble generation for lines that
come back as quotes. If I get a line that is a quote then I wrote code that
looks for faces in the image that comes back and positions a speech bubble
with the line in the right-ish spot relative to that. So that's something
new as well.

—
Reply to this email directly or view it on GitHub
#70 (comment)
.

[atduskgreg]

Issue 4:

http://gregborenstein.com/comics/generated_detective/4/

Experimenting with a more sophisticated layout. This is a challenge as it involves cropping the images to fit the right width. It was pretty hand-held this go 'round but I have some thoughts about how to calculate it per-row in the future...

[MichaelPaulukonis]

wow. wow.

I also finally remembered -- this is reminding me of a black-and-white Glen Baxter.

Only more coherent. With a hint of Edward Gorey.

[atduskgreg]

Oh, Glenn Baxter's stuff is great. Thanks for pointing it out to me. I hadn't seen it before.

Just uploaded Issue 5:

http://gregborenstein.com/comics/generated_detective/5/

I'm starting to automate the panel layout process. But I probably won't have it fully done until next issue. The way I'm doing it is to make a tool where I give it a set of panel dimensions and a page width (and a margin size) and it figures out what height to scale all of the panels to so that they'll fit properly on one row of the page. I think I know how to do it in code so it'll just spit out the dimensions for each panel.

Anyway, I thought this one had a really nice mood. Look forward to hearing thoughts.

[dariusk]

What happens if you pick a subject for a whole page? So like, grab sentences that reference "knife" and then generate the page based on those. I think that might bring coherency to both the text and the images.

[atduskgreg]

@dariusk Oh that's an interesting thought. I'll try it for the next one. My instinct is that I'd have to work hard to make the images non-repetitive. Maybe if I used that same search term for all of the sentences but then varied the image search term somewhat from panel to panel...

[atduskgreg]

Ok issue 6 is up. This one has a title: The Knife

http://gregborenstein.com/comics/generated_detective/6/

I tried out @dariusk's idea of using a single word as the seed for each panel. In this case "knife", as suggested. I think the result is much more cohere, certainly.

Also, I wrote a Processing sketch to automate the process of scaling images to fill a row. You can see some screenshots from it here:

https://www.flickr.com/photos/unavoidablegrain/15581458427/

https://www.flickr.com/photos/unavoidablegrain/15766621855/

I load in the images that will share a row and it figures out the height to scale them to so they fit on the row. This saves a lot of time and inches me closer towards full automation.

[enkiv2]

Do you yet have keyword isolation code? It seems like you could use TF-IDF
on captions to determine what keyword to search for for images -- and,
since captions are quite short, you can save time by just sorting by term
frequency ascending and choosing the term in each caption that occurs least
often in the corpus.

On Tue Nov 11 2014 at 12:54:21 PM Greg Borenstein notifications@github.com
wrote:

Ok issue 6 is up. This one has a title: The Knife

http://gregborenstein.com/comics/generated_detective/6/

I tried out @dariusk https://github.com/dariusk's idea of using a
single word as the seed for each panel. In this case "knife", as suggested.
I think the result is much more cohere, certainly.

Also, I wrote a Processing sketch to automate the process of scaling
images to fill a row. You can see some screenshots from it here:

https://www.flickr.com/photos/unavoidablegrain/15581458427/

https://www.flickr.com/photos/unavoidablegrain/15766621855/

I load in the images that will share a row and it figures out the height
to scale them to so they fit on the row. This saves a lot of time and
inches me closer towards full automation.

—
Reply to this email directly or view it on GitHub
#70 (comment)
.

[atduskgreg]

Yeah I haven't bothered with that stuff at this point and am doing it by
hand. Just grabbing standout phrases or words from each line and putting
them into flickr cc search and then picking the most striking of the
first few results.

The technical bits aren't what's stopping me. Seeing the search results,
if I just took the first result of most searches I'd get something
totally random and often really generic. It's important to me to apply a
little bit of my aesthetic judgment at that point.

Rather than a 100% generative process, I like to think of this as a
generative exosuit that carries me through the aesthetic process,
presents me with algorithmically gathered material, and asks me to make
certain decisions at a few key points and then amplifies those decisions
into the final work.

John Ohno wrote:

Do you yet have keyword isolation code? It seems like you could use TF-IDF
on captions to determine what keyword to search for for images -- and,
since captions are quite short, you can save time by just sorting by term
frequency ascending and choosing the term in each caption that occurs least
often in the corpus.

On Tue Nov 11 2014 at 12:54:21 PM Greg Borenstein
notifications@github.com
wrote:

Ok issue 6 is up. This one has a title: The Knife

http://gregborenstein.com/comics/generated_detective/6/

I tried out @dariusk https://github.com/dariusk's idea of using a
single word as the seed for each panel. In this case "knife", as
suggested.
I think the result is much more cohere, certainly.

Also, I wrote a Processing sketch to automate the process of scaling
images to fill a row. You can see some screenshots from it here:

https://www.flickr.com/photos/unavoidablegrain/15581458427/

https://www.flickr.com/photos/unavoidablegrain/15766621855/

I load in the images that will share a row and it figures out the height
to scale them to so they fit on the row. This saves a lot of time and
inches me closer towards full automation.

—
Reply to this email directly or view it on GitHub

#70 (comment)
.

—
Reply to this email directly or view it on GitHub
#70 (comment).

[MichaelPaulukonis]

Rather than a 100% generative process, I like to think of this as a generative exosuit that carries me through the aesthetic process,

I'm sorry, but NaNoMechaMo is December.

....

I think I like the less-coherent pages better; then again, I like Glen Baxter.

If this becomes automated, let it have the ability to switch strategies, so there are different types of coherence.

[hugovk]

For this one, I tried out some of what we've been discussing here. It's the most "fully generative" issue so far. The search term for all the panels is "woman" and I wrote code that searches flickr and automatically downloads a random image from the search results to act as the source.

If you're not already, you could have Flickr return photos sorted by "interestingness". Then pick the top one or a random one from the top selection. See the sort argument. And you could always search for more than one term (e.g. knife man).

[atduskgreg]

Issue 8 is up:

http://gregborenstein.com/comics/generated_detective/8/

Change to the algorithm this time: I pulled out the sentences for the script fully randomly. So now this is a version that's completely algorithmic, without any intervention on my side that couldn't be automated with more (very straightforward) programming:

• Script is generated from randomly selected sentences
• If script line has an em-dash or a semi-colon split it into two queries
• Select a query word or phrase from each script line
• Run it against Flickr search and pick a random result
• If two images were selected from a single sentence, calculate the size to scale them to share a row
• Do the image processing for the "manga camera" look
• Add a caption or a speech bubble (the bubble happens if the sentence has quotation marks in it)

One interesting thing that happened this time was that it picked a famous public domain image (that Mohammed Ali punch).

I definitely think this baseline fully automated version produces results that feel a more diffuse and less narrative than the earlier ones where I gave it a script.

Now that I've reached this baseline, though, I want to start experimenting with ways of producing an interesting script to drive this process.

@hugovk The problem with Interestingness is two-fold: 1) I'm specifically limiting my search to photos with a permissive license for remixing (i.e. CC and public domain) and nearly none of those show up in interestingness (which tend to be by professional photographers) so when I add that criteria to the search I get back 0 results for a lot of my queries. 2) The photos that show up under interestingness tend to be individual objects or people isolated from a background because of nice shallow depth of field close up. Their title are always either the name of the object/person or some kind of very poetic thing. So when they do match a query they're either the dullest thing in the world (just the object named sitting there) or almost completely random seeming.

But yeah I could try it as a sort rather than a filter. It might bring the aesthetically interesting results to the top a bit more (and especially on my weird queries have a hard time reducing it to the genericness of a lot of the top interestingess photos).

[hugovk]

Yes, I definitely meant a sort rather than a filter.

Interestingness isn't a subset of all photos but an algorithmic way of measuring a photo, based on things like views and favourites. So the results can be sorted by this value, rather than, say, recent. And that shouldn't affect the number or actual results returned, just their ordering.

(Flickr's Explore section is powered by interestingness, and the ones there are usually more pro/unremixable. I guess you were somehow getting results from this section.)

[lilinx]

@atduskgreg this generator is awesome, I feel jealous.
I've been trying to do comic generation through image processing and repeatedly failed to make anything consistent. This one is so great and so inspiring.

[atduskgreg]

@lilinx Thanks!

[atduskgreg]

Issue 9 is up:

http://gregborenstein.com/comics/generated_detective/9/

Script here was "death, death, death, death, death". Mostly warming back up after a break due to travel over the weekend.

[atduskgreg]

Issue 10 is up:

http://gregborenstein.com/comics/generated_detective/10/

Script was "love, love, love, love, love" and the first panel has a really funny accidental joke about Zebras.

[MichaelPaulukonis]

These are fantastic!

[lilinx]

Dude you work is so awesome it's reshaping my inner world to some extent

2014-11-19 5:26 GMT+01:00 Michael Paulukonis notifications@github.com:

These are fantastic!

—
Reply to this email directly or view it on GitHub
#70 (comment)
.

[ikarth]

This one is particularly teetering on the dadaist borders of sense-making that the project as a whole has been tangoing with.

[atduskgreg]

Issue 11:

http://gregborenstein.com/comics/generated_detective/11/

A few significant changes here:

• Added some horror novels from Project Gutenberg to the source list
• Redid the script system so that I give it a sequence of genres instead of search terms. So the script this time was: :horror, :scifi, :romance, :romance, :detective, :horror. And it picks a random sentence from the corpus derived from a text in that genre for each source sentence.
• Automated the process of splitting up source sentences into panels. It splits up the sentence by non-period punctuation, producing one search term per phrase to grab one image.

The result, particularly, of this last choice is interesting creating a choppier, more propulsive rhythm and some interesting juxtapositions...

[MichaelPaulukonis]

I would stay up late clicking refresh to see the next panel....

[atduskgreg]

Issue 12:

http://gregborenstein.com/comics/generated_detective/12/

Neat thing happened here where one of the source images had two panels in it already so it ended up with a textless panel...

Plus the wedding theme here emerged as a total fluke.

[atduskgreg]

Issue 13:

http://gregborenstein.com/comics/generated_detective/13/

Pretty much rolling with the same process at this point. Seeing some interesting themes accidentally arise though it's wandered a bit from being a detective comic at this point...

[atduskgreg]

Issue 14:

http://gregborenstein.com/comics/generated_detective/14/

This time I added some slightly more sophisticated parsing: if there's a semi-color in the script line I split by that only instead of commas. Slows down the rhythm some and allows more variation for captions.

[atduskgreg]

Issue 15:

http://gregborenstein.com/comics/generated_detective/15/

Tried a new script generation strategy: I picked an initial key (in this case "body") and used my search algorithm to pick one sentence from the corpus that matches that key. Then I generated the next key from that sentence and so on in a chain to generate the full script.

I think it created some nice transitions and continuity through the script this time.

[hugovk]

@atduskgreg Maybe it's just a temporary thing, but I'm getting "This webpage is not available. Error code: ERR_CONNECTION_REFUSED" from your website in Chrome and "Unable to connect Firefox can't establish a connection to the server at gregborenstein.com.".

[ikarth]

It's up now, at least for me.

[hugovk]

Yep, me too.

[atduskgreg]

@hugovk thanks for pointing this out. Think it was just a temporary DNS problem with my host.

[hugovk]

I think this deserves a completed label :)