Resources #1

Open
dariusk opened this Issue Oct 25, 2015 · 48 comments

Projects

None yet
@dariusk
Owner
dariusk commented Oct 25, 2015 edited

This is an open issue where you can comment and add resources that might come in handy for NaNoGenMo.

There are already a ton of resources on the old resources threads for the 2013 edition and the 2014 edition.

@javierarce

Maybe someone finds my collection of libraries and APIs useful: https://github.com/javierarce/toolbox

@cpressey

After last year's NaNoGenMo I said I would try to extract useful things from the NaNoGenLab and package them in a more useful way, and, so, yes, this is what I extracted:

  • T-Rext cleans up spacing and punctuation in a text file.
  • Guten-gutter attempts to strip Project Gutenberg boilerplate from a text file (and succeeds more often than gutenizer does)
  • seedbank makes any Python script record and be able to replay the random seeds it chooses, with a one-line change. Useful for improving reproducibility.

They all require Python 2.x (tested on Python 2.7.6 but will probably still work on some earlier versions) but only seedbank requires that you write your script in Python - the other two can be used as stand-alone tools.

They're all in the public domain, as is all the stuff in the NaNoGenLab. So, please steal, fold, spindle, and mutilate as you see fit.

@MichaelPaulukonis

Conceptually, you might want to stumble around The Living Handbook of Narratology - for some overviews of narrative theory, and some alternate takes on what might constitute narrative.

@ikarth
ikarth commented Oct 26, 2015

ProcJam is more visually oriented, but it does have some useful resources that come out of it, like Tracery, a JavaScript library for generating stories from expanding grammars.

@MichaelPaulukonis

Tracey was recently mentioned in Generative Text - the mailing list/google-group that sprouted from the end of NNGM2014. Lately, there's been a lot of partisan ranting about templates. I may resemble some of that remark.

@enkiv2
enkiv2 commented Oct 26, 2015

I'd like to point out that the BotAlly slack group has a NaNoGenMo room,
and that the GenArt slack group that sprouted from the Generative Text
mailing list is mostly full of NaNoGenMo2014 people.

On Mon, Oct 26, 2015 at 11:40 AM Michael Paulukonis <
notifications@github.com> wrote:

Tracey was recently mentioned in Generative Text
https://groups.google.com/forum/#!forum/generativetext - the mailing
list/google-group that sprouted from the end of NNGM2014. Lately, there's
been a lot of partisan ranting about templates. I may resemble some of that
remark.


Reply to this email directly or view it on GitHub
#1 (comment)
.

@ikarth
ikarth commented Oct 26, 2015

Since we're speaking of visual novels, I should probably mention the Yarn dialog editor. I don't know that'll be all that useful directly for generative stuff, but in case someone is trying to procedurally generate a dialog tree, it might be handy. Or you could always generate a Twine file.

@ikarth
ikarth commented Oct 26, 2015

@enkiv2 Where's the signup for the GenArt group? I need to check that out.

@enkiv2
enkiv2 commented Oct 26, 2015

The genart group is at https://generativeart.slack.com ; I think it might
be invite-only. Mike Paulukonis controls the group & can probably send
invites.

On Mon, Oct 26, 2015 at 12:15 PM Isaac Karth notifications@github.com
wrote:

@enkiv2 https://github.com/enkiv2 Where's the signup for the GenArt
group? I need to check that out.


Reply to this email directly or view it on GitHub
#1 (comment)
.

@enkiv2
enkiv2 commented Oct 26, 2015

As for botally, the public signup is here: http://t.co/OiOB7197BH ; people
periodically check for registrations and send invites out.

On Mon, Oct 26, 2015 at 12:18 PM John Ohno john.ohno@gmail.com wrote:

The genart group is at https://generativeart.slack.com ; I think it
might be invite-only. Mike Paulukonis controls the group & can probably
send invites.

On Mon, Oct 26, 2015 at 12:15 PM Isaac Karth notifications@github.com
wrote:

@enkiv2 https://github.com/enkiv2 Where's the signup for the GenArt
group? I need to check that out.


Reply to this email directly or view it on GitHub
#1 (comment)
.

@dariusk
Owner
dariusk commented Oct 26, 2015
@MichaelPaulukonis

Yeah, there's not a lot going on in https://generativeart.slack.com - and if there's a way to allow people to request an invite, I'm all ears. @dariusk - what's the source for the #botALLY registration doc+script?


UPDATE: he's the GenerativeArt invite page: http://tinyurl.com/genartslackrequest

@ikarth
ikarth commented Oct 26, 2015

@MichaelPaulukonis I've seen people use a Google Form to manage signups for Slack groups.

@enkiv2
enkiv2 commented Oct 26, 2015

So far as I can tell, the signup just is a web form that sends an email to
the slack maintainer. Maybe google forms has this functionality by itself?

On Mon, Oct 26, 2015 at 12:29 PM Isaac Karth notifications@github.com
wrote:

@MichaelPaulukonis https://github.com/MichaelPaulukonis I've seen
people use a Google Form to manage signups for Slack groups.


Reply to this email directly or view it on GitHub
#1 (comment)
.

@dariusk
Owner
dariusk commented Oct 26, 2015
@ikarth
ikarth commented Oct 26, 2015

There's been a lot of work done with Neural Networks this year, including text-based output (such as the generated Magic: the Gathering cards). That might be an interesting avenue to pursue.

Abulafia uses MediaWiki for its generators. Not directly useful, but I keep forgetting to mention it here.

Here's a method of procedurally generating a wilderness in a text adventure format. People who are doing a Swallows type simulation approach may find its ideas useful.

@vijithassar

Multiverse JSON is a light JSON syntax/spec and Python compiler script with which to store editorial projects in small logical units so they can be quickly reconfigured according to build parameters you define.

@aparrish

Since last year's NaNoGenMo I've made a few libraries that might be of interest:

I also made Context-Free GenGen, which is sort of like a mini-Tracery except driven by Google Sheets (a la the original GenGen). Probably not great for making a whole novel, but maybe cool for prototyping ideas with context-free grammar generation.

@mgiraldo
@enkiv2
enkiv2 commented Oct 26, 2015

I have a few tools I've made in the past year too:

Some novelty filters:

And, some novelty generators:

On Mon, Oct 26, 2015 at 2:21 PM Allison Parrish notifications@github.com
wrote:

Since last year's NaNoGenMo I've made a few libraries that might be of
interest:

I also made Context-Free GenGen http://cfgg.decontextualize.com/, which
is sort of like a mini-Tracery except driven by Google Sheets (a la the
original GenGen http://tinysubversions.com/gengen/). Probably not great
for making a whole novel, but maybe cool for prototyping ideas with
context-free grammar generation.


Reply to this email directly or view it on GitHub
#1 (comment)
.

@dariusk dariusk added the admin label Oct 26, 2015
@ikarth
ikarth commented Oct 27, 2015

If you're looking for a large corpus of internet comments, the mostly complete Reddit corpus is available.

@coleww
coleww commented Oct 27, 2015

i have been pretty heavily tooling up for nanogenmo this year by way of publishing a ton of poetic node modules to npm.

Another resource i have created is the weirdly-specific-corpora project, which is a fork of dariusk/corpora for any lists that are too weirdly specific to be included in that project. I expect I will end up making a lot of weird lists while generating novels.

@dariusk
Owner
dariusk commented Oct 27, 2015

Someone (I believe @jkirchartz) posted this link to an actual spam blog comment generator template that was accidentally posted as a comment on a guy's blog.

@enkiv2
enkiv2 commented Oct 27, 2015

There is a generator for hilariously poorly written sex scenes
http://www.fiftyshadesgenerator.com/. The source is embedded in the page
there.

On Tue, Oct 27, 2015 at 4:14 PM Darius Kazemi notifications@github.com
wrote:

Someone (I believe @jkirchartz https://github.com/jkirchartz) posted
this link to an actual spam blog comment generator template
http://alexking.org/blog/2013/12/22/spam-comment-generator-script that
was accidentally posted as a comment on a guy's blog.


Reply to this email directly or view it on GitHub
#1 (comment)
.

@MichaelPaulukonis

See sample code in various languages via the Language Survey, issue #17

@ikarth
ikarth commented Oct 30, 2015

@enkiv2 suggested that a short introduction to some historical techniques might be useful in the resources thread to point out possible approaches for people who might not be familiar with some of the precedents of text generation. So I thought I'd write about a few of them.

Dada incorporated several techniques that involved randomly assembling prior texts into new poetry. The cut-up technique takes an existing text, cuts it to pieces, and then reassembles the bits in a new order. This list of Surrealist techniques might also be inspirational.

Oulipo is a group of writers who use constraints to define their writing. Relevant examples include Queneau's Cent Mille Milliards de Poèmes (A Hundred Thousand Billion Poems), which combines ten-line sonnets line-by-line; Perec's Life a User's Manual, which is structured like a knight's tour of a chessboard; Queneau's Excercises in Style, 99 retellings of the same story; Calvino's The Castle of Crossed Destinies, stories interperted via Tarot cards; and Queneau's "A Story as You Like It".

Even where they aren't directly generative, the approaches used by Oulipo writers often point to alternative forms a novel can take. For example, Calvino's Invisible Cities and If on a winter's night a traveler.

Oulipo is also significant because many of its members wrote about systems for generating stories and about incorporating computers into writing. Italio Calvino's essay "Prose and Anticombinatorics," about using the computer to to find the constraints for a murder mystery; Paul Fournel's "Computer and Writer: The Centre Pompidou Experiment"; and Claude Berge's "For a Potential Analysis of Combinatory Literature" are particularly relevant.

Lastly, Jorges Luis Borges tends to crop up a lot, particularly for his short stories: "The Library of Babel", "The Book of Sand", "An Examination of the Work of Herbert Quain", "The Garden of Forking Paths", "Pierre Menard, Author of the Quixote", and "Tlön, Uqbar, Orbis Tertius".

@ikarth
ikarth commented Oct 31, 2015

After @rbechtel mentioned the source code for Micro-Talespin, I went poking around to see what other early AI storytelling systems had source code availible.

Micro-Talespin in Common Lisp, by Warren Sack
another source
[About Talespin](http://lispm.de/mts https://grandtextauto.soe.ucsc.edu/2006/09/13/the-story-of-meehans-tale-spin/)

For Tale-Spin itself: In 2008 Meehan found a copy of the original MLisp source code for Tale-Spin (sans the data files). There's an extensive discussion of the source code in Wardrip-Fruin's Expressive Processing, though I'm not aware of any copies online.

Eliza/Doctor:
In Java
Also in Java
BASIC
JavaScript
Python
The original Lisp source code- github repo

Skald: a Scala reimplementation of Minstrel Remixed, which is itself a Scala reimplementation of Scott Turner's Minstrel (originally in Lisp). Scott Turner on MINSTREL.

@enkiv2
enkiv2 commented Oct 31, 2015

As an example of exercises in minimalism and apophenia, Nick Monfort has a
collection of one kilobyte story generators in python.
https://grandtextauto.soe.ucsc.edu/2008/11/30/three-1k-story-generators/

On Sat, Oct 31, 2015 at 11:20 AM Isaac Karth notifications@github.com
wrote:

After @rbechtel mentioned the source code for Micro-Talespin
#59, I went poking
around to see what other early AI storytelling systems had source code
availible.

Micro-Talespin in Common Lisp, by Warren Sack
http://lispm.de/source/misc/micro-talespin.lisp
another source http://eliterature.org/images/microtalespin.txt
About Talespin
http://lispm.de/mts%20https://grandtextauto.soe.ucsc.edu/2006/09/13/the-story-of-meehans-tale-spin/

For Tale-Spin itself
https://grandtextauto.soe.ucsc.edu/2006/09/13/the-story-of-meehans-tale-spin/:
In 2008 Meehan found a copy of the original MLisp source code for Tale-Spin
(sans the data files). There's an extensive discussion of the source code
in Wardrip-Fruin's Expressive Processing, though I'm not aware of any
copies online.

Eliza/Doctor: https://en.wikipedia.org/wiki/ELIZA
In Java
https://code.google.com/p/simple-semantic-desktop/source/browse/trunk/Progs2/Eliza/eliza.java?r=4
Also in Java http://www.chayden.net/eliza/Eliza.html
BASIC https://www.jesperjuul.net/eliza/ELIZA.BAS
JavaScript https://www.jesperjuul.net/eliza/
Python http://www.jezuk.co.uk/cgi-bin/view/software/eliza
The original Lisp source code http://elizagen.org/\- github repo
https://github.com/jeffshrager/elizagen

Skald https://sites.google.com/a/soe.ucsc.edu/eis-skald/: a Scala
reimplementation of Minstrel Remixed, which is itself a Scala
reimplementation of Scott Turner's Minstrel (originally in Lisp). Scott
Turner on MINSTREL
https://grandtextauto.soe.ucsc.edu/2007/10/30/scott-turner-on-minstrel/.


Reply to this email directly or view it on GitHub
#1 (comment)
.

@MichaelPaulukonis

@ikarth et alia - this year I have been looking at Tale-Spin a bit, even though I don't think I will be using it as a model for what I will be doing. However, I do have some more resources in my NMGM repo and will be adding more there as the month progresses.

@ikarth
ikarth commented Nov 1, 2015

Public service announcement: if you try to download too many files from Project Gutenberg too quickly, they'll ban you for 24 hours.

@dariusk
Owner
dariusk commented Nov 1, 2015
@enkiv2
enkiv2 commented Nov 1, 2015

Gutenberg has a bot policy & instructions on how to avoid rate limits:
https://www.gutenberg.org/wiki/Gutenberg:Information_About_Robot_Access_to_our_Pages

(Not sure if this applies to official mirrors. Unofficial mirrors are also
floating around.)

On Sun, Nov 1, 2015 at 9:44 AM Darius Kazemi notifications@github.com
wrote:

If you're hitting Gutenberg rate limits there are also tons of mirror
sites: http://www.gutenberg.org/MIRRORS.ALL


Reply to this email directly or view it on GitHub
#1 (comment)
.

@hugovk
Collaborator
hugovk commented Nov 1, 2015

Project Gutenberg also has CDs and DVDs of many books for offline use, I've used one of these a lot.
https://www.gutenberg.org/wiki/Gutenberg:The_CD_and_DVD_Project

@TheCommieDuck

https://github.com/TheCommieDuck/nell-extractor

I've just written a quick thing that extracts useful data from CMU's NELL ontology.

Output is a list such as:

lemon_balm_tea beverage agriculturalproduct Lemon Balm Tea

which is an entity called lemon_balm tea, which is a type of beverage and a type of agriculturalproduct, and is called Lemon Balm Tea in English.

Plenty of false-positive entries - beverage general gives about 8,700 entries, mostly different kinds of tea and wine - but it's nice enough if someone wants to try the ontological approach?

I'll probably be refining it throughout the month.

@MichaelPaulukonis

For those interested in Project Gutenberg texts, there is also GITenberg, mentioned in #89 by NNGM participant @sethwoodworth

@hugovk
Collaborator
hugovk commented Nov 3, 2015

If you need food, NYPL has an API around (and CSV dumps of) their historical collection of restaurant menus at What's On The Menu?

http://menus.nypl.org/data

Lots of the dishes and prices have been transcribed and are also available via the API.

I made a Python interface: https://github.com/hugovk/whatsonthemenu

(for https://twitter.com/menubot and http://menubot.tumblr.com)

@ikarth
ikarth commented Nov 3, 2015

For those of us using NLTK: Text Analysis with NLTK Cheatsheet: https://blogs.princeton.edu/etc/files/2014/03/Text-Analysis-with-NLTK-Cheatsheet.pdf

@jemisa
jemisa commented Nov 8, 2015

neural-storyteller is a recurrent neural network for generating little stories about images. Code is at https://github.com/ryankiros/neural-storyteller
and samples https://medium.com/@samim/generating-stories-about-images-d163ba41e4ed

@ikarth
ikarth commented Nov 8, 2015

Linked Data has links to a large number of datasets, many of which are also Open Data.
Datahub lists a large number of datasetes, and can be filtered by license, if you're only interested in public domain or Creative Commons datasets.
Pleiades is a gazetter of ancient geographic places.

@ikarth
ikarth commented Nov 10, 2015

Baba - a Javascript-based designer for text generators. A generator that outputs text generators should be a useful resource for non-coders, but it looks like the live web demo is down so it'll take a little bit of work to get it to run if you download it.

@ikarth
ikarth commented Nov 11, 2015

I've created a thread for beginner's questions and tutorials, as a supplement to the main Resources thread: #152

@hugovk hugovk referenced this issue in dariusk/NaNoGenMo Nov 18, 2015
Open

RESOURCES! #11

@ikarth
ikarth commented Nov 28, 2015

Working in Python? Having unicode issues? Even if you properly take care of the proper encoding and decoding on your end, your source data may be incorrectly encoded. Or, worse, Windows might have mucked it up. If you've got “ or — all over your text, you might want to consider using ftfy. Fixes bad unicode and mojibake; doesn't convert non-unicode to unicode.

@hugovk
Collaborator
hugovk commented Nov 29, 2015

If you want to avoid certain words in your output text, here's a number of things. These are mainly aimed at bots, but should still be generally useful.

Here's a JavaScript, Python, Ruby and PHP word filter:
https://github.com/dariusk/wordfilter

Here's a headline filter:
https://github.com/molly/CyberPrefixer/blob/master/offensive.py

Tips on transphobic joke detection:
http://tinysubversions.com/notes/transphobic-joke-detection/

Some lists of bad words:
https://github.com/shutterstock/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words
https://gist.github.com/ryanlewis/a37739d710ccdb4b406d
http://www.bannedwordlist.com/lists/swearWords.txt

[Inactive] muted Twitter topics:
https://github.com/sjml/bot-innocence

Some general etiquette things:
http://tinysubversions.com/2013/03/basic-twitter-bot-etiquette/
http://www.crummy.com/2013/11/27/0

@greg-kennedy

If you need to plug in some Markov Chains in Perl, you could do a lot worse than the module I built for it this year.
https://github.com/greg-kennedy/MarkovChain

@MichaelPaulukonis

For anyone working with a twitter-sourced corpus, myna is a "JS lib that compiles a tweet to machine speakable text". Or non-twitter-literate text.

For example, the tweet:

"OH: Myna is #SuperAwsome for #TTS and #TwitterBeginners! It is written by @kn bit.ly/ykbz6b"

compiles into plain English:

"Katsuya Noguchi overheard: Myna is Super Awesome for TTS and Twitter Beginners! It is written by Katsuya Noguchi"

@hugovk hugovk referenced this issue in NaPoGenMo/NaPoGenMo2016 Apr 4, 2016
Open

Resources #1

@dariusk
Owner
dariusk commented Oct 8, 2016

Hey all. Since this is our most popular (and most-subscribed to) issue, I just wanted to let people know I have a new issue open to discuss whether we do NaNoGenMo 2016 here on Github again, or somewhere else. Feel free to participate! #199

@dariusk dariusk referenced this issue in NaNoGenMo/2016 Oct 22, 2016
Open

Resources #1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment