Permalink
Browse files

progress report

  • Loading branch information...
1 parent befc98c commit d08f16e4c1cfc0892690700e1e4b15279014dd3b @nwoodward nwoodward committed Dec 9, 2011
Showing with 1 addition and 1 deletion.
  1. +1 −1 20111120 Final paper/blog-meme.tex
View
2 20111120 Final paper/blog-meme.tex
@@ -539,7 +539,7 @@ \subsection{Building Training Set with Amazon Mechanical Turk}
\subsubsection{Initial Training Set (MT1)}
Ranking the meme clusters based upon their meaningfulness or informativeness as described above is a difficult and largely subjective task. Consequently, using machine learning for this task is highly problematic. To aid the process of automated classification of the phrase clusters, we implemented an Amazon Mechanical Turk (MTurk) to utilize human intelligence in building a training set \cite{Barr2006}. MTurk is an Internet service from Amazon for scalable crowdsourcing of simple one-time tasks. It allows \emph{requesters} submit human intelligence tasks (HITs) to be completed by \emph{workers} in exchange for small payments. Requesters approve the results they consider satisfactory, and workers increase their confidence scores with high approval rates.
-In this MTurk HIT we displayed to the user a cluster of phrases and three 100-word snippets of text from the blog data in which the phrases appear, along with seven questions, or HITs, related to the meaningfulness of the phrases. Workers were asked questions such as "Is this a meaningful chunk of information?", "Does this meme relate to a specific topic?" and "Is this a popular phrase or expression?".
+In this MTurk HIT we displayed to the user a cluster of phrases and three 100-word snippets of text from the blog data in which the phrases appear, along with seven questions, or HITs, related to the meaningfulness of the phrases. Workers were asked seven questions, such as "Is this a meaningful chunk of information?", "Does this meme relate to a specific topic?" and "Is this a popular phrase or expression?".
In total, we submitted 1090 meme clusters to MTurk with seven HITs each. Workers were required to have a HIT approval rate greater than 85\% and to have completed at least 50 approved HITs. All meme clusters were reviewed by three workers to allow for varying subjective assessments. After removing the highest and lowest total scores to eliminate any individual biases, we calculated the mean score of the cluster and normalized on a 0-1 scale. Each meme cluster was then compared to a range of threshold values for the purposes of the Support Vector Machine, as described in the following section. The labels collected by this initial MTurk HIT will be referred to as Mechanical Turk Collection 1(MT1).

0 comments on commit d08f16e

Please sign in to comment.