naive bayes vs logreg

README.md

@@ -100,7 +100,8 @@ Excerpts from the [Foreword](./docs/foreword_ro.pdf) and [Preface](./docs/prefac
 - [How was classification, as a learning machine, developed?](./faq/classifier-history.md)
 - [What are some good books/papers for learning deep learning?](./faq/deep-learning-resources.md)
 - [What are the different dimensionality reduction methods in machine learning?](./faq/dimensionality-reduction.md)
-- [# What is Euclidean distance in terms of machine learning?](./faq/euclidean-distance.md)
+- [What is Euclidean distance in terms of machine learning?](./faq/euclidean-distance.md)
+- [What is the major difference between naive Bayes and logistic regression?](./faq/naive-bayes-vs-logistic-regression.md)
 
 ### Questions about the Book
 

faq/README.md

@@ -43,7 +43,8 @@ Sebastian
 - [How was classification, as a learning machine, developed?](./classifier-history.md)
 - [What are some good books/papers for learning deep learning?](./deep-learning-resources.md)
 - [What are the different dimensionality reduction methods in machine learning?](./dimensionality-reduction.md)
-- [# What is Euclidean distance in terms of machine learning?](./euclidean-distance.md)
+- [What is Euclidean distance in terms of machine learning?](./euclidean-distance.md)
+- [What is the major difference between naive Bayes and logistic regression?](./naive-bayes-vs-logistic-regression.md)
 
 ### Questions about the Book
 

faq/naive-bayes-vs-logistic-regression.md

@@ -0,0 +1,13 @@
+# What is the major difference between naive Bayes and logistic regression?
+
+On a high-level, I would describe it as "generative vs. discriminative" models.
+
+- Generative classifiers learn a model of joint probabilities p(x, y) and use Bayes rule to calculate p(x|y) to make a prediction
+- Discriminative models learn the posterior probability p(x|y) "directly"
+
+You can think of discriminative models as "distinguishing between people that speak different languages without actually learning the language".
+
+In discriminative models, you have "less assumptions", e.g,. in naive Bayes and classification, you assume that your p(x|y) follows (typically) a Gaussian, Bernoulli, or Multinomial distribution, and you even violate the assumption of conditional independence of the features. In favor of discriminative models, Vapnik wrote once "one should solve the classification problem directly and never solve a more general problem as an intermediate step".
+(Vapnik, Vladimir Naumovich, and Vlamimir Vapnik. Statistical learning theory. Vol. 1. New York: Wiley, 1998.)
+
+I think it really depends on your problem though which method to prefer. I can't find a reference now, but e.g. in classification, naive Bayes converges quicker but has typically a higher error than logistic regression. On small datasets you'd might want to try out naive Bayes, but as your training set size grows, you likely get better results with logistic regression.