Skip to content

hanhanwu/Hanhan_Data_Science_Resources2

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

471 Commits

images

images

 
 

2016_NDSS_Driller.pdf

2016_NDSS_Driller.pdf

 
 

Aite-Group_Biometrics_Report_NDS_2017.pdf

Aite-Group_Biometrics_Report_NDS_2017.pdf

 
 

BioCatch_WP_PREVENTING_FRAUD_IN_MOBILE_ERA.pdf

BioCatch_WP_PREVENTING_FRAUD_IN_MOBILE_ERA.pdf

 
 

Data Science Hiring Guide.pdf

Data Science Hiring Guide.pdf

 
 

ExplainedVariance_ResudualVariance.png

ExplainedVariance_ResudualVariance.png

 
 

LICENSE.txt

LICENSE.txt

 
 

Lessons_from_Large-Scale_Machine_Learning_Deployments_on_Spark.pdf

Lessons_from_Large-Scale_Machine_Learning_Deployments_on_Spark.pdf

 
 

Mastering-Apache-Spark-2.0.pdf

Mastering-Apache-Spark-2.0.pdf

 
 

README.md

README.md

 
 

RoughSets_FuzzyRoughSets.R

RoughSets_FuzzyRoughSets.R

 
 

Security_Data_Science.md

Security_Data_Science.md

 
 

Standford_CNN_Notes1.pdf

Standford_CNN_Notes1.pdf

 
 

WestGrid_commands.md

WestGrid_commands.md

 
 

What is Security Automation Whitepaper RC2.pdf

What is Security Automation Whitepaper RC2.pdf

 
 

about_calibration.ipynb

about_calibration.ipynb

 
 

alert-fatigue-best-1.pdf

alert-fatigue-best-1.pdf

 
 

autism_assets.png

autism_assets.png

 
 

aws_sagemaker.png

aws_sagemaker.png

 
 

bayesian.png

bayesian.png

 
 

boruta_statistics.pdf

boruta_statistics.pdf

 
 

gain:lift chart.png

gain:lift chart.png

 
 

gephi data set.zip

gephi data set.zip

 
 

gephi.png

gephi.png

 
 

learn_markov_chain.R

learn_markov_chain.R

 
 

onion limitations.png

onion limitations.png

 
 

spark_vs_dask.png

spark_vs_dask.png

 
 

ua_string.png

ua_string.png

 
 

validation_methods_sumamry.ipynb

validation_methods_sumamry.ipynb

 
 

validation_methods_summary.R

validation_methods_summary.R

 
 

Repository files navigation

Hanhan_Data_Science_Resources2

SUMMARIZED RESOURCES

Data Science Hacks

TREE BASED MODELS & ENSEMBLING

Advantages of Tree Boosting Methods

  • Won't be affected by the curse of dimentionality, because distance metric is not used in them.

Light GBM

  • Reference: https://www.analyticsvidhya.com/blog/2017/06/which-algorithm-takes-the-crown-light-gbm-vs-xgboost/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+AnalyticsVidhya+%28Analytics+Vidhya%29
  • Leaf-wise - Optimization in Accuracy: Other boosting algorithms use depth-wise or level-wise, while Light BGM is using leaf-wise. With this method, Light GBM becomes more complexity and has less info loss and therefore can be more accurate than other boosting methods.
  • Sometimes, overfitting could happen, check LightGBM Parameter tuning here: https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html
    • Suggestions for faster speed.
    • Suggestions for better accuracy.
    • Suggestions for dealing with overfitting.
  • Using Histogram Based Algorithms
    • Many boosting tools as using pre-sorted based algorithms (default XGBoost algorithm) for decision tree learning, which makes optimization more difficult.
    • LightGBM uses the histogram based algorithms, which bucketing continuous features into discrete bins, to speed up training procedure and reduce memory usage.
      • Reduce Calculation Cost of Split Gain: pre-sorted based cost O(#data) to calculate; histogram based needs O(#data) to construcu histogram but O(#bins) to calculate Split Gain. #bins often smaller than #data, and this is why if you tune #bins to a smaller number, it will speed up the algorithm.
      • Use histogram subtraction for further speed-up: To get one leaf's histograms in a binary tree, can use the histogram subtraction of its parent and its neighbor, only needs to construct histograms for one leaf (with smaller #data than its neighbor), then can get histograms of its neighbor by histogram subtraction with small cost( O(#bins) ).
      • Reduce Memory usage: with small number of bins, can use smaller data type to store trainning data; no need to store extra info for pre-sorting features
  • Sparse Optimization: Only need O(2 x #non_zero_data) to construct histogram for sparse features.
  • Optimization in network communication: it implements Collective Communication Algorithms which can provide much better performance than Point-to-Point Communication.
  • The way LGBM handles missing data is to ignore them and later allocate them to whiever side reduces the loss most
  • Oprimization in Parallel Learning
    • Feature Parallel - Different from traditional feature parallel, which partitions data vertically for each worker. In LightGBM, every worker holds the full data. Therefore, no need to communicate for split result of data since every worker know how to split data. Then Workers find local best split point{feature, threshold} on local feature set -> Communicate local best splits with each other and get the best one -> Perform best split
    • Data Parallel - However, when data is huge, feature parallel will still be overhead. Use Data Parallel instead. Reduce communiation. Reduced communication cost from O(2 * #feature* #bin) to O(0.5 * #feature* #bin) for data parallel in LightGBM. Instead of "Merge global histograms from all local histograms", LightGBM use "Reduce Scatter" to merge histograms of different(non-overlapping) features for different workers. Then workers find local best split on local merged histograms and sync up global best split. LightGBM use histogram subtraction to speed up training. Based on this, it can communicate histograms only for one leaf, and get its neighbor's histograms by subtraction as well.
    • Voting Parallel - Further reduce the communication cost in Data parallel to constant cost. It uses two stage voting to reduce the communication cost of feature Histograms.
  • Advantages
    • Faster Training - histogram method to bucket continuous features into discrete bins.
    • Better Accuracy than other boosting methods, such as XGBoost.
    • Performe on large dataset.
    • Parallel Learning.
  • More about LightGBM parameters: https://lightgbm.readthedocs.io/en/latest/Parameters.html

CatBoost

The Comparison Between XGBoost, LightGBM and CatBoost

  • The 2019 paper: https://arxiv.org/pdf/1809.04559.pdf
  • When using GPU, XGBoost tend to appear to be the fastest in training (fixed set of params). It seems that at this moment, LightGBM GPU version still has some issue.
  • When there are many features, XGBoost might run out of memory while CatBoost can converge into a good solution in the shortest time.

DATA PREPROCESSING

For more data preprocessing, check DATA PREPROCESSING section

Check Dataset Shifting

  • For me, I will majorly use it to check whether the new dataset still can use current methods created from the previous dataset.
    • For example, you are using online model (real time streaming), and you need to evaluate your model periodically to see whether it still can be applifed to the new data streaming. Or for time series, you want to check whether the model built for a certain time range applies to other time. And many other situations, that the current model may no longer apply to the new dataset
  • Types of Data Shift
    • Covariate Shift - Shift in features. Then for the new model, you may need to modify feature selection, or find those features may lead to data shift and don't use them as selected features.
      • If the features in both the dataset belong to different distributions then, they should be able to separate the dataset into old and new sets significantly. These features are drifting features.
    • Prior probability shift - Shift in label. For example, when you use Byesian model to predict multiple categories, then all the classes appeared in testing data has to appear in training, if not then it is Prior probability shift.
    • Concept Shift - Shift in the relationship between features and the label. For example, in the old dataset, Feature1 could lead to Class 'Y', but in the new dataset, it could lead to Class 'N'.
  • My Code: https://github.com/hanhanwu/Hanhan_Data_Science_Practice/blob/master/deal_with_data_shifting.ipynb
  • Reference: https://www.analyticsvidhya.com/blog/2017/07/covariate-shift-the-hidden-problem-of-real-world-data-science/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+AnalyticsVidhya+%28Analytics+Vidhya%29

Dimension Reduction

t-SNE, non-linear dimensional reduction

A little more about Factor Analysis

  • Fator Analysis is a variable reduction technique. It is used to determine factor structure or model. It also explains the maximum amount of variance in the model. Such as PCA.
    • EFA (Exploratory Factor Analysis) – Identifies and summarizes the underlying correlation structure in a data set
    • CFA (Confirmatory Factor Analysis) – Attempts to confirm hypothesis using the correlation structure and rate ‘goodness of fit’.

Dimension Reduction Must Know

  • Reference: https://www.analyticsvidhya.com/blog/2017/03/questions-dimensionality-reduction-data-scientist/?utm_content=bufferc792d&utm_medium=social&utm_source=linkedin.com&utm_campaign=buffer
  • Besides different algorithms to help reduce number of features, we can also use existing features to form less features as a dimensional reduction method. For example, we have features A, B, C, D, then we form E = 2A+B, F = 3C-D, then only choose E, F as the features for analysis
  • Cost function of SNE is asymmetric in nature. Which makes it difficult to converge using gradient decent. A symmetric cost function is one of the major differences between SNE and t-SNE.
  • For the perfect representations of higher dimensions to lower dimensions, the conditional probabilities for similarity of two points must remain unchanged in both higher and lower dimension, which means the similarity is unchanged.
  • LDA aims to maximize the distance between class and minimize the within class distance. If the discriminatory information is not in the mean but in the variance of the data, LDA will fail.
    • LDA is used in linearly seperated classification problems. For non-linear classes' relationship, can try kernel functions to convert to linear relationship first
    • LDA doesn't work well when the number of features is larger than the number of observations
  • Both LDA and PCA are linear transformation techniques. LDA is supervised whereas PCA is unsupervised. PCA maximize the variance of the data, whereas LDA maximize the separation between different classes.
  • When eigen values are roughly equal, PCA will perform badly, because when all eigen vectors are same in such case you won’t be able to select the principal components because in that case all principal components are equal. When using PCA, it is better to scale data in the same unit.
    • To make this concept simple, PCA is built on checking feature variance. So features with larger values and tend to have larger variance could misleading PCA results, and therefore we need to normalize the data before using PCA. I guess, maybe because some features have much larger variance, eigen tend to show roughly equal
  • When using PCA, features will lose interpretability and they may not carry all the info of the data. You don’t need to initialize parameters in PCA, and PCA can’t be trapped into local minima problem. PCA is a deterministic algorithm which doesn’t have parameters to initialize. PCA can be used for lossy image compression, and it is not invariant to shadows.
    • A deterministic algorithm has no param to initialize, and it gives the same result if we run again.
  • Logistic Regression vs LDA:
    • If the sample size is small and distribution of features are normal for each class. In such case, linear discriminant analysis (LDA) is more stable than logistic regression.

MODEL EVALUATION

  • For my past notes on evaluation methods, check: https://github.com/hanhanwu/readings/blob/master/Evaluation_Metrics_Reading_Notes.pdf

  • 7 important model evaluation metrics and cross validation: https://www.analyticsvidhya.com/blog/2016/02/7-important-model-evaluation-error-metrics/

  • 11 important model evaluation metrics and corss validation: https://www.analyticsvidhya.com/blog/2019/08/11-important-model-evaluation-error-metrics/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+AnalyticsVidhya+%28Analytics+Vidhya%29

  • Confusion Matrix

  • F1 - the benefit of F1 score is to prevent all the records got predicted as positive or negative.

    • Because F1 = 2*precision*recall/(precision + recall), for example precision = 0, recall=1, F1 will only be 0.5.

  • Besides F Sccore, there is G measure (G mean)

    • G mean1 - GSP = sqrt(recall * precision)
      • While F score is the harmonic mean of precision and recall, GSP (G mean1) is the geometric mean.
    • G mean2 - GSS = sqrt(recall * sensitivity)

  • Lift / Gain charts are widely used in campaign targeting problems. This tells us till which decile can we target customers for an specific campaign. Also, it tells you how much response do you expect from the new target base.

    • The diagonal shows random situation, and you should compare your model with this diagonal (similar to ROC logic). The graph tells you how well is your model segregating responders from non-responders
    • The first decile means, at 10% population (x-axis), if you got 14% responders, it means you have 140% lift at first decile
    • The very first step is to create this gain/lift chart, so that you can plot other charts: gain/lift chart
    • With graph Lift@Decile, you can figure out till which decile, your model still works well, so that later you will know to which level you can adjust your model
    • lift@decile = number of respindents at that decile/total number of respondents

  • Kolmogorov-Smirnov (K-S) chart is a measure of the degree of separation between the positive and negative distributions. The K-S is 100, the higher the value the better the model is at separating the positive from negative cases.

  • The ROC curve is the plot between sensitivity and (1- specificity). (1- specificity) is also known as false positive rate and sensitivity is also known as True Positive rate. To bring ROC curve down to a single number, AUC, which is the ratio under the curve and the total area. .90-1 = excellent (A) ; .80-.90 = good (B) ; .70-.80 = fair (C) ; .60-.70 = poor (D) ; .50-.60 = fail (F). But this might simply be overfitting. In such cases it becomes very important to do in-time and out-of-time validations. For a model which gives class as output, will be represented as a single point in ROC plot. In case of probabilistic model, we were fortunate enough to get a single number which was AUC-ROC. But still, we need to look at the entire curve to make conclusive decisions.

    • Gini Coefficient = 2*AUC – 1. It indicates how large the ROC area is above the diagnoal.
    • Compared with Gain/Lift Chart, Lift is dependent on total response rate of the population. ROC curve on the other hand is almost independent of the response rate, because the numerator and denominator of both x and y axis will change on similar scale in case of response rate shift.
    • When there is data imbalance problem, ROC may not work well, especially when positive class is very small, in this case try precision-recall curve (PR curve) might be better.

  • ROC-AUC vs AVP vs F1 vs Balanced Accuracy

    • When data is imbalanced, AVP is more reliable than ROC and Balanced Accuracy. Because when data is imbalanced (much more negative than positive), TN can be high, and therefore balanced accuracy can be higher (TNR is higher) while ROC-AUC can be larger (FPR is smaller).
    • Comparing with AVP, F1 sets precision and recall the same weight, while AVP is using increased recall (δ recall) from the previous threshold as the weight for current precision.

  • Logloss (binary cross-entropy)

    • It takes the probability (certainty) of prediction into account.
      • log-loss = negative average of the log of corrected predicted probability, check details here
        • Check the curves of logloss, it penalizes more on false predicted results.
      • Lower logloss value, the better.

  • RMSE: The power of ‘square root’ empowers this metric to show large number deviations. Often used as a metric for linear regression (convex function).

    • RMSE is highly affected by outlier values. Hence, make sure you’ve removed outliers from your data set prior to using this metric. As compared to mean absolute error, RMSE gives higher weightage and punishes large errors.
    • Tend to stuck at local minimum for non-convex function, such a logistic regression.
      • Logloss is a better choice in this situation.

  • (RMSLE) Root Mean Squared Logarithmic Error

  • k-fold cross validation is widely used to check whether a model is an overfit or not. If the performance metrics at each of the k times modelling are close to each other and the mean of metric is highest. For a small k, we have a higher selection bias but low variance in the performances. For a large k, we have a small selection bias but high variance in the performances. Generally a value of k = 10 is recommended for most purpose.

  • To measure linear regression, we could use Adjusted R² or F value.

  • Variance inflation factor (VIF) is used as an indicator of multicollinearity.

    • Higher VIF, the feature is more likely to be highly correlated with other features.
    • Normally VIF > 10 is very high correlation, VIF > 5 is risky.

  • To measure logistic regression:

    • AUC-ROC curve along with confusion matrix to determine its performance.
    • The analogous metric of adjusted R² in logistic regression is AIC. AIC is the measure of fit which penalizes model for the number of model coefficients. Therefore, we always prefer model with minimum AIC value.
    • AIC vs BIC:
      • https://methodology.psu.edu/AIC-vs-BIC
      • http://www.differencebetween.net/miscellaneous/difference-between-aic-and-bic/
      • AIC is an estimate of a constant + the relative distance between the unknown true likelihood function of the data and the fitted likelihood function of the model, so that a lower AIC means a model is considered to be closer to the truth.
      • BIC is an estimate of a function of the posterior probability of a model being true, under a certain Bayesian setup, so that a lower BIC means that a model is considered to be more likely to be the true model.
      • They both penalize model coefficients. They are similar, but BIC penalize complex models more.
    • Null Deviance indicates the response predicted by a model with nothing but an intercept. Lower the value, better the model.
    • Residual deviance indicates the response predicted by a model on adding independent variables. Lower the value, better the model.
    • Regularization becomes necessary when the model begins to ovefit / underfit. This technique introduces a cost term for bringing in more features with the objective function. Hence, it tries to push the coefficients for many variables to zero and hence reduce cost term. This helps to reduce model complexity so that the model can become better at predicting (generalizing).

  • Probability vs Response

    • Probability means predict continuous values (such as probability), Response means predict specific classes.
    • Response Measures
      • Confused Matrix >> Precision-Recall, F Score, Specificity & Sensitivity, ROC-AUC/Banalced Accuracy/G-Mean, Gini Coefficient
      • Concordant & Discordant
    • Probability Measures
      • Logloss (often used by Kaggle) - it focuses on penalizing false classification. So the goal to improve your model is to minimize logloss.
      • Adjusted R-square - used to reduce collineary problem (reduce correlated features) in regression. We can check both Adjusted R-Square and R-Square, if R-Square is much higher, it means we have unnecessary features that do not contribute much in the model.
        • adjusted R-square = 1 - (1-R-square)*(n-1)/(n-k-1)
          • n - number of samples
          • k - number of features
      • Expected Variance & Residual Variance
        • R-Square = Expected Variance/Total Variance. It's the percentage of the response variable variation that is explained by the model.
          • so higher expected variance can be better, although it can suffer simiar critics that RMSE got.
          • R-Square = 1- MSE(model)/MSE(baseline)
            • MSE(model): Mean Squared Error of the predictions against the actual values
            • Mean Squared Error of average prediction against the actual values
          • R-Square, RSS (residual sum of squares) = MSE*n, will increase when there are more features, but the test error may not drop. Therefore, R-Square, RSS should NOT be used for selecting models that have differernt number of features.
        • Better to check residual plot with R square, if there is a pattern in the residual plot instead of the randomness, then even if R square is high, there is still unexplained pattern in the data
        • Residual Variance (Unexplained Variance) = Total Variance - Expected Variance, lower the better.
      • RMSE, sensitive to large outliers, mean is not statistical robust
      • Since mean is not statistical robust, we can try:
        • Quantile of Errors
        • Median Absolute Percentage = median(|(yi_true - yi_forecast)/yi_true|)
      • Brier Score
        • It is calculated by Uncertainty, Resolution and Reliability scores
        • You just need ground truth and the probability prediction results to do the calculation.
        • Predictions that are further away from the expected probability are penalized, but less severely as in the case of log loss.
        • Like the average log loss, the average Brier score will present optimistic scores on an imbalanced dataset, rewarding small prediction values that reduce error on the majority class.
        • This is the best tutorial I have found from description to implementation: https://timvangelder.com/2015/05/18/brier-score-composition-a-mini-tutorial/
      • Logloss vs MSE
        • Logloss
          • Each predicted probability is compared to the actual class output value (0 or 1) and a score is calculated that penalizes the probability based on the distance from the expected value. The penalty is logarithmic, offering a small score for small differences (0.1 or 0.2) and enormous score for a large difference (0.9 or 1.0).
          • A model with perfect skill has a log loss score of 0.0. In order to summarize the skill of a model using log loss, the log loss is calculated for each predicted probability, and the average loss is reported.
          • It can be misleading for largely imbalanced data, because when predicting 0 or smaller probabilities, the loss will be smaller.
        • MSE (Squared Error) works better for continuous output while Logloss works better for numerical classification.
        • Similar to MSE, logloss will increase when the predicted probability diverges from the actual value. At the same time, Logloss takes into account the uncertainty of the predictions based on how much it varies from the ground truth.
        • MSE would be bad if we're aiming to have our model output discrete values (such as 0, 1), because the model would end up trying to keep values close to 0 or 1 (basically estimating the mean of the distribution) and wouldn't learn to do anything very sophisticated around distribution estimation. However when your targets are continuous, RMSE/MSE works fairly well - in other words, squared error is a good measure for regression but not for numerical classification.
        • Logloss does maximum likelihood estimation of a multinomial distribution, it also punishes bad class labelling, but only when those labels have high confidence. So Logloss is better for numerical classification.

  • Data Validation

  • Calibration

    • We all know that we can predict classes with probability in classification. The return probability of the predicted classes can be overconfident or underconfident. Well calibrated classifiers can return probabilities that also serve as the confidence level of the prediction.
    • Also because of the biased probability some classifiers can return, calibration also serve as a postprocessing method after classification to reduce the prediction bias.
    • My code & description: https://github.com/hanhanwu/Hanhan_Data_Science_Resources2/blob/master/about_calibration.ipynb
      • Part 1 - Compare the calibration of multiple classifiers
        • If the plot is closer to the diagonal, the higher accuracy the model has. Sometimes you cannot tell which model is better through plot so added Brier score in the code.
        • We use Brier score as the evaluation, lower Brier, more accurate, better calibrated predictions. We can also see whether each classifier push the values towards to 0 or 1, or push away from them.
      • Part 2 - Calibration as classification post-processing
        • The purpose of post-processing is to improve the calibration results of the classification models.
        • We use naive bayesian as the base model, comapred without calibration, with sigmoid calibration and with non-paramatric isotonic calibration.
    • To learn more about calibration: http://www.analyticsvidhya.com/blog/2016/07/platt-scaling-isotonic-regression-minimize-logloss-error/?utm_content=buffer2f3d5&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer

Applied Data Science in Python/R/Java

Statistics in Data Science

6 distributions

  • Overall
    • The expected value of any distribution is the mean of the distribution
  • Bernoulli Distribution
    • only 2 possible outcomes.
    • The expected value for a random variable X in this distributioon is p (probability)
    • The variance for a random variable X in this distributioon is p(1-p)
  • Uniform Distribution
    • The probability of all the N number of outcomes are equally likely.
    • a is min of uniform distribution, b is max of uniform distribution, the probability of a random variable in X is 1/(b-a); the probability of a range (x1,x2) is (x2-x1)/(b-a)
    • mean = (a+b)/2
    • variance = (b-a)^2/12
    • Standard Uniform Distribution has a=0, b=1, if x in [0,1] range, probability is 1, otherwise 0
  • Binomial Distribution    * 2 outcomes, the probability for each is p, 1-p. In each trail, the probability of each outcome is the same.
    • Each trail is independent.
    • n is the number of trails, p is the probability of success in each trail => mean=n*p, variance=n*p*(1-p)
    • If the probability of 2 outcomes p, 1-p are the same, then the distribution is normal distribution.
    • Odds: Odds are defined as the ratio of the probability of success and the probability of failure. For example, a fair coin has probability of success is 1/2 and the probability of failure is 1/2 so odd would be 1.
      • odds = # of event happen / # of event not happen
      • odds ratio = odds of event happend / odds of event not happen, usually used to compare the outcome of 2 events
      • log odds = log(odds), log odds ratio = log(odds ratio), applying log is to normalze the values, since the value range of odds is between 0 and infinite, the gaps between the odds of event happen and the odds of event not happen can be huge. Log can reduce the gap, and also make the distribution of the odds symmetric
  • Normal Distribution
  • Poisson Distribution
    • Example of how to apply poisson: https://www.analyticsvidhya.com/blog/2022/06/predicting-the-scoreline-of-a-football-match-using-poisson-distribution/?utm_source=feedburner&utm_medium=email
      • Estimate the probability of something could happen in an event
    • It is applicable in situations where events occur at random points of time and space where in our interest lies only in the number of occurrences of the event.
      • e.g number of times of have ice-cream in a year; number of flowers in a garden. etc
    • Events are independent from each other.
    • The probability of success over a short interval must equal to the probability of success over a longer interval.
    • The probability of success in an interval approaches zero as the interval becomes smaller.
    • λ is the rate at which an event occurs, t is the length of a time interval, and X is the number of events in that time interval. mean = λ*t.
  • Exponential Distribution
    • Compared with poisson distribution, exponential distribution means the time interval between 2 events.
      • e.g the time interval between eating 2 ice-creams
    • It is widely used in survival analysis. λ is called the failure rate of a device at any time t, given that it has survived up to t. For a random variable X, mean=1/λ, variance=(1/λ)^2. The greater the rate, the faster the curve drops and the lower the rate, flatter the curve.
  • Relationship between distributions
    • Bernoulli Distribution is a special case of Binomial Distribution with a single trial
    • Poisson Distribution is a limiting case of binomial distribution under the following conditions:
      • The number of trials is indefinitely large or n → ∞.
      • The probability of success for each trial is same and indefinitely small or p →0.
      • np = λ, is finite
    • Normal distribution is another limiting form of binomial distribution under the following conditions:
      • The number of trials is indefinitely large, n → ∞
      • Both p and q are NOT indefinitely small, and p=q
    • Normal distribution is also a limiting case of Poisson distribution with the parameter λ →∞
    • If the times between random events follow exponential distribution with rate λ, then the total number of events in a time period of length t follows the Poisson distribution with parameter λt.
  • Skewness List for Distributions: https://mathworld.wolfram.com/Skewness.html
    • With this method, it can check whether a distribution is skewed
    • Positive value means right skewed (positively skewed)
    • Negative value means left skewed (negatively skewed)
    • 0 means not skewed
  • reference: https://www.analyticsvidhya.com/blog/2017/09/6-probability-distributions-data-science/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+AnalyticsVidhya+%28Analytics+Vidhya%29

Maximum Likelihood Estimation

Bayesian Statistics

Find All Calculators here

Terminology

Confusing Concepts

  • Confounding Variable vs Interaction
    • Confounding variable is the variable that will affect both dependent and independent variables. I will increase the variance and introduce bias in the model results.
      • For example, gaining weight is the dependent variable, lack of exercise is an independent variable, age is the confounding variable that will affect both of these variables.
      • Better to control some the variable values used in data sample and experiments, such as limit the age within 1 group for the experiment.
    • Interaction indicates how variable A affects the dependent variable differs when variable B changes among different levels.
      • In some model output, we will output the interaction between features to uderstand further about the data.
  • Errors and Residuals: https://en.wikipedia.org/wiki/Errors_and_residuals
    • Statistical Error is the amount by which an observation differs from its expected value (population mean). Since the mean of the population is not observable in most of the cases, error is not an observable either.
    • Residual is observable, it is an observable estimate of the unobservable statistical error. Residual is the amount by which an observation differs from its sample mean (not population mean).
  • Explained Variation vs Residual Variation
    • Residual Variation is unobservable.... It indicates the variance around the regression line.
    • This video is good: https://www.coursera.org/learn/regression-models/lecture/WMAET/residual-variance Explained Variation vs Residual Variation
    • Residual Variance = Total Variance - Explained Variance
    • R-Squared = Explained Variance/Total Variance, so explained variance higher, the better, although it will get similar criticism as R-Square for not be able to handle multicolineary
  • Heteroskedasticity: led by non-constant variance in error terms. Usually, non-constant variance is caused by outliers or extreme values.
  • Coefficient and p-value/t-statistics: coefficient measures the strength of the relationship of 2 variables, while p-value/t-statistics measures how strong the evidence that there is non-zero association
  • Anscombe's quartet comprises four datasets that have nearly identical simple statistical properties, yet appear very different when graphed: https://en.wikipedia.org/wiki/Anscombe's_quartet
    • Same mean, variance, different distributions.
  • Correlation & Covariance: In probability theory and statistics, correlation and covariance are two similar measures for assessing how much two attributes change together.
    • Covariance indicates the direction of linear relationship between 2 variables. Correlation indicates both direction and strength of the linear relationship between 2 variables.
    • correlation(A, B) = covariance(A, B)/(std(A)*std(B))
      • covariance(A, B) = sum((xi - avg(Y)) * (yi - avg(Y)))/N
      • Correlation is between [-1, 1], covariance is between (-infinity, +infinity)
  • Rate vs Proportion: A rate differs from a proportion in that the numerator and the denominator need not be of the same kind and that the numerator may exceed the denominator. For example, the rate of pressure ulcers may be expressed as the number of pressure ulcers per 1000 patient days.
  • Bias is the error due to the over-simplicity of the model, tends to lead to underfitting and hard to get high predictive accuracy. Variance is the error due to the over-complex of the model, tends to be highly sensitive to high degrees of variation in the training data and leads to overfitting, could also bring in more noise.
    • The total error is lower when both bias and variance are lower. So during parameter tuning, better to find that sweet spot when bias, variance and the error are all lower.
      • There is a function called bias_variance_decomp() to output the bias and variance of the model. Example 1, Example 2
    • About variance here, according to above description, if we check multiple models on the same set of data, the difference is called variance. Meanwhile there is model variance, which means the same model showing different results because of the randomness in different data input. These 2 variance can be differentiated by, whether it's same model or multiple model, whether it's multiple dataset or same dataset.

  • OLS and Maximum likelihood are the methods used by the respective regression methods to approximate the unknown parameter (coefficient) value. OLS is to linear regression. Maximum likelihood is to logistic regression. Ordinary least square(OLS) is a method used in linear regression which approximates the parameters resulting in minimum distance between actual and predicted values. Maximum Likelihood helps in choosing the the values of parameters which maximizes the likelihood that the parameters are most likely to produce observed data.
  • Standard Deviation – It is the amount of variation in the population data. It is given by σ. Standard Error – It is the amount of variation in the sample data. It is related to Standard Deviation as σ/√n, where, n is the sample size, σ is the standandard deviation of the population. A low standard deviation indicates that the data points tend to be close to the mean (also called the expected value) of the set, while a high standard deviation indicates that the data points are spread out over a wider range of values. The standard deviation is the square root of the variance.
  • 95% confidence interval does not mean the probability of a population mean to lie in an interval is 95%. Instead, 95% C.I means that 95% of the Interval estimates will contain the population statistic.
    • "Margin of Error" is the half width of confidence interval.
    • Larger the dataset, narrower the confidence interval is.
    • If a sample mean lies in the margin of error range then, it might be possible that its actual value is equal to the population mean and the difference is occurring by chance.
  • Difference between z-scores and t-values are that t-values are dependent on Degree of Freedom of a sample, and t-values use sample standard deviation while z-scores use population standard deviation.
  • The Degree of Freedom – It is the number of variables that have the choice of having more than one arbitrary value. For example, in a sample of size 10 with mean 10, 9 values can be arbitrary but the 10th value is forced by the sample mean.
  • Residual Sum of Squares (RSS) - It can be interpreted as the amount by which the predicted values deviated from the actual values. Large deviation would indicate that the model failed at predicting the correct values for the dependent variable. Regression (Explained) Sum of Squares (ESS) – It can be interpreted as the amount by which the predicted values deviated from the the mean of actual values.
  • Residuals is also known as the prediction error, they are vertical distance of points from the regression line
  • Co-efficient of Determination = ESS/(ESS + RSS). It represents the strength of correlation between two variables. Correlation Coefficient = sqrt(Co-efficient of Determination), also represents the strength of correlation between two variables, ranges between [-1,1]. 0 means no correlation, 1 means strong positive correlation, -1 means strong neagtive correlation.

About Data Sampling

  • Probability sampling can be representative, non-probability sampling may not

  • Probability Sampling

    • Random sample. (I guess R sample() is random sampling by default, so that each feature has the same weight)
    • Stratified sample
    • Clustering Sampling - When it's difficult to study the target population spread across a wide area and random sampling won't work well, we divide the data into different clusters. Such as you sample the data in each grade/city/geneder/etc.
    • Systematic Sampling - it starts from a random position but choose data sample in certain intervals and the selection is in circular manner, which means once reached to the end of the list, it will go back to the top to do further selection.
      • Equal probability method - fixed interval when selecting the next data point.

  • Nonprobability Sampling

    • Quota sample
    • Purposive sample
    • Convenience sample

  • Bias that can occur during sampling

    • Selection Bias - sample not representative of the population
    • Undercoverage Bias - Some groups of the population are inadquately represented in the sample
    • Survivorship Bias - It's a logic error that only choose the sample that supports the conclusion and skip those didn't work. this tend to conclusion error.

  • References

  • Comprehensive and Practical Statistics Guide for Data Science - A real good one!

    • Sample Distribution and Population Distribution, Central Limit Theorem, Confidence Interval
    • Hypothesis Testing
    • ANOVA (Analysis of Variance), continuous and categorical variables, ANOVA also requires data from approximately normally distributed populations with equal variances between factor levels.
    • Chi-square test, categorical variables
    • Regression and ANOVA, it is important to know the degree to which your model is successful in explaining the trend (variance) in dependent variable. ANOVA helps finding the effectiveness of regression models.
    • An example of hypothesis test with chi-square:
      • chi-square tests the hypothesis that A and B are independent, that is, there is no correlation between them. Chi-square is used to calculate the correlation between categorical variables
      • In this example, you have already calculated chi-square value as 507.93
      • B feature has 2 levels, "science-fiction", "history"; A feature has 2 levels, "female", "male". So we can form a 2x2 table. The degree of freedom = (2-1)*(2-1) = 1
      • Use the calculator here to calculate significant level, type degree of freedom as 1, probability as 0.001 (you can choose a probability you'd like). The calculated significant level is 10.82756617
      • chi-square value 507.93 is much larger than the significant level, so we reject the hypothesis that A, B are independent and not correlated

  • Hypothesis Testing

    • https://www.analyticsvidhya.com/blog/2017/01/comprehensive-practical-guide-inferential-statistics-data-science/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+AnalyticsVidhya+%28Analytics+Vidhya%29
    • When we want to get more accuracy results from data samples, we often try to:
      • Increase sample size
      • Test more other random samples
    • But what if you don't have time/money/computer memory to do above methods, we can try to calculate random chance probability
    • Relevant Basics First
      • z-value/z-score: It tells how many standard deviation the observed value is away from mean. For example, z-score = 1.7 indicates that the observed value is 1.7 standard deviation away from mean. Associated with the standard normal distribution only. z-score = (observed_value - mean)/standard_deviation
      • p-value: it tells the probability that observed value is away from mean. Therefore, in order to get p-value, you check z-table with the z-score you got. Associated with the standard normal distribution only.
        • standard deviation of sample means = population standard deviation/sqrt(N), N is the sample size. This is knows as standard error of means. You check this satdard error of means to tell how accurate your random samples that can determind population mean.
        • Greater the sample size, lower standard error of means, and more accuracy to use the sample mean to determine the population mean. Sample size should be NOT more than 10% of the population when sampling is done without replacement.
        • All the samples should be fixed size
        • Sample size must be sufficient, >= 30. When the population is skewed or asymmetric, the sample size should be large. If the population is symmetric, then we can draw small samples as well.
        • For other distributions, z-score = (sample_mean - population mean)/sample_standard_deviation = (sample_mean - population mean)/population standard deviation/sqrt(N), therefore you can also get p-value to show the probability. But the precondition is, you know the population standard deviation, population mean
      • Significant Level & Confidence - no matter what distribution, finally you get a probability (p-value) of how much the sample mean is higher than population mean (because you use sample mean minus population mean). High probability shows the acceptance of randomness, while low probabilty shows the behaviour differences. With Significant Level, we can decide what is high probability and what is low probability
        • For example, with significant level 5%, also known as α level, if the probability you got is less than 5%, it means there is behaviour of difference between 2 population. Then we can say, we have 95% confidence that the sample mean is not driven by randomness, but driven by behaviour difference.
    • Steps to do Hypothesis Testing
      • NULL HYPOTHESIS (H0) - The difference between sample and population mean is due to randomness (also means there is no difference between sample and population)
      • ALTERNATIVE HYPOTHESIS (H1)
      • After you got the p-value, if it's higher than significant level (say, 5%), reject NULL Hypothesis
        • The decision to reject the null hypothesis could be incorrect, it is known as Type I error.
        • The decision to retain the null hypothesis could be incorrect, it is know as Type II error.
      • One Tail Test - your alternate hypothesis is greater or less than the population mean so it is also known as Directional Hypothesis test
      • If you are not sure, it's Two Tail Test/Non Directional Hypothesis. In two tail test, real significant level for each direction = significant level/2
      • In one tail test, we reject the Null hypothesis if the sample mean is either positive or negative extreme any one of them. But, in case of two tail test we can reject the Null hypothesis in any direction (positive or negative).
    • reference: https://www.analyticsvidhya.com/blog/2015/09/hypothesis-testing-explained/

  • Non-Parametric Tests

    • Parametric tests are used when the information about the population parameters is completely known whereas non-parametric tests are used when there is no or few information available about the population parameters. That is to say, non-parametric tests make no assumption about the data.
    • But, if there is a parametric solution to the problem, using non-parametric tests could lead to much lower accuracy
    • Pros for non-parametric tests:
      • When there is no parametric test solution, using non-parametric test is better, and it works even when the data sample is small
      • It also works for all data types, such as categorical, ordinal, interval or data with outliers
    • Cons for non-parametric tests:
      • Because non-parametric methods won't reduce the problem of estimating f to a small number of params as parametric methods do, so in order to achieve higher accuracy, non-parametric methods need much larger number of observations
      • Critical value tables for non-parametric tests are not inlcuded in many software packages...
    • Steps to do Hypothesis Testing with non-parametric tests
      • H0 (NULL Hypothesis): There is no significant difference between sample mean and population mean
      • H1 (ALTERNATIVE Hypothesis): There is a significant difference between sample mean and population mean
      • Set significant level, decide it's one-tail or two-tail problem. Decide decision rule (when t reject null hypothsis)
      • Test Statistics - In non-parametric tests, the sample is converted into ranks and then ranks are treated as a test statistic
      • Compare test statistics and decision rule, to decide reject/accept null hypothesis
    • Different Types of non-parametric tests
      • Mann Whitney U test/Mann Whitney Wilcoxon test/Wilcoxon rank sum test
        • It is an alternative to independent sample t-test
        • You have 2 samples, sample one has size n1, sample two has size n2. R1, R2 are the sum of ranks for each sample respectively.
          • In order to calculate R1, R2, you arrange the values of both samples together, in ascending order (from left to right), and you give them 1,2,3...,(n1+n2) index.
            • For those values such as X that appeared multiple times, say p times, you use the sum of index of X values devided by p (sum(index_of_this_same_value)/p) as the new index for each X, in this way, you can make sure the sum of all the final idex equals to n*(n+1)/2, it also equals to (1+2+3+...+n)
        • U1 = n1*n2 + n1*(n1+1)/2 - R1, U2 = n1*n2 + n2*(n2+1)/2 - R2
        • U = min(U1, U2), U1+U2 = n1n2. U lies in range (0, n1n2). 0 means the 2 samples are totally different from each other, while n1*n2 indicates some relationship between the 2 groups.
        • Find critical value here: http://ocw.umb.edu/psychology/psych-270/other-materials/RelativeResourceManager.pdf
        • If U <= critical value, reject H0, else accept H0
      • Wilcoxon Sign-Rank Test
        • There are n values in each group. You calculate the difference between the 2 values that share the same index in both groups
        • Then order the difference by ignoring their positive/neagtive sign. Ranking them, using the same method used in Mann Whitney U test, that for same absolute value, the rank is the average of their initial ranks
        • Now add their original positive/negative sign to the ranks
        • W1 = sum(positive ranks), W2 = abs(sum(neagtive ranks))
        • W = min(W1, W2)
        • Choose critical value based on α and one-tail/two-tail here: http://www.stat.ufl.edu/~winner/tables/wilcox_signrank.pdf
        • If W <= critical value reject H0, else accept H0
      • Sign Test
        • This method is quite similar to Wilcoxon Sign-Rank Test. The difference is, after you got signed ranks, you ignore the values, but only focus on +, - signs.
        • W1 = number of + sign, W2 = number of - sign
        • W = min(W1, W2)
        • Still choose critical value based on α and one-tail/two-tail here: http://www.stat.ufl.edu/~winner/tables/wilcox_signrank.pdf
        • If W <= critical value reject H0, else accept H0
      • Kruskal-Wallis Test
        • This test is extremely useful when you are dealing with more than 2 independent groups and it compares median among k populations
        • This test is an alternative to One way ANOVA when the data violates the assumptions of normal distribution and when the sample size is too small.
        • The way you do rankings is almost the same as the above methods
        • H = (12/n*(n+1) * sum(pow(R1, 2)/n1 + pow(R2, 2)/n2 +... + pow(Rk, 2)/nk)) - 3*(n-1), n is the total sample size, k is the number of samples, Ri is the sum of ranks of ith sample, ni is the sample size of ith sample
        • Table of critical values: http://webspace.ship.edu/pgmarr/Geo441/Tables/Kruskal-Wallis%20H%20Table.pdf
        • If H >= critical value, reject H0, else accept H0
      • Spearman Rank Correlation
        • If there is correlation between 2 datasets, use this method
        • H0: There is no relation between the 2 groups; H1: there is relation between the 2 groups
        • Spearman Rank Correlation should be used to check the strength and direction of the correlation. Spearman rank correlation is a non-parametric alternative to Pearson correlation coefficient.
        • I will use built-in packages or online tools to calculate this value...
        • Check critical value here: http://webspace.ship.edu/pgmarr/Geo441/Tables/Spearman%20Ranked%20Correlation%20Table.pdf
        • degree of freedom = n-2, n is the sample size. If your calculated result >= critical value, reject H0, else accept
    • reference: https://www.analyticsvidhya.com/blog/2017/11/a-guide-to-conduct-analysis-using-non-parametric-tests/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+AnalyticsVidhya+%28Analytics+Vidhya%29
      • For Spearman Rank Correlation, it seems that the critical value at freedom of degree 8 with α = 0.025 (2 tail) is larger than 0.67, so it should accept the null hypothesis

  • Probability cheat sheet: http://www.cs.elte.hu/~mesti/valszam/kepletek

  • Likelihood vs Probability: http://mathworld.wolfram.com/Likelihood.html

    • Likelihood is the hypothetical probability that a past event would yield a specific outcome.
    • Probability refers to the occurrence of future events, while Likelihood refers to past events with known outcomes.

  • Probability basics with examples

    • binonial distribution: a binomial distribution is the discrete probability distribution of the number of success in a sequence of n independent Bernoulli trials (having only yes/no or true/false outcomes).
    • The normal distribution is perfectly symmetrical about the mean. The probabilities move similarly in both directions around the mean. The total area under the curve is 1, since summing up all the possible probabilities would give 1.
    • Area Under the Normal Distribution
    • Z score: The distance in terms of number of standard deviations, the observed value is away from the mean, is the standard score or the Z score. Observed value = µ+zσ [µ is the mean and σ is the standard deviation]
    • Find Z Table here

  • Very Basic Conditional Probability and Bayes Theorem

    • Independent, Exclusive, Exhaustive events
    • Each time, when it's something about statistics pr probability, I will still read all the content to guarantee that I won't miss anything useful. This one is basic but I like the way it starts from simple concepts, using real life examples and finally leads to how does Bayes Theorem work. Although, there is an error in formula P (no cancer and +) = P (no cancer) * P(+) = 0.99852*0.99, it should be 0.99852*0.01
    • There are some major formulas here are important to Bayes Theorem:    * P(A|B) = P(A AND B)/P(B)
      • P(A|B) = P(B|A)*P(A)/P(B)
      • P(A AND B) = P(B|A)*P(A) = P(A|B)*P(B)
      • P(b1|A) + P(b2|A) + .... + P(bn|A) = P(A)

  • Dispersion - In statistics, dispersion (also called variability, scatter, or spread) is the extent to which a distribution is stretched or squeezed. Common examples of measures of statistical dispersion are the variance, standard deviation, and interquartile range.

  • Common Formulas

  • Linear regression line attempts to minimize the squared distance between the points and the regression line. By definition the ordinary least squares (OLS) regression tries to have the minimum sum of squared errors. This means that the sum of squared residuals should be minimized. This may or may not be achieved by passing through the maximum points in the data. The most common case of not passing through all points and reducing the error is when the data has a lot of outliers or is not very strongly linear.

  • Person vs Spearman: Pearson correlation evaluated the linear relationship between two continuous variables. A relationship is linear when a change in one variable is associated with a proportional change in the other variable. Spearman evaluates a monotonic relationship. A monotonic relationship is one where the variables change together but not necessarily at a constant rate.

  • Coefficient

  • Linear Algebra with Python calculations

  • To comapre the similarity between 2 curves

    • Try to compare from these perspectives:
      • Distance
      • Shape
      • Size of the Area in between
    • Kolmogorov–Smirnov test - distance based
      • its null hypothesis this, the smaple is drawn from the reference graph
      • So, if the generated p-value is smaller than the threshold, reject null hypothesis, which means the 2 curves are not similar
    • Dynamic Time Wrapping (DTW) - distance based
    • Check the consistency of Peak and non-peak points of the 2 curves - shape based

  • Simulation Methods

  • Saddle Points

    • The saddle point will always occur at a relative minimum along one axial direction (between peaks) and at a relative maximum along the crossing axis. Saddle Point Wiki
    • How to Escape Saddle Point Efficiently
      • Strict saddle points vs Non-strict saddle points: non-strict saddle points can be flat in the valley, strict saddle points require that there is at least one direction along which the curvature is strictly negative
      • GD with only random initialization can be significantly slowed by saddle points, taking exponential time to escape. The behavior of PGD (projected GD) is strikingingly different — it can generically escape saddle points in polynomial time.
      • Difference between Projected Gradient Descent (PGD) and Gradient Descent (GD)

  • How to calculate Confidence: http://www.stat.yale.edu/Courses/1997-98/101/confint.htm

    • For Unknown Population Mean and Known Standard Deviation
    • For Unknown Population Mean and Unknown Standard Deviation - t-distribution, standard error (SE)
    • Both assumes the distribution is bell shape (normal distribution or t-distribution)
    • When there are only 2 datasets to compre, we can use t-distribution, but when there are more than 2, t-distribution won't handle it. Try ANOVA.

  • ANOVA

    • It compares samples based on means. It can handle 2+ samples. By comparing the means, ANOVA checks the impact of one or more factors.
    • When there are just 2 samples, t-distribution and ANOVA get same results
    • Terminology
      • Grand Mean µ: The mean of all sample means
      • Null Hypothesis in ANOVA is valid, when all the sample means are equal, or they don't have any significant difference
      • Alternative Hypothesis is valid when at least one of the sample mean is different from others. But you ay need to use other methods to tell which is the different sample mean
      • Between-group variability
        • It tells the variation between the distributions of samples. If the distributions are close or overlap, the grand mean will be similar to sample means; but if the distributions are further away, the difference between the grand mean and the sample means can be large
        • (n1*pow(µ1-µG,2) + n2*pow(µ2-µG,2) + n3*pow(µ3-µG,2) + .... + nk*pow(µk-µG,2))/(k-1) = SSbetween/(k-1)
        • To calculate Between-group variability, is quite similar to calculate standard deviation/variance
          • How to calculate standard deviation: https://www.youtube.com/watch?v=pFGcMIL2NVo
          • Variance = pow(standard deviation, 2)
          • The difference between calculating standard deviation and Between-group variability
            • There are weights for each pow((µi-µG),2), and the weight is ni, the sample size of group i. µi is the sample mean for group i, and µG is the grand mean
            • The sum of those squared sample mean difference are divided by degrees of freedom. If there are k (µi-µG), then the degree of freedom is k-1
            • Finally, you don't need to calculate squared root
      • Within-group variability
        • It's about the variation in each sample, and sum them up. This is because a group of samples that overlap a lot and can have the same sample mean & grand mean with another group of samples which have no oberlap. Between-group variability cannot help, we need within-group variability too.
        • sum(pow(xij-µj, 2))/(N-k) = SSwithin/(N-k), µj is the sample mean of sample j, xij is the ith value in sample j. N-k is the degree of freedom, N is the sum of sizes, k is the number of samples. You can think it's minus 1 from each sample, so that's k
      • F-Ratio/F-Statistics
        • It measures whether the means of different samples are significantly different or not. Lower F-Ration indicates more similar the samples and we cannot reject the null hypothesis
        • F = Between group variability / Within group variability
          • This tells us, the larger the between-group variability, the more likely those samples are different
          • When F-Ratio > F-critical α, which means it's lands in the critical region, we reject null hypothesis, and say the differences between samples are significant. Also, when you are using tools such as Excel, it generates p-value at the same time, if this p-value is lower than α, we reject null hypothesis too
          • For different α, F-critical value can be found here: http://www.socr.ucla.edu/applets.dir/f_table.html
            • df means degree of freedom here. df1 is between group degree of freedom, df2 is within group degree of freedom
        • Unlike z-distribution, t-distribution, F-distribution does not have any negative values
      • One-way ANOVA
        • The limitation of one-way ANOVA: cannot tell which sample is different
        • To find the different group, you compare each pair of the groups
        • eta square = SSbetween/SStotal, SStotoal = SSwithin * (N-k)/(N-1), eta square is used to measure how much between group difference contributes to the variability. It also helps tell whether the independent variablies really have an impact on dependent variable
        • Make sure there is only one variable affects the results
      • Two-Way ANOVA
        • When there can be 2 factors/variables affect the results, this ANOVA method measures which variable affects more
        • 2 null hypothesis, each for one variable, 1 alternative hypothesis
        • You compare F-ratio & F-critical value or p-value & α value, to decide whether to reject the 2 null hypothesis
        • There is also F-ratio and F-critical value for interaction, to measure whether there is combined effect from the 2 variables. When F-ratio < F-critical value, no combined effect
      • Multi-variate ANOVA (MANOVA)
        • Works when there are multiple variables
        • It cannot tell which variable affects the results, so you need multiple one-way ANOVA summary to figure out the variable that affects the result
      • Bonferroni approach: Methods used to find samples that represent the population
      • Tukey’s HSD : https://www.sfu.ca/~jackd/Stat302/Wk04-2_Full.pdf
      • Least significant difference test (LSD): https://www.utd.edu/~herve/abdi-LSD2010-pretty.pdf
    • Reference: https://www.analyticsvidhya.com/blog/2018/01/anova-analysis-of-variance/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+AnalyticsVidhya+%28Analytics+Vidhya%29
      • The author is giving detailed Excel methods
      • In its one-way ANOVA, finally the author found group A is quite different from group B, C. But I don't think this difference could prove constant music can really improve the performance. If students in group A are much better students then group B, C, group A can still be quite different. I think the final result only indicates group A is quite different from group B, C. If they want to prove constant music could really improve performance, how about ask the same group, say group A, to test under variable music/no sound situation first, then test in constant music situation and check how different the results are. In a word, try to make sure there is only one variable

Machine Learning Algorithms

  • KNN with R example: https://www.analyticsvidhya.com/blog/2015/08/learning-concept-knn-algorithms-programming/

  • SVM with Python example: https://www.analyticsvidhya.com/blog/2015/10/understaing-support-vector-machine-example-code/?utm_content=buffer02b8d&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer

  • Basic Essentials of Some Popular Machine Learning Algorithms with R & Python Examples: https://www.analyticsvidhya.com/blog/2015/08/common-machine-learning-algorithms/?utm_content=buffer00918&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer

    • Linear Regression: Y = aX + b, a is slope, b is intercept. The intercept term shows model prediction without any independent variable. When there is only 1 independent variable, it is Simple Linear Regression, when there are multiple independent variables, it is Multiple Linear Regression. For Multiple Linear Regression, we can fit Polynomial Courvilinear Regression.
    • When to use Ridge or Lasso: In presence of few variables with medium / large sized effect, use lasso regression. In presence of many variables with small / medium sized effect, use ridge regression. Lasso regression does both variable selection and parameter shrinkage, whereas Ridge regression only does parameter shrinkage and end up including all the coefficients in the model. In presence of correlated variables, ridge regression might be the preferred choice. Also, ridge regression works best in situations where the least square estimates have higher variance.
    • Logistic Regression: it is classification, predicting the probability of discrete values. It chooses parameters that maximize the likelihood of observing the sample values rather than that minimize the sum of squared errors (like in ordinary regression).
    • Decision Tree: serves for both categorical and numerical data. Split with the most significant variable each time to make as distinct groups as possible, using various techniques like Gini Gain, Information Gain, Chi-square. A decision tree algorithm is known to work best to detect non – linear interactions. The reason why decision tree failed to provide robust predictions because it couldn’t map the linear relationship as good as a regression model did.
      • Gini Gain vs Information Gain vs Gain Ratio
        • They are all attribute selection methods in decision tree
        • Gini Gain forces the resulting tree to be binary
          • Gini Impurity = sum(P(i) * (1 - P(i))), i is class i
            • Gini impurity is the probability of a random sample being classified correctly if you randomly pick a label according to the distribution in the branch.
          • Gini Gain = Gini_Impurity_All - Gini_Impurity_Attribute
            • Gini_Impurity_All calculates gini impurity at dependent variable level without checking any attribute
            • Gini_Impurity_Attribute calculates gini impurity for that attribute, so the dependent variable counts are counted under each value group of the attribute
          • When the split is pure, it's 0 Gini Impurity
        • Information Gain allows multiway splits.
          • Entropy = -sum(P(i)*log(P(i), 2)), i is class i
            • Entropy calculates how lack of purity the labels are after the split. Information gain is the opposite, it represents the purity of labels after the split.
          • Information Gain = Entropy_All - Entropy_Attribute = 1 - Entropy
            • Entropy_All calculates entropy at dependent variable level without checking any attribute
            • Entropy_Atribute calculates entropy for that attribute, so the dependent variable counts are counted under each value group of the attribute
        • Information Gain has bias towards the attributes that have larger amount of different values, since this will lead to higher number of branches and each branch is pure. This could make the algorithm useless. To overcome this problem, Gain Ratio has been used.
          • Gain Ratio = Information Gain of attribute/ Split Entropy of attribute
          • Comparing Entropy for attribute A and Split Entropy for attribute A:
            • Entropy(A) = -sum(P(Dj|D)*log(P(D), 2))
            • Split_Entropy(A) = -sum((Dj|D)*log(P(Dj|D), 2))
      • About Entropy
        • We all know that formula, -sum(p(i)*log(p(i), 2)), and this is calculated for each group/category/etc. in the whole dataset
        • Sometimes, we need to compare entropy for different datasets, and in each dataset, we have multiple groups, then here comes normalized entropy, for each group, -sum(p(xi)*log(p(xi), 2)/log(n, 2)), n means the total records in this group, NOT overall records of the dataset
      • Besides entropy and gini can be used to split decision trees, chi square can be used for the split of the categorical variables in the decision tree. Higher chi square, higher the different between partent and child node and higher homogeneity.
    • SVM: seperate groups with a line and maximize the margin distance. Good for small dataset, especially those with large number of features
      • The effectiveness of SVM depends on Selection of Kernel, Kernel Parameters and Soft Margin Parameter C, they can also help reduce errors and overfitting
      • The larger C parameter, the heavier penality for misclassification. So when C param is set to infinite, no room for error, optimal hyperplane (if exists) should seperate the data completely
      • Gamma value indicates how far the influence points to the hyperplane. Higher the Gamma value is, closer the influence points are
    • Naive Bayes: the assumption of equally importance and the independence between predictors. Very simple and good for large dataset, also majorly used in text classification and multi-class classification. Likelihood is the probability of classifying a given observation as 1 in presence of some other variable. For example: The probability that the word ‘FREE’ is used in previous spam message is likelihood. Marginal likelihood is, the probability that the word ‘FREE’ is used in any message.
    • KNN: can be used for both classification and regression. Computationally expensive since it stores all the cases. Variables should be normalized else higher range variables can bias it. Data preprocessing before using KNN, such as dealing with outliers, missing data, noise
    • K-Means
    • Random Forest: bagging, which means if the number of cases in the training set is N, then sample of N cases is taken at random but with replacement. This sample will be the training set for growing the tree. If there are M input variables, a number m<<M is specified such that at each node, m variables are selected at random out of the M and the best split on these m is used to split the node. The value of m is held constant during the forest growing. Each tree is grown to the largest extent possible. There is no pruning. Random Forest has to go with cross validation, otherwise overfitting could happen.
    • PCA: Dimensional Reduction, it selects fewer components (than features) which can explain the maximum variance in the data set, using Rotation. Personally, I like Boruta Feature Selection. Filter Methods for feature selection are my second choice. Remove highly correlated variables before using PCA
    • GBM (try C50, XgBoost at the same time in practice)
    • Difference between Random Forest and GBM: Random Forest is bagging while GBM is boosting. In bagging technique, a data set is divided into n samples using randomized sampling. Then, using a single learning algorithm a model is build on all samples. Later, the resultant predictions are combined using voting or averaging. Bagging is done is parallel. In boosting, after the first round of predictions, the algorithm weighs misclassified predictions higher, such that they can be corrected in the succeeding round. This sequential process of giving higher weights to misclassified predictions continue until a stopping criterion is reached.Random forest improves model accuracy by reducing variance (mainly). The trees grown are uncorrelated to maximize the decrease in variance. On the other hand, GBM improves accuracy my reducing both bias and variance in a model.

  • Online Learning vs Batch Learning: https://www.analyticsvidhya.com/blog/2015/01/introduction-online-machine-learning-simplified-2/

  • Optimization - Genetic Algorithm

  • Survey of Optimization

  • Optimization - Gradient Descent

    • Reference: https://www.analyticsvidhya.com/blog/2017/03/introduction-to-gradient-descent-algorithm-along-its-variants/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+AnalyticsVidhya+%28Analytics+Vidhya%29
    • Challenges for gradient descent
      • data challenge: cannot be used on non-convex optimization problem; may end up at local optimum instead of global optimum; may even not an optimal point when gradient is 0 (saddle point)
      • gradient challenge: when gradient is too small or too large, vanishing gradient or exploding gradient could happen
      • implementation chanllenge: memory, hardware/software limitations
    • Type 1 - Vanilla Gradient Descent
      • "Vanilla" means pure here
      • update = learning_rate * gradient_of_parameters
      • parameters = parameters - update
    • Type 2 - Gradient Descent with Momentum
      • update = learning_rate * gradient
      • velocity = previous_update * momentum
      • parameter = parameter + velocity – update
      • With velocity, it considers the previous update
    • Type 3 - ADAGRAD
      • ADAGRAD uses adaptive technique for learning rate updation.
      • grad_component = previous_grad_component + (gradient * gradient)
      • rate_change = square_root(grad_component) + epsilon
      • adapted_learning_rate = learning_rate * rate_change
      • update = adapted_learning_rate * gradient
      • parameter = parameter – update
      • epsilon is a constant which is used to keep rate of change of learning rate
    • Type 4 - ADAM
      • ADAM is one more adaptive technique which builds on adagrad and further reduces it downside. In other words, you can consider this as momentum + ADAGRAD.
      • adapted_gradient = previous_gradient + ((gradient – previous_gradient) * (1 – beta1))
      • gradient_component = (gradient_change – previous_learning_rate)
      • adapted_learning_rate = previous_learning_rate + (gradient_component * (1 – beta2))
      • update = adapted_learning_rate * adapted_gradient
      • parameter = parameter – update
    • Tips for choose models
      • For rapid prototyping, use adaptive techniques like Adam/Adagrad. These help in getting quicker results with much less efforts. As here, you don’t require much hyper-parameter tuning.
      • To get the best results, you should use vanilla gradient descent or momentum. gradient descent is slow to get the desired results, but these results are mostly better than adaptive techniques.
      • If your data is small and can be fit in a single iteration, you can use 2nd order techniques like l-BFGS. This is because 2nd order techniques are extremely fast and accurate, but are only feasible when data is small enough

  • Factorization Machines

    • Large dataset can be sparse, with Factorization, you can extract important or hidden features
    • With a lower dimension dense matrix, factorization could represent a similar relationship between the target and the predictors
    • The drawback of linear regression and logistic regression is, they only learn the effects of all features individually, instead of in combination
    • For example, you have Fields Color, Category, Temperature, and Features Pink, Ice-cream, Cold, each feature have different values
      • Linear regression: w0 + wPink * xPink + wCold * xCold + wIce-cream * xIce-cream
      • Factorization Machines (FMs): w0 + wPink * xPink + wCold * xCold + wIce-cream * xIce-cream + dot_product(Pink, Cold) + dot_product(Pink, Ice-cream) + dot_product(Cold, Ice-cream)
        • dot-product: a.b = |a|*|b|cosθ, when θ=0, cosθ=1 and the dot product reaches to the highest value. In FMs, dor product is used to measure the similarity
        • dot_product(Pink, Cold) = v(Pink1)*v(Cold1) + v(Pink2)*v(Cold2) + v(Pink3)*v(Cold3), here k=3. This formula means dot product for 2 features in size 3
      • Field-aware factorization Machines (FFMs)
        • Not quite sure what does "latent effects" meantioned in the tutorial so far, but FFMs has awared the fields, instead of using dot_product(Pink, Cold) + dot_product(Pink, Ice-cream) + dot_product(Cold, Ice-cream), it's using Fields here, dot_product(Color_Pink, Temperature_Cold) + dot_product(Color_Pink, Category_Ice-cream) + dot_product(Temperature_Cold, Category_Ice-cream), Color & Temperature, Color & category, Temperature & Category
    • xLearn library
      • Sample input (has to be this format, libsvm format): https://github.com/aksnzhy/xlearn/blob/master/demo/classification/criteo_ctr/small_train.txt
      • Detailed documentation about parameters, functions: http://xlearn-doc.readthedocs.io/en/latest/python_api.html
      • Personally, I think this library is a little bit funny. First of all, you have to do all the work to convert sparse data into dense format (libsvm format), then ffm will do the work, such as extract important features and do the prediction. Not only how it works is in the blackbox, but also it creates many output files during validation and testing stages. You's better run evrything through terminal, so that you can see more information during the execution. I was using IPython, totally didin't know what happened.
      • But it's fast! You can also set multi-threading in a very easy way. Check its documentation.
    • My code: https://github.com/hanhanwu/Hanhan_Data_Science_Practice/blob/master/Factorization_Machines.ipynb
      • My code is better than reference
    • Reference: https://www.analyticsvidhya.com/blog/2018/01/factorization-machines/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+AnalyticsVidhya+%28Analytics+Vidhya%29

Data Visualization

Big Data

  • Dask - Machine Learning in Parallel
    • It copies many functions from numpy, sklearn, pandas and even Spark, aiming at allow you to do parallel data science work.
    • Dask Documentation; http://dask.pydata.org/en/latest/
    • My code: https://github.com/hanhanwu/Hanhan_Data_Science_Practice/blob/master/try_dask.ipynb
      • Cannot say I'm a big fan of Dask now.
      • For data preprocessing, it's no better than using pandas and numpy, since they are much faster than dask and have more functions. I tried, it took me so much time and finally decided to change back to numpy & pandas. However, sometimes the data file can be too large for pandas to load, Dask will help you manage the memory into chunks, so you can use Dask to load the giant data file.
      • But you can convert the preprocessed data into dask dataframe
      • The parallel processing for machine learning didn't make me feel it's much faster than sklearn.
      • Although dask-ml mentioned it supports both sklearn grid search and dask-ml grid search, but when I was using sklearn grid search, it gave large amount of error and could not tell what caused the error.
      • I think for larger dataset, Spark must be faster if its machine learning supports the methods. We can also convert pandas dataframe to Saprk dataframe to overcome the shortage of data preprocessing functions.

Build Machine Learning Platform

Cloud

TEXT ANALYSIS

Non-Machine Learning Data Analysis Examples

AI

Experiences/Suggestions from Others

Data Science Skillset Tests

Interview Tips

TRAIN YOUR BRAIN

OTHER

About

more data science resources

Resources

Readme

License

CC-BY-SA-4.0 license
Activity

Stars

14 stars

Watchers

6 watching

Forks

13 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages