-
Notifications
You must be signed in to change notification settings - Fork 2
Data class Jan 16 Through 24
Jan 16 CLASS
HOW TO MAKE A FUNCTION
My_Function <- function(string)
{ return(data) }
BASIC CODE GIVEN
input_to_data <- function () { con <- file("stdin") data <- (readLines(con, warn = FALSE) ) close(con) return(data) }
string_to_vector <- function (x) { return(as.numeric(unlist(strsplit(x, split=" ")))) }
data <- input_to_data() n <-(string_to_vector(data))
F= strrep(“hi”,5)
// prints it 5 times sort(vector_name, decreasing=”False”) // sorts a vextor
Example in practice
for (i in 1:n) { spaces <- strrep(" ", n-i) stairs <- strrep("#", i) cat(paste(spaces, stairs), sep = "\n") }
======
tail(data,n) // // tail returns the last 5 items of a problem
%% is the remainder If gr = [73.67.48.33]
So vv <- gr %% 5 // [3,2,3,3] Pass <- gr > 37 [T,T,T,F] Round
Jan 17th class Programming Fundamentals :
Data Types Int, float, strings, vectors, data frames, key value pairs, bool, None, NaN ( for nums that cant be e.g. divide by 0), None etc Images are matrixes or tensors and occasionally strings jpgs NOTE A vector must be all of one type A data frame is a collection of data frames Key Value Pairs for example key a holds 1 and key b holds foo It is a key value pair {‘Key’: ‘item’, ‘a’:1, b: “hope”,...}
We access a key pair this way my_d[‘a’]=1 in key value pair
With the vector vv = [1,2,3,4,5] We accessed
- END NOTE*
(2) FUNCTIONS
My_f <- function ()
{ stuff }
def my_f() : #stuff
(3) operators +, -, /, *, %, ** (exp), &, !, is , != And, or, not, ==,
(4) & (5) control flor (4) python :
For i in range (12): For ( i in 1:10):
(5) if else Python has elif
If ( something is tru): Do this Else do that
(6) libraries # need to know how to pull in a library In R: E.g. library(repr) # now we can use repr In Python: Import numpy
You can also do a shortcut version Import numpy as no
(7) objects // you dont need to know how to make a class but you do need to know how to work with an object // Anything that exists in memory, that is loaded and you have given a name to it, then it is an object
Objects : have attributes. Note R like to use these key value pairs to define its objects.
GGplot2 is everyone’s favorite R library
Git Git standard workflow 1 git add commit cycle { Do work → add → commit}
Work Add Commit O ---------------> O -----------------> O The lines are edges, the nodes are the circles Github is based on graph theory and it is intended to be treated that way
The XY graphs we grew up with are plots, a bar chart is a scatter plot
Note main head or node of your code is called master So you branch and work on a Dev
Work Add Commit
Master O ---------------> O -----------------> O-------- ONew stuff someone I did
O New stuff someone else did
(c)Sometimes you like what he did so you rebase your code when he puts it back to master
Work Add Commit
Master O ---------------> O -----------------> O-------- ONew stuff someone I did
O New stuff someone else did
(d) We forked which means we made a coppy
Master O ---------------> O -----------------> O-------- O | | | Master O ---------------> O -----------------> O-------- O // our code is diff
// when jupyter looks at a notebook all is sees is a json file
Git config - l tells you the config of your project
Step 1) cp -R 01-linear(tab complet)/ 01-Line(tabcomplete So it is cp -R Step 2) Step 3) git checkout ORIGINIAL NAME # THROWS AWAY CHANGES Git checkout 01-linear(tab complete) // Instead of doing work, Step 4) git status // we should see untracked files # red output 01-linear_algebra
Extra option if you have work you did Step 5) git log Outputs a sha or a secure hash Which looks like 0fef…. Copy it Step 6) Git reset sha ( paste 9t) Non Extra option (7) git remote -v (8) git remote add upstream https://github.com/original_owner/original_repo (9) git pull upstream master // note if you just do git pull it automatically does git pull from origin master
Yay we did it. Now if we want to download and install our thing cp -R 02-intro_to_r/ 02-intro_to_r_Nathan/ Git commit -m ‘latest’
*** note *** rm-rf is dangerous Never do sudo rm-rf/ # this deletes your entire system from the root
Vector space = if you do a linear combination on a vector you are still in the same vector space. If you do a linear combo in R 2 you are still in R2
Length = # of element Magnitude = square root of (x1^2 + x2^2,….Xn^2)^(0.5) // // works for any type of distance
The function norm gives you the magnitude We specify norm of type 2 to get the vector Called the L2 norm, what we used norm( vector name, type=”2”) // note type 2 is the euclidean distance // we might use a different distance formula, like manhattan distance
We do square root in r
|| a|| = sqrt(a[1]**2 + b[1]**2 )
So if degrees were 90, it would be orthogonal , cos theta = 0 So <u,v> = uv cos theta= <u,v> = uv *0=0 So they would be nothing alike
In nlp we can find cosine similarity Cos theta = < u,v> div ( u.v) <u,v> = uv cos theta
<u,v>= Sum u sub i time uvi x %*% y
So norm(a, type=”2”) To write it ourselves sqrt(a %*% b)
Transpose Collumn vectors in a data fram are variables of the same type. E.g. if measuring ages, then all the ages are one collumn If measuring salary then all are stored in a collumn Since it is a data frame and not a matrix, they do not have to be the same
Row vectors in a data frame are a measurement E.g. Bob is 6’2 make 40k and is single That is one row
Transpose lets you change between rows and collums Transpose of a row is a collumn the transpose of a collumn is a row a^T is what transpose looks like when written
PAAACCTTT NEVER TAKE A TRANSPOSE OF SOMETHING THAT IS NOT NUMERIC PPAACCCTTT
When it is a collumn it has commas (e, f, g, h) = (e F G h)
When it is a row it has no collumns (e f g h )
For example A = (e, f, g, h) t(A) = transpose A = ( e f g h )
Ask about projections negative V sub || i = i* (v.i) / i.i Decomposition into parallel & into perpendic
If they watch first five mins, never finish and remove from list you could view it as negative
Y = 3x - 2 y= -5x + 6
x2 = 3x1 - 2 x2= -5x1 + 6
-3x1 + +x2 = -2 5x1 + x2 = 6
(-3 1) (x1 = -2 x2) (5 1) (x1 = 6 x2)
( -3 1 ( x1 = (-2 5 1) x2) 6) A x = b Ax = b
A inv A = I A inv A x = A inv b I x = A inv b x = A inv b
R DOES THIS FOR US USING SOLVE Solve solves the matrix for us
Note R prefers collumns Just like our Prof And Python prefers rows
Y = mx + b
Y = b+ mx Y = B0 +b1 x1 // this is a dot product Works for any xi like Y = B0 +bi xi
B0 + B1 X1 = Yi
First point B0 + Bi . -3 = 4 B0 + Bi . 2 = -3
(1 -3 (B0 = (4 1 2) B1) -3)
X is a data frame
X B = y
Capitals for matrixes Bar and lowercase for vectors
We can solve this in r So solve (X,y) give us B
N> P is super collumn. An over determined system. 10 variables in the census and 330 million people
P>N happens in places in places like the human genome
jAN 22ND CLASS Git pull origin master
cp -R 03-vectors_in_R/ 03-vectors_in_R_IsaacCad/ Changes name of the file Copies - Recursively original filename/ copy filename
Statistical learning textbooks So we divide most models into supervised learning and unsupervised learning Supervised Unsupervised Regression, classification clustering Has a target ( a val we try To predict like a value or a class)
We ask ourselves these questions ( in both orders What is a good benchmark for my model? What is a good metric for for measuring the success of my model?)
We can compare a regression model to a naive benchmark.
For example we can compare to the mean
#Plotting, vectors data frames are all core Matplotlib gives us plotting Numpy gives us vectors So pandas gives us data frames
%matplotlib inline
A well defined data science task has
Domain¶ This problem is drawn from the analysis of population demographic data. Problem Statement Given certain information about males from the United States, we will use supervised learning to develop a regression model that can predict a member of the population's wage. We examine a number of factors that relate to wages for a group of males from the Atlantic region of the United States. In particular, we wish to understand the association between an employee’s age and education, as well as the calendar year, on his wage. Dataset and Inputs The dataset to be examined contains survey information from the central Atlantic region of the United States. Solution Statement A solution to this problem will be a regression model such as a linear regression, a decision tree regressor, or a support vector regressor. Benchmark Model Given that we seek a regression model a good naive benchmark would be to use either the mean or the median of the wages for the dataset. Evaluation Metrics Given that this is a regression task, we can measure the success of our model using the R2R2 metric, the Mean Absolute Error, or the Mean Square Error.
So for classification we are not looking for a continuous distribution like income and age We are looking for a binary value. Does it go up? Or does it go down.
Maybe naive bayes is our benchmark We can use the F score to measure precision and recall Accuracy is only a good measure when you have a balanced class. When you have an imbalanced class.
Distplot plots the distribution of 2 plots
sns.distplot(market_df.Direction == ‘Up’)
For unsupervised learning there are no easy benchmarks But you can use the silhouette score
In our gene expressions example
We use PrincipleComponentAnlysis We project into R2
Problem statement should include input, output and what you do
If you want to go fancy in your data set description you can do a simple regression of one var against your data set
UCI machine learning repository http://archive.ics.uci.edu/ml/datasets.html http://archive.ics.uci.edu/ml/datasets/Madelon
20% effort will get you 80% of the work
How to download the data On a mac do option and click
You should not open this in Excel
Vector Notebooks
Everything we do here in R works with Numpy Pandas Recycling, Filtering and Vectroization
Vector Length is immutable. You can not change its length
Matrixes are vectors
Cbind binds collumns together to make a matrix simliar to how R bind works to make a metrix
#The ordering of a matix is collumwise
R does recycling
It recylces the pattern of the vector until it has enough
So id you did c(1,2,3) + c(6,0,9,20,22) Elment 1,2,3 1,2,3, 4, 5
Yields 7,2,12,21,24 explan c(6,0,9,20,22) 1,2,3, 1, 2
Vector multiplication is defined in R differently from Linear Algebra c(1,2,4) * c(2,2,2) It will do an element wise multiplication and will use recylcing if necessary = (2,4,8)
Different from dot product %88%
Y <- c(3,5,7,9,11) y<- c(2,4) Output : 4, 8 y<- c(3:5) Output : 7,9,11
Generating a vector, you can generate a vector using : D
ORDER OF OPERATONS PE COLAN MDAS or PECMDAS
By means increment by 3 seq(12, 30, by 3) seq(1.1,1.2 , length
Rep does repeach
Rep(1:3, 2)
X <-
Any(x>4)
all(x>4)
Rectified linear unit, or relu
Element vectorwise multiplication x<- seq(-5,5, length = 20) // makes 20 stops
x* ( x>0)
Will print 0 for false and will print out pos val
Vector Operations Notebook
c<- [1,2,5] C round(sqrt)
Recall equation for gausian is f(x) = e^((-x)^2) To turn a vector into matrix matrix(vector_name, ncol=2)
R has a Null value and NA
So if you tried to find the mean Where (x <- c,88,NA,132) THEN mean(x) Output : error
mean(x, na.rm=T) # this removes all NA’s from x, Output ( 78) mean(x) Where (x <- c,88,null,132) means(x) = 76
Filtering , we have been using masks to Filter
z_filter[Mask] A mask is a boolean vector z_filter[TRUE,FALSE,TRUE,TRUE]
x[x>3] <- 0 // assigns 0 to all values of x where x>3
which(z*z>8)
Returns all index values when x z*z > 8
There is a vectorized ifelse x<- ifelse(x %%2 == 0, 0,x)
Jan 24TH CLASS sRegression, classification or clustering. Pick which one is best
We will make the directory ipynb
I dont know what the language of the future will be, but it will be called FORTRAN So numpy, R and tensor flow are wrappers for FORTRAN
We use BLAS, Basic Linear Algebra Subroutines Like ax+y
Lapack Is for fancier things like row reductions etc
Notes on the project : summary statistics of the data ( mode mean median if it is discrete then the unique values, if it is continuous then there is none)
Mean is a good benchmark for Most common clas is a good benchamrk for classification
R^2, MSE, RMSE, MAE
It is a good idea to comment on the distrinution of the target and the target class
For classsification
Bench <- (mode)
Accuracy . // if evenly distributed tarfet
F score // look for ways to tweak percision and accuracy // if unevenly distributed
Auroc
Add in headings, # The largest heading
https://help.github.com/articles/basic-writing-and-formatting-syntax/#headings
Usual logon and update ssh ubuntu@34.214.168.175 git pull upstream master Or Git pull
$ tree -L 1 Shows us the level $ tree -L 2 Shows us the 2 levels
// in julia notebooks we uncommented and installed
install.packages("mlbench") So we install this file on our container//
So on AWS WE ALSO have a cache of images called dsnb or docker notebooks
So we have docker on aws and we run a container J on top of that and we run a notebook on J
// docker is not a virtual machine it is using the system resources
#So onto his julia notebook the way he wants us to deal with string functions BreastCancer$Class <- as.integer(BreastCancer$Class == 'malignant')
#If we wanted to get rid of rows with empy data sets BreastCaner <- na.omit(BreastCancer)
#If we wanted to kill a row we kill the ID row BreastCancer$Id <- NULL
You cant use the accuracy of your score to figure this out
Now we working with notebook number 5 are pulling things
You should give lists names, if not you would get numbers assigned but that is silly, you would You can add to lists
Collumns are all the same type they are vectors Rows can be different They are a key value pair ( like dictionaires in python or hash in Java)
Input $ j <- list(name="Joe", salary=55000, union=T) Input $J $name 'Joe' $salary 55000 $union TRUE
Input: j$name Output: 'Joe'
Input: unlist(j)
Output: <”joe”, “5500”, “T”>
Note A matrix can be thought of as a vector of vectors. A dataframe can be thought of as a list of lists or a vector of lists. Let's briefly consider why we would need such a data structure.
Note a list or a matrix is a vector of vectors A tensor is a matrix of matrixes
A data fram can be a list of rows or a list of collumns
Input: colnames(balloons) <- c('color', 'size', 'act', 'age', 'inflated') Input : colnames #He assign a vector of names to the collumns because there were no collumn names Output: “color, size…”
Input: n = dim(balloons)[1] rownames(balloons) <- paste("balloon_", 1:n, sep="")
Output:
Input: balloond$color
Output: Yellow Yellow Yellow .. yellow purple Input:balloons[,color]
Output:
If we want a data frame with just a few collums and not all balloons[c(‘size’,’act’)]
The one thing you need from matrixes or df Df[ 2,4] Gives us row 2, collumn 2
DF[5, ] Gives us all collum values for row 5
Df[ 5:6, ] Will give us all collumns for row 5 and row 6
DF[ , 6] Gives us all rows for collumn 6 Df[ row, collumn] Df[ vector_1, vector_Z]
balloons[grepl('_1.', rownames(balloons)),]
#So the period lets it be any problem #So like 11, 12, 13.. 19
#balloons[grepl('_1.\d', rownames(balloons)),] Lets you do any digit afterwards
balloons[balloons$color == 'PURPLE',] Returns all the rows where the color is purple