Skip to content

Exploratory Data Analysis

Jump to bottom
AmruthaVee edited this page Sep 9, 2021 · 1 revision

2. Exploratory Data Analysis

Exploratory Data Analysis is performed to gain some insights on the data. As we do not have any features that define our data we won't be able to extract many insights. For the plots we have used seaborn and altair libraries.

First, we load the dataset:

df_path = "/home/lnc/0. Latest News Classifier/01. Dataset Creation/" df_path2 = df_path + 'News_dataset.csv' df = pd.read_csv(df_path2, sep=';')

And now we plot a graph to display the number of paragraphs in each of our article.

`bars = alt.Chart(df).mark_bar(size=50).encode( x=alt.X("Category"), y=alt.Y("count():Q", axis=alt.Axis(title='Number of articles')), tooltip=[alt.Tooltip('count()', title='Number of articles'), 'Category'], color='Category'

)

text = bars.mark_text( align='center', baseline='bottom', ).encode( text='count()' )

(bars + text).interactive().properties( height=300, width=700, title = "Number of articles in each category", )`

The above code displays a bar graph with the number of paragraphs in each category. In a similar way we can plot a graph to display the % of paragraphs in each category. Also we can display the paragraph length by category.

df['News_length'] = df['Content'].str.len() plt.figure(figsize=(12.8,6)) sns.distplot(df['News_length']).set_title('News length distribution');

News Length Distribution

Now, Let's remove from the 95% percentile onwards to better appreciate the histogram:

quantile_95 = df['News_length'].quantile(0.95) df_95 = df[df['News_length'] < quantile_95]

plt.figure(figsize=(12.8,6)) sns.distplot(df_95['News_length']).set_title('News length distribution');

95% Distribution

We can get the number of news paragraphs with more than 700 characters: df_more10k = df[df['News_length'] > 700] len(df_more10k)

Now, let's plot a boxplot: plt.figure(figsize=(12.8,6)) sns.boxplot(data=df, x='Category', y='News_length', width=.5);

Box Plot

After this point, we cannot do Data Exploration. That's when Feature Engineering comes into picture. So, we'll save the Dataset in Pickle file.

with open('News_dataset.pickle', 'wb') as output: pickle.dump(df, output)

Clone this wiki locally