Skip to content

Dataset Creation

Jump to bottom
AmruthaVee edited this page Sep 9, 2021 · 5 revisions

1. Dataset Creation

The data for the dataset creation is taken from three different articles

  1. Autonomous Cars
  2. Bill Resource
  3. Companies Working on Autonomous Cars respectively.

The data is extracted using Python code. First, we obtain a list of paragraphs from the articles by using the respective URL. # url definition url = "https://www.cbinsights.com/research/autonomous-driverless-vehicles-corporations-list/"

# Request r1 = requests.get(url) r1.status_code

# We'll save in coverpage the Article content coverpage = r1.content

# Soup creation soup1 = BeautifulSoup(coverpage, 'html5lib')

# News identification coverpage_news = soup1.find_all('h2', class_='linkro-darkred') len(coverpage_news)

Now let's extract the text from the articles:

`# Empty lists for content, links and titles news_contents = [] list_links = [] list_titles = []

for n in np.arange(0, number_of_articles):

# Getting the link of the article
link = url + coverpage_news[n].find('a')['href']
list_links.append(link)

# Getting the title
title = coverpage_news[n].find('a').get_text()
list_titles.append(title)

# Reading the content (it is divided in paragraphs)
article = requests.get(link)
article_content = article.content
soup_article = BeautifulSoup(article_content, 'html5lib')
body = soup_article.find_all('p', class_='mol-para-with-font')

# Unifying the paragraphs
list_paragraphs = []
for p in np.arange(0, len(body)):
    paragraph = body[p].get_text()
    list_paragraphs.append(paragraph)
    final_article = " ".join(list_paragraphs)
    
# Removing special characters
final_article = re.sub("\\xa0", "", final_article)
    
news_contents.append(final_article)`

Let's put them into:

a dataset which will the input of the models (df_features) a dataset with the title and the link (df_show_info)

# df_features df_features = pd.DataFrame( {'Article Content': news_contents })

# df_show_info df_show_info = pd.DataFrame( {'Article Title': list_titles, 'Article Link': list_links})

df_features

df_show_info

After the dataset is created we convert the data into .csv file which concludes our web scraping and Dataset creation

#Converting the dataset in .csv file df = pd.DataFrame( {'Article Content':list_titles, }) df['Category'] = 'Companies Working on Autonomous Cars' df.to_csv('content1.csv', index=False, encoding='utf-8')

Clone this wiki locally