Skip to content
/ Risk-Analysis-using-Topic-Models-on-Annual-Reports Public

Risk Analysis using Topic Models on Annual Reports. First EDIC semester project (PhD).

5 stars 2 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

Diego999/Risk-Analysis-using-Topic-Models-on-Annual-Reports

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

315 Commits
.gitignore
.gitignore
 
 
0_config.py
0_config.py
 
 
10k_filled.sql
10k_filled.sql
 
 
10k_filled_old.sql
10k_filled_old.sql
 
 
1_download_data.py
1_download_data.py
 
 
2a_process_data.py
2a_process_data.py
 
 
2b_process_data_from_db.py
2b_process_data_from_db.py
 
 
2c_analyze_postprocessing.py
2c_analyze_postprocessing.py
 
 
3_extract_sections.py
3_extract_sections.py
 
 
4a_analyze_topics_static.py
4a_analyze_topics_static.py
 
 
4b_check_results.py
4b_check_results.py
 
 
4c_analyze_topics_through_time.py
4c_analyze_topics_through_time.py
 
 
4d_infer_topics.py
4d_infer_topics.py
 
 
4e_visualize.py
4e_visualize.py
 
 
5_download_output_variables.py
5_download_output_variables.py
 
 
5_sentiment_analysis_per_topic.py
5_sentiment_analysis_per_topic.py
 
 
5b_create_dataset.py
5b_create_dataset.py
 
 
5c_filter_dataset.py
5c_filter_dataset.py
 
 
5c_filter_dataset_visualization.py
5c_filter_dataset_visualization.py
 
 
5d_visualize_sector_industry.py
5d_visualize_sector_industry.py
 
 
5e_visualize_sector_industry_unique_companies.py
5e_visualize_sector_industry_unique_companies.py
 
 
5f_similarity_per_year.py
5f_similarity_per_year.py
 
 
5g_topic_dist_vs_sector_or_year_sim_for_reports_having_sector.py.py
5g_topic_dist_vs_sector_or_year_sim_for_reports_having_sector.py.py
 
 
5h_topic_dist_or_year_sim_for_all_reports.py
5h_topic_dist_or_year_sim_for_all_reports.py
 
 
5i_topic_word_cloud.py
5i_topic_word_cloud.py
 
 
5j_topic_dist_or_year_sim_per_company.py
5j_topic_dist_or_year_sim_per_company.py
 
 
5k_topic_popularity_over_year_for_all_reports.py
5k_topic_popularity_over_year_for_all_reports.py
 
 
5l_topic_popularity_over_cycle_years_for_all_reports.py
5l_topic_popularity_over_cycle_years_for_all_reports.py
 
 
5m_report_similarity_by_company.py
5m_report_similarity_by_company.py
 
 
6a_tokenize_for_word_embeddings_all_10k.py
6a_tokenize_for_word_embeddings_all_10k.py
 
 
6a_tokenize_for_word_embeddings_section_level.py
6a_tokenize_for_word_embeddings_section_level.py
 
 
7_word_embeddings_visualization.py
7_word_embeddings_visualization.py
 
 
8_volatility_prediction_baseline.py
8_volatility_prediction_baseline.py
 
 
README.md
README.md
 
 
companies.sql
companies.sql
 
 
generate_statistics.py
generate_statistics.py
 
 
help_pos.txt
help_pos.txt
 
 
paper.pdf
paper.pdf
 
 
parser_utils.py
parser_utils.py
 
 
requirements.txt
requirements.txt
 
 
sector_industry.sql
sector_industry.sql
 
 
stopwords_big.txt
stopwords_big.txt
 
 
stopwords_small.txt
stopwords_small.txt
 
 
utils.py
utils.py
 
 

Repository files navigation

Crawling-Edgar-Data-Annual-Report-10-k-From-U.S.-Securities-and-Exchange-Commission

Download annual report 10-k of companies through U.S. Securities and Exchange Commission from first quarter 1993 to last quarter of the current year. Analyze Risk Factors section using topic modeling. The code is delivered as it is. This project is not maintained. You can find the paper here.

Requirements

It relies on Python 3.5. Due to a bug somewhere in Python & Gensim with multiprocessing, don't forget to run export OMP_NUM_THREADS=1 && python3 myfile.py for the step 4.

Description of the dataset

The script will create the followings folders:

  • data
  • data/gz: which contains the compressed version of company indices from EDGAR, grouped by quarter and year.
  • data/pd: which contains the Pandas DataFrame version of company indices, grouped by quarter and year. Also includes a Pandas DataFrame merging all dataframes.
  • data/ar: the annual reports
    • data/ar/CIK/[names/*.txt]: names contain all the company names associated with the CIK.
    • txt files are the annual reports.

At the end, the overall size is around 750 GB.

Issues/Pull Requests/Feedbacks

Don't hesitate to contact for any feedback or create issues/pull requests.

About

Risk Analysis using Topic Models on Annual Reports. First EDIC semester project (PhD).

Resources

Readme
Activity

Stars

5 stars

Watchers

3 watching

Forks

2 forks
Report repository

Releases

No releases published

Packages

 
 
 

Languages