DARTSectorWebCrawler

Crawling Demo

UI Viewer Demo

Language

Python (Developed and test with version 3.8.5)
JavaScript

Package Requirements

PyQt5
PyQtWebEngine<=5.15.3

Manual

Crawler

from qtweb_crawling import DartCrawlerWindow
from PyQt5.QtWidgets import QApplication

app = QApplication(sys.argv)
wnd = DartCrawlerWindow()
wnd.resize(600, 800)
wnd.show()
app.exec_()

Press 'Start' button to start crawling from DART web-page.
Press 'Get Result' button to serialize crawl result to local file in path {./result_list.pkl}.

Tree

You can make tree hierarchy using Tree class.

from tree import Tree
import pickle

with open('./result_list.pkl', 'rb') as fp:
    node_list = pickle.load(fp)
tree = Tree()
tree.add_leaf_nodes(node_list)

UI (Tree & Table Viewer)

import sys
import pickle
from tree import Tree
from treeview import CorpTreeViewWidget
from PyQt5.QtWidgets import QApplication

with open('./result_list.pkl', 'rb') as fp:
    node_list = pickle.load(fp)

app = QApplication(sys.argv)
wgt_ = CorpTreeViewWidget()
wgt_.resize(600, 600)
wgt_.show()
corp_tree = Tree()
corp_tree.add_leaf_nodes(node_list)
wgt_.setCorpTree(corp_tree)
app.exec_()

Develop History

웹크롤링 - DART 기업개황 업종별 기업 리스트 가져오기 (1)
웹크롤링 - DART 기업개황 업종별 기업 리스트 가져오기 (2)
웹크롤링 - DART 기업개황 업종별 기업 리스트 가져오기 (3)
웹크롤링 - DART 기업개황 업종별 기업 리스트 가져오기 (Final)