Skip to content


Folders and files

Last commit message
Last commit date

Latest commit



5 Commits

Repository files navigation

Getting Start with Weibo Data Collection

This tutorial is to help you setup the environment to collect data from Weibo using a Java program. The program follows OAuth2.0 authorization mechanism to get access to the Weibo data. It fetches data through Weibo API and stores the data to the MySQL database. A simple analysis tool for data processing is also developed in the program.

What to you need to get started?

  1. A Weibo developer account. Please go to to create (update) your developer account and create an application (站内应用). After creating your application, you need to mark down your 'appkey' and 'app_secret'.

  2. A Java IDE (e.g., Eclipse. In the following, I will use Eclipse IDE as an example)

  3. MySQL installed.

How to setup?

  1. Import the project "WeiboCrawler" in the attached package to your Java IDE (Eclipse, here).

  2. Connect Java with MySQL

    a) Using MySQL commands create a database named 'sinaweibo' from the terminal.

    b) Open Eclipse. Go to the project property page. Under 'Java Build Path' -> 'Libraries' tap, click 'Add External JARs'. Add the jar 'mysql-connector-java-5.1.18-bin.jar' (available in the attached package as well).

    c) From the 'Package Explore' in the Eclipse, locate the file 'examples/(default package)/'. Find the function 'InitialMysql()'. Set up the proper configuration values for 'database_name', 'url', 'user' and 'password'.

    d) Test the connection to MySQL. Run the function InitialMysql() only [e.g., you disable all the other functions in main()]. If success, you will see the message "Succeeded connection to the Database."

Ref: Please find in for a simple tutorial of connection MySQL from Java.

  1. Get the Weibo access token

    We follow OAuth2.0 policy provided by Sina Weibo ( The simple tutorial will guide you to get your access token.

    Note that, you do not need to download the Java SDK as mentioned in the tutorial, since our program was built from that SDK and we kept all the necessary files. The two files mentioned in the tutorial are located under 'src/' and 'examples/weibo4j.examples/'.

  2. Start the data collection process

    a) Locate the file 'examples/(default package)/'.

    b) Configure the main() function Specify the value 'access_token' and 'Entrance_user_id' in the main() function. The 'Entrance_user_id' is the UID, a unique identifier for each Weibo user. You can randomly specify one. Please read the [Weibo API] documents for more details. [Weibo API]:

    ** A suggestion is that to choose the user with a small number of followings and followers (around hundreds) to start with.

    ** Note that the data collection process uses a recursive manner. It starts from the root user, and add the root user to the table (tbIndex = 1). It then adds all the followers and followings (*** the followings may not be included due to performance issue in this program. Please also have a check) at the end of the table with increasing tbIndex’s. The process keeps going by exploring users (users’ profiles and their relationships) from tbIndex = 2 user and adds new users to the end of the table.

    You should now see the data collection process is running. Good luck.

Author: Chen Junting Last modified: 2012-06-20

Copyright (c) 2012 Chen Junting Distributed under GNU GPL v3.


No description, website, or topics provided.






No releases published


No packages published