Skip to content

[自用轮子] 一个多线程/多进程工具模块(生产者-消费者模式实现)

License

Notifications You must be signed in to change notification settings

Tishacy/QSpider

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

35 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

QSpider

License: MIT Pyversion Version

An easy to use tools module for writing multi-thread and multi-process programs.

Install

QSpider could be easily installed using pip:

$ pip install qspider

Example

import requests
from qspider import concurrent

# Define a source list for task function to parse.
def get_source():
    """Return a url list."""
    return ['http://www.baidu.com' for i in range(500)]

# Define the task function and add a thread_func decorator
# The thread_func decorator needs a source list, and other options (num_workers, has_result ...) as arguments
@concurrent.thread_func(source=get_source(), num_workers=100, has_result=True)
def my_task(task_source):
    """A customized task function.
    Process the task_source and return the processed results.

    Arguments
    :param task_source: the elem in the source list, which is a url here.
    :rtype: (int) A http status code.
    """
    url = task_source
    res = requests.get(url, timeout=5)
    return res.status_code

# Execute the task function.
results = my_task()
print(results)

Results of the example is as below:

[Info] 500 tasks in total.
[ ✔ ] 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 500/500 [eta-0:00:00, 0.9s, 542.9it/s]
[200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, ..., 200, 200, 200, 200]

Releases

  • v0.1.1: First release with basic classes.
  • v0.1.2: Reconstruct code, add ThreadManager, ProcessManager and other tool classes.
  • v0.1.3: Fix multiprocess locking bug on Windows.
  • v0.1.4:
    • Add silent argument in manager._run method.
    • Enhance the display style of the progress message.
  • v0.1.5:
    • Make task be either a class, a function or a class method.
    • Add concurrent decorators for convenient use.
    • Add concurrent decorator examples.
  • v0.1.6:
    • Update templates.
    • Replace multiprocessing queue.
    • Support with statement.
    • Optimize displays in jupyter notebook and windows powershell.

License

Copyright (c) 2020 tishacy.

Licensed under the MIT License.

About

[自用轮子] 一个多线程/多进程工具模块(生产者-消费者模式实现)

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages