Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build_qa_database.py插数的时候有个小问题 #8

Open
ShuGao0810 opened this issue Aug 6, 2019 · 0 comments
Open

build_qa_database.py插数的时候有个小问题 #8

ShuGao0810 opened this issue Aug 6, 2019 · 0 comments

Comments

@ShuGao0810
Copy link

ShuGao0810 commented Aug 6, 2019

数据量少于BULK_COUNT未执行插数,修改了一下:

def init_ES():
    pie = ProcessIntoES()
    # 创建ES的index
    pie.create_mapping()
    start_time = time.time()
    index = 0
    count = 0
    action_list = []
    BULK_COUNT = 1000  # 每BULK_COUNT个句子一起插入到ES中

    for line in open(pie.music_file, 'r', encoding='utf8'):
        if not line:
            continue
        item = json.loads(line)
        index += 1
        action = {
            "_index": pie._index,
            "_type": pie.doc_type,
            "_source": {
                "question": item['question'],
                "answers": '\n'.join(item['answers']),
            }
        }
        action_list.append(action)
        if index >= BULK_COUNT:
            pie.insert_data_bulk(action_list=action_list)
            index = 0
            count += 1
            print(count)
            action_list = []
            
    if index < BULK_COUNT:
        pie.insert_data_bulk(action_list=action_list)
    
    end_time = time.time()
    print("Time Cost:{0}".format(end_time - start_time))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant