baseline_1_num_topics_200_normalize_vectors

    main(chosen_model_no=1,
         use_spacy=False,
         use_soft_cosine_similarity=False,
         num_topics=200,
         no_below=5,
         no_above=0.5,
         normalize_vectors=True)

lsi_bow Loading tokens 2019-03-24 23:29:35,313 : INFO : adding document #0 to Dictionary(0 unique tokens: []) 2019-03-24 23:29:38,670 : INFO : adding document #10000 to Dictionary(119578 unique tokens: ['-PRON-', 'action', 'affect', 'ally', 'an']...) 2019-03-24 23:29:41,514 : INFO : adding document #20000 to Dictionary(192872 unique tokens: ['-PRON-', 'action', 'affect', 'ally', 'an']...) 2019-03-24 23:29:44,143 : INFO : adding document #30000 to Dictionary(250954 unique tokens: ['-PRON-', 'action', 'affect', 'ally', 'an']...) 2019-03-24 23:29:44,403 : INFO : built Dictionary(255118 unique tokens: ['-PRON-', 'action', 'affect', 'ally', 'an']...) from 30885 documents (total 7885592 corpus positions) 255118 2019-03-24 23:29:44,965 : INFO : discarding 231029 tokens: [('-PRON-', 30238), ('an', 16990), ('game', 23546), ('in', 27568), ('of', 29090), ('the', 30318), ('this', 15532), ('to', 29821), ('with', 25261), ('a', 29432)]... 2019-03-24 23:29:44,966 : INFO : keeping 24089 tokens which were in no less than 5 and no more than 15442 (=50.0%) documents 2019-03-24 23:29:45,096 : INFO : resulting dictionary: Dictionary(24089 unique tokens: ['action', 'affect', 'ally', 'base', 'brand']...) 24089 2019-03-24 23:29:51,936 : INFO : collecting document frequencies 2019-03-24 23:29:51,936 : INFO : PROGRESS: processing document #0 2019-03-24 23:29:52,309 : INFO : PROGRESS: processing document #10000 2019-03-24 23:29:52,689 : INFO : PROGRESS: processing document #20000 2019-03-24 23:29:52,972 : INFO : PROGRESS: processing document #30000 2019-03-24 23:29:52,994 : INFO : calculating IDF weights for 30885 documents and 24088 features (3234325 matrix non-zeros) Corpus as Bag-of-Words Latent Semantic Indexing (LSI/LSA) 2019-03-24 23:29:53,105 : INFO : using serial LSI version on this node 2019-03-24 23:29:53,105 : INFO : updating model with new documents 2019-03-24 23:29:53,106 : INFO : preparing a new chunk of documents 2019-03-24 23:29:54,235 : INFO : using 100 extra samples and 2 power iterations 2019-03-24 23:29:54,235 : INFO : 1st phase: constructing (24089, 300) action matrix 2019-03-24 23:29:55,622 : INFO : orthonormalizing (24089, 300) action matrix 2019-03-24 23:30:05,091 : INFO : 2nd phase: running dense svd on (300, 20000) matrix 2019-03-24 23:30:09,764 : INFO : computing the final decomposition 2019-03-24 23:30:09,774 : INFO : keeping 200 factors (discarding 10.423% of energy spectrum) 2019-03-24 23:30:10,517 : INFO : processed documents up to #20000 2019-03-24 23:30:10,673 : INFO : topic #0(973.254): 0.264*"or" + 0.214*"player" + 0.181*"new" + 0.179*"world" + 0.166*"all" + 0.165*"one" + 0.164*"play" + 0.142*"more" + 0.140*"use" + 0.138*"up" 2019-03-24 23:30:10,674 : INFO : topic #1(324.338): -0.507*"player" + -0.208*"mode" + 0.188*"world" + 0.165*"puzzle" + 0.162*"find" + 0.157*"but" + 0.157*"do" + -0.141*"new" + -0.141*"battle" + -0.140*"play" 2019-03-24 23:30:10,675 : INFO : topic #2(297.568): 0.416*"new" + 0.410*"world" + -0.389*"or" + -0.139*"do" + 0.138*"battle" + -0.132*"one" + -0.129*"if" + 0.129*"adventure" + 0.121*"story" + -0.111*"not" 2019-03-24 23:30:10,676 : INFO : topic #3(268.256): 0.401*"level" + -0.337*"or" + 0.272*"enemy" + -0.272*"new" + -0.234*"world" + 0.185*"puzzle" + 0.149*"each" + 0.147*"mode" + -0.140*"create" + 0.123*"up" 2019-03-24 23:30:10,677 : INFO : topic #4(261.542): -0.316*"enemy" + 0.312*"puzzle" + 0.300*"new" + 0.243*"level" + -0.220*"battle" + 0.208*"play" + -0.202*"weapon" + -0.179*"ship" + 0.172*"player" + -0.156*"or" 2019-03-24 23:30:10,678 : INFO : preparing a new chunk of documents 2019-03-24 23:30:11,168 : INFO : using 100 extra samples and 2 power iterations 2019-03-24 23:30:11,169 : INFO : 1st phase: constructing (24089, 300) action matrix 2019-03-24 23:30:12,271 : INFO : orthonormalizing (24089, 300) action matrix 2019-03-24 23:30:21,052 : INFO : 2nd phase: running dense svd on (300, 10885) matrix 2019-03-24 23:30:22,744 : INFO : computing the final decomposition 2019-03-24 23:30:22,744 : INFO : keeping 200 factors (discarding 11.025% of energy spectrum) 2019-03-24 23:30:23,091 : INFO : merging projections: (24089, 200) + (24089, 200) 2019-03-24 23:30:24,879 : INFO : keeping 200 factors (discarding 4.176% of energy spectrum) 2019-03-24 23:30:25,399 : INFO : processed documents up to #30885 2019-03-24 23:30:25,400 : INFO : topic #0(1144.542): 0.260*"or" + 0.208*"player" + 0.178*"world" + 0.169*"all" + 0.169*"one" + 0.166*"new" + 0.163*"play" + 0.141*"more" + 0.138*"use" + 0.136*"up" 2019-03-24 23:30:25,401 : INFO : topic #1(387.464): 0.509*"player" + 0.202*"mode" + -0.178*"do" + -0.170*"but" + -0.165*"puzzle" + -0.164*"find" + -0.149*"world" + 0.144*"battle" + 0.138*"play" + 0.132*"or" 2019-03-24 23:30:25,402 : INFO : topic #2(349.636): 0.480*"world" + -0.340*"or" + 0.322*"new" + -0.166*"do" + 0.140*"battle" + -0.134*"if" + 0.134*"adventure" + -0.129*"not" + 0.127*"story" + -0.125*"play" 2019-03-24 23:30:25,403 : INFO : topic #3(324.005): 0.587*"level" + -0.377*"or" + 0.318*"puzzle" + -0.232*"world" + 0.146*"enemy" + 0.143*"mode" + 0.134*"each" + 0.105*"challenge" + 0.102*"through" + -0.101*"build" 2019-03-24 23:30:25,404 : INFO : topic #4(310.213): 0.435*"enemy" + -0.308*"puzzle" + -0.290*"player" + 0.250*"weapon" + -0.203*"world" + -0.197*"play" + -0.187*"new" + 0.177*"battle" + 0.162*"ship" + 0.148*"fight" 2019-03-24 23:30:25,404 : INFO : creating matrix with 30885 documents and 24089 features

Query appID: 570 (Dota 2)