{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":23808489,"defaultBranch":"develop","name":"lda","ownerLogin":"lda-project","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2014-09-08T21:11:26.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/29574507?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1701730007.0","currentOid":""},"activityList":{"items":[{"before":"0009c6229af4ddf80408ed0ebf4092114f7153f7","after":"016037863acf08cfdaa29534c7b496be79762bd0","ref":"refs/heads/develop","pushedAt":"2024-02-23T22:03:45.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ariddell","name":"Allen Riddell","path":"/ariddell","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/19950?s=80&v=4"},"commit":{"message":"Construct nzw in fortran order to reduce cache misses in hot loops.\n\nConstruct nzw in fortran order to reduce cache misses in hot loops.\nas it is always iterater over in the topic direction.\n\nThis improves the speed of:\n  - _sample_topics\n  - _loglikelihood\n  - ldac2dtm\n\nBenchmark script:\n\n    $ cat bench_lda_fast.py\n    #!/usr/bin/python3\n\n    import os\n    import sys\n    import time\n\n    import lda\n    import lda.utils\n\n    ldac_fn = sys.argv[1]\n    t0 = time.time()\n    dtm = lda.utils.ldac2dtm(open(ldac_fn), offset=0)\n    print(\"ldac2dtm took: {} seconds\".format(time.time() - t0))\n    t1 = time.time()\n    n_iter = 100\n    n_topics = 100\n    random_seed = 1\n    model = lda.LDA(n_topics=n_topics, n_iter=n_iter, random_state=random_seed)\n    print(\"Model construction took: {} seconds\".format(time.time() - t1))\n    t2 = time.time()\n    doc_topic = model.fit_transform(dtm)\n    print(\"Running model took: {} seconds\".format(time.time() - t2))\n    print(\"Total: {} seconds\".format(time.time() - t0))\n\nOriginal code on reuters.ldac:\n\n    $ python bench_lda_fast.py ../lda/tests/reuters.ldac\n    ldac2dtm took: 0.6914153099060059 seconds\n    Model construction took: 0.0010924339294433594 seconds\n    INFO:lda:n_documents: 395\n    INFO:lda:vocab_size: 4258\n    INFO:lda:n_words: 84010\n    INFO:lda:n_topics: 100\n    INFO:lda:n_iter: 100\n    INFO:lda:<0> log likelihood: -1258788\n    INFO:lda:<10> log likelihood: -734487\n    INFO:lda:<20> log likelihood: -708068\n    INFO:lda:<30> log likelihood: -698147\n    INFO:lda:<40> log likelihood: -691363\n    INFO:lda:<50> log likelihood: -687491\n    INFO:lda:<60> log likelihood: -684165\n    INFO:lda:<70> log likelihood: -682336\n    INFO:lda:<80> log likelihood: -680400\n    INFO:lda:<90> log likelihood: -678905\n    INFO:lda:<99> log likelihood: -676797\n    Running model took: 8.397335290908813 seconds\n    Total: 9.089883804321289 seconds\n\nNew code on reuters.ldac:\n\n    $ python bench_lda_fast.py ../lda/tests/reuters.ldac\n    ldac2dtm took: 0.6984512805938721 seconds\n    Model construction took: 0.001010894775390625 seconds\n    INFO:lda:n_documents: 395\n    INFO:lda:vocab_size: 4258\n    INFO:lda:n_words: 84010\n    INFO:lda:n_topics: 100\n    INFO:lda:n_iter: 100\n    INFO:lda:<0> log likelihood: -1258788\n    INFO:lda:<10> log likelihood: -734487\n    INFO:lda:<20> log likelihood: -708068\n    INFO:lda:<30> log likelihood: -698147\n    INFO:lda:<40> log likelihood: -691363\n    INFO:lda:<50> log likelihood: -687491\n    INFO:lda:<60> log likelihood: -684165\n    INFO:lda:<70> log likelihood: -682336\n    INFO:lda:<80> log likelihood: -680400\n    INFO:lda:<90> log likelihood: -678905\n    INFO:lda:<99> log likelihood: -676797\n    Running model took: 6.758747816085815 seconds\n    Total: 7.458248615264893 seconds\n\nOld code on a bigger dataset:\n\n    $ python bench_lda_fast.py words_5000_docs_10000_sample_freq_0.05.ldac\n    ldac2dtm took: 29.605552911758423 seconds\n    Model construction took: 0.0009605884552001953 seconds\n    INFO:lda:n_documents: 10000\n    INFO:lda:vocab_size: 5000\n    INFO:lda:n_words: 2501477\n    INFO:lda:n_topics: 100\n    INFO:lda:n_iter: 100\n    INFO:lda:<0> log likelihood: -36823892\n    INFO:lda:<10> log likelihood: -27142527\n    INFO:lda:<20> log likelihood: -26367123\n    INFO:lda:<30> log likelihood: -26054203\n    INFO:lda:<40> log likelihood: -25886173\n    INFO:lda:<50> log likelihood: -25782106\n    INFO:lda:<60> log likelihood: -25719117\n    INFO:lda:<70> log likelihood: -25671755\n    INFO:lda:<80> log likelihood: -25639538\n    INFO:lda:<90> log likelihood: -25611296\n    INFO:lda:<99> log likelihood: -25589423\n    Running model took: 260.66207242012024 seconds\n    Total: 290.2686412334442 seconds\n\nNew code on a bigger dataset:\n\n    $ python bench_lda_fast.py words_5000_docs_10000_sample_freq_0.05.ldac\n    ldac2dtm took: 27.78724980354309 seconds\n    Model construction took: 0.0008566379547119141 seconds\n    INFO:lda:n_documents: 10000\n    INFO:lda:vocab_size: 5000\n    INFO:lda:n_words: 2501477\n    INFO:lda:n_topics: 100\n    INFO:lda:n_iter: 100\n    INFO:lda:<0> log likelihood: -36823892\n    INFO:lda:<10> log likelihood: -27142527\n    INFO:lda:<20> log likelihood: -26367123\n    INFO:lda:<30> log likelihood: -26054203\n    INFO:lda:<40> log likelihood: -25886173\n    INFO:lda:<50> log likelihood: -25782106\n    INFO:lda:<60> log likelihood: -25719117\n    INFO:lda:<70> log likelihood: -25671755\n    INFO:lda:<80> log likelihood: -25639538\n    INFO:lda:<90> log likelihood: -25611296\n    INFO:lda:<99> log likelihood: -25589423\n    Running model took: 199.07503652572632 seconds\n    Total: 226.86320090293884 seconds\n\nTo generate sample datasets:\n\n    generate_ldac() {\n        local n_words=\"${1}\";\n        local n_docs=\"${2}\";\n        local sample_freq=\"${3}\";\n\n        awk \\\n            -v n_words=\"${n_words}\" \\\n            -v n_docs=\"${n_docs}\" \\\n            -v sample_freq=\"${sample_freq}\" \\\n            '\n            BEGIN {\n                sample_freq_threshold = 1 - sample_freq;\n\n                for ( doc_idx = 0; doc_idx < n_docs; doc_idx++ ) {\n                    delete words;\n                    current_n_words = 0;\n\n                    for ( word_idx = 0; word_idx < n_words; word_idx++ ) {\n                        if ( rand() > sample_freq_threshold ) {\n                            # Only retain word if it passes the threshold.\n                            words[word_idx] = 1;\n                            current_n_words += 1;\n                        }\n                    }\n\n                    printf \"%d\",  current_n_words;\n\n                    for ( word_idx in words ) {\n                        printf \" %d:1\", word_idx;\n                    }\n\n                printf \"\\n\";\n            }\n        }' > \"words_${n_words}_docs_${n_docs}_sample_freq_${sample_freq}.ldac\"\n    }\n\n    generate_ldac 5000 10000 0.05","shortMessageHtmlLink":"Construct nzw in fortran order to reduce cache misses in hot loops."}},{"before":"aee65d31606a6944818cda32d6f151f347989781","after":"0009c6229af4ddf80408ed0ebf4092114f7153f7","ref":"refs/heads/develop","pushedAt":"2023-12-29T15:13:50.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ariddell","name":"Allen Riddell","path":"/ariddell","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/19950?s=80&v=4"},"commit":{"message":"Update .mailmap","shortMessageHtmlLink":"Update .mailmap"}},{"before":"715e7dfba806e0cf81fbb7ef7ed1efeef3f6cf28","after":"aee65d31606a6944818cda32d6f151f347989781","ref":"refs/heads/develop","pushedAt":"2023-11-28T22:30:30.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ariddell","name":"Allen Riddell","path":"/ariddell","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/19950?s=80&v=4"},"commit":{"message":"Add GH Actions workflow","shortMessageHtmlLink":"Add GH Actions workflow"}},{"before":"8f0d1a0e6b523d018944666ff9b4c6eaf32ff6ff","after":"715e7dfba806e0cf81fbb7ef7ed1efeef3f6cf28","ref":"refs/heads/develop","pushedAt":"2023-11-10T20:53:08.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ariddell","name":"Allen Riddell","path":"/ariddell","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/19950?s=80&v=4"},"commit":{"message":"Add cibuildwheel","shortMessageHtmlLink":"Add cibuildwheel"}},{"before":"b8afeda0bf740a86ce1f11f48f2a6a3735b978be","after":"8f0d1a0e6b523d018944666ff9b4c6eaf32ff6ff","ref":"refs/heads/develop","pushedAt":"2023-10-27T19:02:10.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ariddell","name":"Allen Riddell","path":"/ariddell","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/19950?s=80&v=4"},"commit":{"message":"Add Poetry for packaging and dependency management","shortMessageHtmlLink":"Add Poetry for packaging and dependency management"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEA434WgA","startCursor":null,"endCursor":null}},"title":"Activity · lda-project/lda"}