LDA_FFT and LDADE_FFT

[amritbhanu]

• LDA_FFT(k=10) are getting similar performance as LDADE FFT and tf-idf_SVM.
• runtime is slower in this order tf-idf svm, LDA_FFT, LDADE_FFT
• We have more explainability from one domain with just k=10 topics which you cant get from SVM_tfidf.

[timm]

@amritbhanu this is visually clear but please run thru stats.py to give us the full story. @Suvodeep90 is working "the" definitive version of that code. please work with him for that

also, is LDADE being allowed to tune k? imagine if FFT at k=10 kills LDADE at any k. that would be... cool.

also, u got runtimes or #evals for these? is LDADE crazy slower than FFT?

also, you doing that thing where you build N FFTs in training, pick the best (using training data) then carry that over to test? if yes, what N?

u caching the generated rules? that would be good to see.

when you do classification with the LDA data, what u using?

what is the target variable? severity=S? what S?

[amritbhanu]

For recall , FFT wins only 1/6 cases. For precision, FFT wins 3/6 times. Accuracy 4/6 times, based on scott-knott. In other cases FFT is doing as well as LDADE.

Yes LDADE is crazily slow with about on an average 50 times, i got the runtimes.

Yes i am building N=32 trees, D=5 and pick the best one from training data.

No I didnt cache the rules this time. Let me get that! Will take 2-3 days.

when you do classification with the LDA data, what u using?

Using FFT for classification, or do you mean something else?

Target variable is the max severity seen is labelled as positive and all others as negative class.

[amritbhanu]

have asked @Suvodeep90 for the updated stats.py code, once he provides me with that i will get it done.