[SPARK-21209][MLLLIB] Implement Incremental PCA algorithm

[sandecho]

What changes were proposed in this pull request?

A new feature called Incremental Principal Component Analysis Algorithm(IPCA) has been proposed. It divides the incoming data in batch size and compute the PCA of the individual batch to generate Principal Component of entire data.

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
Unit Testing
IPCA.zip

[sandecho]

ok to test

[sandecho]

ok to test

[sethah]

• Only committers can trigger the tests.
• MLlib is in maintenance only mode, so we wouldn't accept this patch as is.
• If this were to go into ML, I think you'd need to discuss it more thoroughly on the JIRA. Another good alternative is to make this a Spark package.
• It's best if you write unit tests and adhere to the style guides when you submit new patches.

I would recommend closing this PR until more discussion has taken place about whether or not this is a good fit for Spark ML. Thanks!

[sandecho]

@sethah Thank you. I accept your recommendation. I will take it to ML. Secondly I have written unit tests and I have also adhere to style guidelines. But my concern is that no one is having a discussion on the JIRA. Even the creator of the JIRA @wbstclair is not reachable.

[wbstclair]

Sorry for being out of touch! I have been busy with work. I will look at your code soon. I also have an implementation which adheres to the style which is a decent first draft, but needs to be tested. I think we can build a decent offering out of the two.

…

On Mar 1, 2018 9:54 PM, "Sandeep Kumar Choudhary" ***@***.***> wrote: @sethah <https://github.com/sethah> Thank you. I accept your recommendation. I will take it to ML. Secondly I have written unit tests and I have also adhere to style guidelines. But my concern is that no one is having a discussion on the JIRA. Even the creator of the JIRA @wbstclair <https://github.com/wbstclair> is not reachable. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#20708 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AT0Q5y_MZgrSW-hpifmCdBU-o54tP1MDks5taN6QgaJpZM4SY4uO> .

[sandecho]

Thanks @wbstclair . That's a good suggestion. Although I would have to take it to ML from MLLIB, rest will be the same.

[sandecho]

ok to test