Documentation on PCA expected max memory usage

[ivirshup]

While running the distributed PCA defined here, I've found that I need to tune my cluster so that I can accommodate the finalize task without nodes crashing out due to memory usage.

It would be nice to be able to estimate what amount of memory is required for a particular data size ahead of time.

[TomAugspurger]

Yeah, this would be nice to document. You might want to check what’s in the dask-array documentation, or the paper the implementation is based on: https://arxiv.org/abs/1301.1071

…

On Mar 26, 2024, at 4:01 PM, Isaac Virshup ***@***.***> wrote: While running the distributed PCA defined here, I've found that I need to tune my cluster so that I can accommodate the finalize task without nodes crashing out due to memory usage. It would be nice to be able to estimate what amount of memory is required for a particular data size ahead of time. — Reply to this email directly, view it on GitHub <#985> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOISFKPLAKGBZ5EYIAZ3Y2HOZVBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJLJONZXKZNENZQW2ZNLORUHEZLBMRPXI6LQMWBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTLDTOVRGUZLDORPXI6LQMWSUS43TOVS2M5DPOBUWG44SQKSHI6LQMWVHEZLQN5ZWS5DPOJ42K5TBNR2WLKBZGQ2DKNJXGQ2YFJDUPFYGLJLJONZXKZNFOZQWY5LFVIZDEMBZGMZDAOBSG2TXI4TJM5TWK4VGMNZGKYLUMU>. You are receiving this email because you are subscribed to this thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.