In this work, we consider learning sparse models in large scale setting, where the number of samples and the feature dimension can grow as large as millions or billions. Two immediate issues occur under such challenging scenarios: (i) com- putational cost; (ii) memory overhead.