Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

dlsa Distributed Least Squares Approximation

implemented with Apache Spark

Introduction

In this work, we develop a distributed least squares approximation (DLSA) method that is able to solve a large family of regression problems (e.g., linear regression, logistic regression, and Cox's model) on a distributed system. By approximating the local objective function using a local quadratic form, we are able to obtain a combined estimator by taking a weighted average of local estimators. The resulting estimator is proved to be statistically as efficient as the global estimator. Moreover, it requires only one round of communication. We further conduct shrinkage estimation based on the DLSA estimation using an adaptive Lasso approach. The solution can be easily obtained by using the LARS algorithm on the master node. It is theoretically shown that the resulting estimator possesses the oracle property and is selection consistent by using a newly designed distributed Bayesian information criterion (DBIC). The finite sample performance and the computational efficiency are further illustrated by an extensive numerical study and an airline dataset.

System Requirements

  • Spark >= 2.3.1

  • Python >= 3.7.0 Note that Spark < 3.0 is only compatible with Python <3.8

  • R >= 3.5 (optional)

    • lars

    See setup.py for detailed requirements.

Make a Python module

  • You firstly need to pack the core code into Python module
make zip

A dlsa.zip file will then be created within the folder projects/.

  • Then you should be able to upload it to the Spark cluster inside your Python script.
spark.sparkContext.addPyFile("dlsa.zip")

Run the PySpark code on the Spark platform

projects/bash/spark_dlsa_run.sh

or simply run

projects/logistic_dlsa.py

References

About

Distributed least squares approximation (dlsa) implemented with Apache Spark

Topics

Resources

License

Releases

No releases published

Packages

No packages published