Skip to content
General LOFAR data processing on the SurfSARA clusters
Branch: restructure
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Documentation Status Build Status License: GPL v3 PyPI version version status alt text codecov Coverage alt text BCH compliance Updates

Due to the large computational requirements for LOFAR datasets, processing bulk data on the grid is required. This manual will detail the Dutch grid infrastructure, the submission process and the types of users anticipated to use the LOFAR reduction tools.


SurfSARA is the Dutch locations of the CERN Computational Grid and its facilities are available for general scientific computing. Because the LOFAR telescope requires significant computational resources, the reduction pipelines have been fitted to run on the Dutch Grid nodes with minimal user interaction. The GRID_LRT software package automates LOFAR data staging, job description, Pre-Factor parallelization, job submission and management of intermediate data.


  • User account to the lofar ui at
  • Login to the PiCaS client at
  • Active Grid certificate for launching jobs/accessing storage
  • Membership of the LOFAR VO.
  • Astron LTA credentials for staging LOFAR data


The up to date installation instructions are here.



If you actively use GRID_LRT, please cite this software as such below:

      author       = {Alexandar P. Mechev} 
      title        = {apmechev/GRID_LRT: v0.5.0},
      month        = sep,
      year         = 2018,
      doi          = {10.5281/zenodo.1438833},
      url          = {}

If you're using GRID processed data, also consider citing the paper below, outlining the procedure of running LOFAR data through a High Throughput Cluster:

   author = { {Mechev}, A. and {Oonk}, J.~B.~R. and {Danezi}, A. and {Shimwell}, T.~W. and                             
{Schrijvers}, C. and {Intema}, H. and {Plaat}, A. and {Rottgering}, H.~J.~A.},
    title = "{An {A}utomated {S}calable {F}ramework for {D}istributing {R}adio {A}stronomy {P}rocessing {A}cross {C}lusters and {C}louds}",
booktitle = {Proceedings of the International Symposium on Grids and Clouds (ISGC) 2017, held 5-10 March, 2017 at Academia Sinica, Taipei, Taiwan (ISGC2017). Online at \url{}, id.2},
     year = 2017,
archivePrefix = "arXiv",
   eprint = {1712.00312},
 primaryClass = "astro-ph.IM",
    month = mar,
      eid = {2},
      doi = {10.22323/1.293.0002},
    pages = {2},
   adsurl = {\url{}},
  adsnote = {Provided by the SAO/NASA Astrophysics Data System}

Tutorial Notebook

Best way to get acquainted with the software is with the tutorial notebook available at GRID_LRT/tutorials/LRT_demo.ipynb

Setting up Jupyter on loui

$> ssh
[10:42 me@loui ~] > mkdir ~/.jupyter
[10:42 me@loui ~] > export PATH=/cvmfs/$PATH
[10:42 me@loui ~] > export LD_LIBRARY_PATH=/cvmfs/$LD_LIBRARY_PATH
[10:42 me@loui ~] > jupyter notebook password

Running a Jupyter notebook on loui

Assuming you have ssh login to loui, you can run this notebook on your own machine by using ssh port forwarding :

$> ssh -L 8888:localhost:8888
[10:42 me@loui ~] > source /home/apmechev/.init_jupyter

With that shell running, you can open the browser on your local machine and go to localhost:8888, and browse to the tutorials folder.

Grid job submission and queuing

Data Staging

In order to stage the data using the ASTRON LTA api, you need credentials to the ASTRON LTA service. These credentials need to be saved in a file on the lofar ui at ~/.stagingrc in the form

You can’t perform that action at this time.