Frequently Asked Questions

rmandelb edited this page Nov 17, 2014 · 15 revisions

Table of Contents

Overview

Science

Data

Data Analysis

Teams

Submissions

Scoring

Using GitHub

Meetings and Conferences

Other

Overview

What is the Great3 Challenge?

The Great3 Challenge is a shape measurement challenge for weak gravitational lensing. In order to learn about the universe with weak lensing, we have to understand how well we can actually measure the galaxy shapes that go into our measurements. This blinded challenge helps us see how the various methods compare to each other and how different effects impact our results. We hope it will be useful both for people with existing shape measurement codes they'd like to test, and for people from non-astronomy fields who think this is an interesting computational problem and want to get involved. There is more detailed information in the Great3 Handbook.

What is the timeline of the challenge?

We released the first simulations (an alpha release) to the public in August 2013. After a 2-month validation period, we released a slightly improved version as a beta release on October 18, 2013, with 12 branches being made available at this time. This release marks the start of the 6-month challenge period, with the challenge ending April 30, 2013. The challenge moved from beta release to regular challenge phase in late November, and the rest of the data was released in mid-December.

Who can enter?

Anyone can enter! You can enter by yourself or in a team.

What if I don't know anything about astronomy/cosmology?

Please enter! We're happy to answer questions, and you can check out the challenge handbook for more information on the science.

What is this GalSim thing you keep talking about?

GalSim: The modular galaxy image simulation toolkit is an open-source software project, using object-oriented Python wrappers around C++ numerical routines. It's also a public GitHub project. We used it to generate all the images in the simulations for the challenge. You can download it yourself to make your own simulations or to use some of its routines as part of your data processing. While GalSim was initially developed with Great3 in mind, it is capable of making more complex simulations than those in Great3.

What do I need to be able to enter?

This is a data-processing challenge, so we assume you're familiar with handling large amounts of data (multiple TB). If you want to use some of the analysis software we're distributing, you should also be familiar with installing and running programs from the command line in a Unix-like environment.

We plan on distributing a simple end-to-end pipeline to analyze the data, but it won't be good enough to win the challenge. See here. We suggest writing or using your own software for at least some of the analysis steps.

There are also pre-submission scripts on this repository (see here), which take the galaxy shapes you've measured and turn them into summary files to upload on the submission website. These scripts, and the example script described above, are written in Python. You don't need to understand Python to use them, though you do need a Python installation. Of course, you can also write your own scripts.

You do NOT need to know anything about weak lensing or shape measurement before you begin. We have an overview in the handbook, and you can also follow the steps of the simple pipeline once it is released.

How do I keep up-to-date on the challenge? Do you have a mailing list?

We do have a mailing list: great3-participants. You can also watch this repository for updates if you have a GitHub account (click on the "Watch" button at the top of the page, and change your selection to "Watching").

Science

Where do the galaxy images come from?

We used Hubble Space Telescope data from the COSMOS project to make our RealGalaxy images, and parametric models such as Sersics to make the simpler images. Both kinds were processed through GalSim to add shears, PSFs, etc. There's more information on the RealGalaxies in the SHERA paper, and more information on everything else in the Great3 handbook and the GalSim paper.

Did you consider complication X?

For detailed science questions, we refer you to the Great3 Handbook.

Data

Where is the data?

The simulation data is downloadable from http://great3.jb.man.ac.uk/leaderboard/data and a US mirror linked from that site. Warning: it's about 3TB, downloadable in chunks corresponding to the branches of the challenge.

What if my Internet connection isn't fast enough?

We have a limited number of hard drives containing all the data for participants who can't download everything. You can apply to us to send one to you; you'll have to make a commitment to participate in the challenge, though.

Data Analysis

What if I don't know how to (correct for a PSF/combine multiple images/something else)?

We have released some example scripts to do parts of these processes as described here, and plan to release more around the time of the mid-challenge meeting. You can use these directly for the steps you can't complete yourself, or use them as a template for a new method of your own. Or you can find collaborators to work with you. Also, you can opt to only participate in branches for which you have the software - there is no requirement to participate in all branches.

I know how to do X, but not Y. Can you match me up with somebody who can do Y, but not X?

If you need collaborators, we suggest browsing the open issues to find people who are searching, or opening a new issue on this repository so other challenge members who may also be looking for collaborators can find you.

Can I edit the scripts you're distributing through GitHub?

Of course! You can't push your changes to this repository, though. You're welcome to fork us or to edit locally. Please post on the issue page if you have some improvement that you think would benefit others who are participating in the challenge that you'd like to share more widely.

Teams

What makes a team?

Any person or group of people doing analysis together. The same team can submit results from multiple methods, if they wish. Different branches can be submitted by different subsets of the team, if not everybody wants to work on all the branches, or the groups can make separate teams if they wish.

Can I be on more than one team? With the same collaborators? With different collaborators? By myself?

Yes; yes (see the previous question); yes; yes.

What's the difference between a team and a method?

A team is a group of people working together; a method is a specific variety of analysis performed by a team. You can submit more than one method to each branch, but only your highest-scoring method will appear on the leaderboards. Regardless of how many methods you have, you may only make 1 submission per branch per day.

There is more information on what constitutes a method on the Great3 leaderboard page.

Why are there asterisks next to some teams?

Teams with significant contributions from members of the Great3 Executive Committee are not eligible to win, so they get asterisks next to their names. They won't be counted for prizes (so, if a team with an asterisk gets the highest score in a branch, and a team without an asterisk gets the second-highest score, the team with the second-highest score would still get the first-place points).

Submissions

Where do I submit my results?

On the [Great3 leaderboard website](http://great3.jb.man.ac.uk/leaderboard/.

What format do my results need to be in?

There is a complete and detailed description here.

Scoring

How are the submissions scored?

We have metrics to calculate how well the different teams are doing. They're described in the Great3 Handbook, and calculated automatically based on submissions.

How is the overall leaderboard calculated?

We give teams points for their placements on individual leaderboards, up to their best 5 placements, and the team with the most points wins. More detail can be found in the Great3 Handbook.

Using GitHub

Do I need an account to use GitHub?

You don't need an account to download the software or read the issues and wiki, but you will need an account to post or comment on issues or to watch the repository for updates. It's free, and you can sign up here.

How do I ask a question? How do I report a problem?

Open an issue! Go to the main repository page, then click on the tab on the right-hand side that says "issues", then click the button that says "new issue". Or, alternately, just click here for the issues page or here to open a new issue.

How do I download the software to get started?

You can download the current version of the software directly by clicking on the "download" button in the bottom right corner of the main repository page. Alternately, you can type git clone https://github.com/barnabytprowe/great3-public.git in a command line in the directory where you'd like to install the software; that will create a new directory called great3-public containing the software. Then it's easy to update if we change things: just type git pull in the great3-public directory to get the new software.

We also have a GalSim file that describes some features of GitHub we've found to be useful. It's called git.txt, in the devel/ directory of the GalSim repository, or you can find it here on the web.

Why are you doing this on GitHub, anyway?

We've found that GitHub is an easy way to organize lots of people working on a project, with good support for side-projects and branches. It also lets us keep the software and the discussion about the software in close contact.

Meetings and Conferences

Do you have meetings or conferences? If so, how do I get involved?

We do. We had a Great3 Kickoff Meeting in Pasadena, CA from August 20-22 (Tues-Thurs). See this website for information and for PDFs of talks. A mid-challenge meeting took place in early January - see announcement here. There was also a meeting after the close of the challenge at Carnegie Mellon University in Pittsburgh, PA, USA.

Other

Who's organizing this?

Rachel Mandelbaum and Barnaby Rowe are our fearless co-leaders. There's also an executive committee of 10 other people who've helped with the decision-making about the exact challenge parameters, plus a mailing list of around 50 more people who contributed thoughts and suggestions.

16 people so far have contributed to the main GalSim repository.

What's this executive committee thing?

The executive committee contains the people who know the challenge parameters. It's not a blind challenge for them, so they're barred from winning the challenge, and they get asterisks next to their team names if they contribute in a significant way to the method used by that team. The members are, alphabetically: Jim Bosch, Chihway Chang, Frederic Courbin, Mandeep Gill, Mike Jarvis, Rachel Mandelbaum, Hironao Miyatake, Reiko Nakajima, Barnaby Rowe, Jason Rhodes, Melanie Simet, and Joe Zuntz.

How do I get in touch with the organizers?

You can:

Where can I find more information?

We suggest the Great3 website, the Great3 Handbook, the GalSim reference paper, the leader board website, and the issues page of this repository.

Where can I ask more questions?

Right here on GitHub!