-
Notifications
You must be signed in to change notification settings - Fork 81
/
ctlearn.xml
46 lines (46 loc) · 19.7 KB
/
ctlearn.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="../assets/xml/rss.xsl" media="all"?><rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Universe OpenAstronomy (Posts about CTLearn)</title><link>http://openastronomy.org/Universe_OA/</link><description></description><atom:link href="http://openastronomy.org/Universe_OA/categories/ctlearn.xml" rel="self" type="application/rss+xml"></atom:link><language>en</language><lastBuildDate>Thu, 30 May 2024 01:00:38 GMT</lastBuildDate><generator>Nikola (getnikola.com)</generator><docs>http://blogs.law.harvard.edu/tech/rss</docs><item><title>Week 10 & 11: The End…</title><link>http://openastronomy.org/Universe_OA/posts/2020/08/20200826_0322_sahilyadav27/</link><dc:creator>Sahil Yadav</dc:creator><description><p>The past 3 months have been really amazing. I’ve learned a lot about working on a collaborative project and working with deadlines.</p>
<p>In the past two weeks, I polished off the work done for the project in the PR, so that it is ready to merge. There were a few merge conflicts that needed to be addressed.</p>
<p>Then, I worked on a generate_runlist file to make runlists for converting MAGIC ROOT data to HDF5 data. In a few days, training will start on the approximately 5000 MAGIC data runs using CTLearn’s models.</p>
<!-- TEASER_END -->
<p>I’ll be continuing to work and contribute to this project in the future as required as well. Hopefully this will be useful for researchers working on related problems in the future.</p>
<p>I’m grateful for my mentors, Google, and OpenAstronomy for this amazing GSoC 2020 experience. Signing off!</p>
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=a4a1414cd574" width="1" height="1"></description><category>CTLearn</category><guid>http://openastronomy.org/Universe_OA/posts/2020/08/20200826_0322_sahilyadav27/</guid><pubDate>Wed, 26 Aug 2020 02:22:12 GMT</pubDate></item><item><title>Week 8 & 9: Buzzer Beater?</title><link>http://openastronomy.org/Universe_OA/posts/2020/08/20200810_1708_sahilyadav27/</link><dc:creator>Sahil Yadav</dc:creator><description><h4>Week 9 &amp; 10: Buzzer Beater?</h4><p>In the past two weeks, I worked on creating the MCHeader Container for MAGIC data. After discussing with people working at MAGIC, we used the RunHeaders TTree variables to create the Container.</p>
<p>For the arrival direction and telescope pointing, we decided to use the MSrcPosCam variable from the ROOT file, which stores the shower direction relative to the telescope, i.e arrival direction-telescope pointing. Thus, we don’t need to think much about the Alt-Az transformations as well. And we don't have to create a separate column for telescope pointing in the HDF5 file. This saves computation time too.</p>
<p>This solves two of the major issues we were having last time. Now, we will soon start training some MAGIC gamma data files from the two telescopes using the CTLearn deep learning models and get some benchmark results on the same.</p>
<!-- TEASER_END -->
<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Ee0UeDUTC37W9_-ZioSEZQ.jpeg"><figcaption>Devin Booker hitting the buzzer-beater in Kawhi and PG’s face. Hoping this is how GSoC ends.</figcaption></figure><p>We were also trying to create an event source for VERITAS data, but there were some issues with reading the data. We’ll see how this error progresses.</p>
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=80e03f9a5e1b" width="1" height="1"></description><category>CTLearn</category><guid>http://openastronomy.org/Universe_OA/posts/2020/08/20200810_1708_sahilyadav27/</guid><pubDate>Mon, 10 Aug 2020 16:08:08 GMT</pubDate></item><item><title>Week 7 & 8: Full Throttle Ahead!</title><link>http://openastronomy.org/Universe_OA/posts/2020/07/20200727_1715_sahilyadav27/</link><dc:creator>Sahil Yadav</dc:creator><description><p>In the last blog post, I was discussing about creating a MAGICEventSource and calling it inside DL1DataWriter to make HDF5 files with the CTA ML data format from the ROOT files.</p>
<p>I have integrated the MAGICEventSource to writer.py and also updated it to ctapipe v0.8.0. This code is housed at <a href="https://github.com/cta-observatory/dl1-data-handler/pull/90">PR #90</a>. There were a few small issues with the metadata and transformations. The current ctapipe MCHeader container has variables needed to be filled by the input file, but the ROOT file has separate RunHeaders, so we were in a dilemma as to which metadata to store.</p>
<p>Finally, we decided to create our own MCHeader container inside the MAGICEventSource and use this to store metadata instead of using the ctapipe container.</p>
<!-- TEASER_END -->
<p>Also, for the simtel files that have been analysed till now, the telescope pointing Alt and Az are constant and can be hardcoded when we train the deep learning models, to subtract them from the arrival direction Alt and Az. But for the ROOT files, this value is not constant so we will have to create another column into the HDF5 file and pass this value to CTLearn when training on ROOT files.</p>
<p>There was also some confusion regarding the transformations applied to these Alt and Az values before dumping them to the HDF5 file. We were finally able to figure out what was going wrong with our approach, and plotted the difference between arrival direction and pointing position.</p>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*FG648RrN8nWpKPCJ04IF8g.png"><figcaption>Pointing position vs Arrival direction</figcaption></figure><p>So now all that is left is create the metadata container, and the MAGIC dataset to start training the CTLearn models before we start working on VERITAS data.</p>
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=87dd0823ba1b" width="1" height="1"></description><category>CTLearn</category><guid>http://openastronomy.org/Universe_OA/posts/2020/07/20200727_1715_sahilyadav27/</guid><pubDate>Mon, 27 Jul 2020 16:15:13 GMT</pubDate></item><item><title>Week 5 & 6: Half-time</title><link>http://openastronomy.org/Universe_OA/posts/2020/07/20200715_0456_sahilyadav27/</link><dc:creator>Sahil Yadav</dc:creator><description><p>The first evaluation is successfully complete!</p>
<p>In the last blog post, I discussed adding a script to the DL1DataHandler to convert ROOT files to HDF5. But, in order to maintain symmetry with the current DL1DataWriter, we decided to instead create a MAGICEventSource from this root2hdf5 format that initializes the DL1 data container with ROOT data and then is sent to the CTAMLDataDumper to dump this data into an HDF5 file. This will be analogous to the SimTelEventSource that is currently being used for CTA data from simtel files.</p>
<p>So, I have been working on creating this MAGICEventSource. The basic code for the class structure is complete, and the containers are initialized correctly. All that remains now is to add some code to the DL1DataWriter and CTAMLDataDumper to accept and recognize MAGIC data and use the MAGICEventSource instead of the SimTelEventSource.</p>
<!-- TEASER_END -->
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EOSlgwziXKXKk76EYbnN2A.png"></figure><p>In the meantime, I also adapted the generate_runlist script to create runlists specifically for MAGIC data, since the MAGICEventSource reads file masks instead of a couple of M1 and M2 files for each observation.</p>
<p>Hence, both these changes will be pushed soon to the main repository and then we can start working on reading VERITAS data similarly as well as training MAGIC data using CTLearn models.</p>
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=e343ac6bd699" width="1" height="1"></description><category>CTLearn</category><guid>http://openastronomy.org/Universe_OA/posts/2020/07/20200715_0456_sahilyadav27/</guid><pubDate>Wed, 15 Jul 2020 03:56:26 GMT</pubDate></item><item><title>Week 3 & 4: First blood</title><link>http://openastronomy.org/Universe_OA/posts/2020/06/20200628_1322_sahilyadav27/</link><dc:creator>Sahil Yadav</dc:creator><description><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*eVG6uotsjTa5V092bqJ_uw.jpeg"><figcaption>“Magic”</figcaption></figure><p>Since the last blog post, where there was a discussion about creating a script to convert the ROOT file into an HDF5 file with the CTA ML data format. So, during week 3 and 4 I was working on making this script. There were a few issues in this conversion.</p>
<p>In the current DL1DataHandler, the event number is created conveniently in accordance with CTA data. But, the MAGIC data uses a different way to store event numbers. There are 2 arrays for each camera, one for the EvtNumber and another for StereoEvtNumber. The StereoEvtNumber array is mapped from the EvtNumber array. So, I used all the stereo events and stored their values in the HDF5 file. Mono study can be also done on these stereo events. Since MAGIC doesn’t currently do mono analysis on events triggering only one telescope, that part is currently omitted.</p>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aDisON-sEtx0T9-w9Lvkeg.png"></figure><p>So, now that this mapping is figured, we also mapped all the variables required in the HDF5 file with them in the ROOT file. Once everything was set up, I tried reading this converted file using the DL1DataReader. The first run yielded amazing results.</p>
<!-- TEASER_END -->
<figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*gwcmu83NnKO8jYQUFYl5yA.png"><figcaption>First run output for MAGIC cam 1 and 2 from the converted HDF5 file</figcaption></figure><p>So now I made <a href="https://github.com/cta-observatory/dl1-data-handler/pull/90">PR #90</a> to add this script to the DL1DataHandler. There are a few additions, like reading a runlist instead of filename, adding metadata, etc.</p>
<p>Hence, the first milestone of the project is complete along with the first month.</p>
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=dc1ba79de370" width="1" height="1"></description><category>CTLearn</category><guid>http://openastronomy.org/Universe_OA/posts/2020/06/20200628_1322_sahilyadav27/</guid><pubDate>Sun, 28 Jun 2020 12:22:03 GMT</pubDate></item><item><title>Week 1 & 2: Tip-Off</title><link>http://openastronomy.org/Universe_OA/posts/2020/06/20200613_0950_sahilyadav27/</link><dc:creator>Sahil Yadav</dc:creator><description><p>The first week started off with having a video call with the mentors and trying to pave the path ahead for the next 3 months. There were 2 options: Either convert ROOT files to the hdf5 format from CTA or establish a new DL1 reader for ROOT. I decided to run some tests and get an idea about the speed and memory requirements for both the methods. We were more inclined to create a new class for the ROOT files so that we don’t have to save a separate hdf5 file for each ROOT file when using CTLearn. The reading times for both the file types were also similar, so we decided to implement a new child class for ROOT files.</p>
<p>In order to move ahead with this plan, I first wrote down the entire DL1DataWriter code, highlighting the hdf5 dependent parts. This way, I was able to get a better understanding of the code and its intricacies. After talking with Tjark some more, we decided to implement 2 child classes for hdf5 and ROOT which inherited the parent Writer class.</p>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*JDweXFtW-o-ZqcIbIfzkoQ.png"></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*BfDNzmeG5NYEQn8xTOGZrg.png"><figcaption>MC simulated images from MAGIC Cam 1 &amp; 2, an event which only triggered one telescope and not the other.</figcaption></figure><p>After another video call, Ari suggested that I convert the ROOT files into an hdf5 file with the CTA ML Data format to understand the differences between the formats of MAGIC and CTA. Although there is already a library ctapipe_io_magic to produce the MAGICEventSource and use it for DL1DataReader and produce an hdf5 file, there were a lot of issues with trying to use it.</p>
<!-- TEASER_END -->
<p>So we decided to implement our own code to convert the file. This way, any ROOT file can be converted to hdf5 and used by CTLearn before the actual project is complete and in place. During week 2, I worked on writing code for the same. It is mostly complete and a few things need to be ironed out which will be done in the next call on 15th.</p>
<p>Once this is done, I’ll start to work on creating separate child classes for ROOT and hdf5 inheriting from DL1DataWriter as discussed above.</p>
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=1eb6744a41ea" width="1" height="1"></description><category>CTLearn</category><guid>http://openastronomy.org/Universe_OA/posts/2020/06/20200613_0950_sahilyadav27/</guid><pubDate>Sat, 13 Jun 2020 08:50:12 GMT</pubDate></item><item><title>Week 0: The first “bug”</title><link>http://openastronomy.org/Universe_OA/posts/2020/05/20200518_1259_sahilyadav27/</link><dc:creator>Sahil Yadav</dc:creator><description><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nlIMK3SPIJkzkJy8W4iukw.png"></figure><p>As a child, I used to try and count the number of stars in the sky, absolutely mesmerised by the twinkling dots. From the long nights on the roof looking above in fascination started the admiration for astronomy.</p>
<p>As a junior in high school, my older cousin brother introduced me to Machine Learning. Seems weird to say that, but that’s how things panned out. He was working towards his PhD at the time in swarm robotics and gave me a primer of artificial intelligence. Ever since then, I started to read up more on the subject. On reaching college, I pursued a few projects to get a better feeling of academia and the research environment.</p>
<p>During my sophomore year at BITS, GSoC was recommended to me by many of my seniors as a nice way to work and capitalize the summers. I glanced through the participating organizations and OpenAstronomy caught my attention. Specifically, the CTLearn sub-organization seemed like a match made in heaven, using Machine Learning algorithms for gamma-ray data.</p>
<!-- TEASER_END -->
<p>I quickly introduced myself to the community and went through the codebase. The community was very warm and welcoming. When I was a bit comfortable with the code, a few tasks were put forward, to assess our skills and understanding.</p>
<p>Navigating through the code and documentation while performing the tasks was a very enthralling experience. I thoroughly enjoyed it and learned a lot in those few weeks. Soon, I created the proposal, made some edits and submitted the application.</p>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*a8-t18GDsgp-7_B1-wQxLQ.png"><figcaption>Damian Lillard hitting the game-winner over Paul George, sending the Thunder home. Game 5, Playoffs 2019. Approximately encapsulates how I felt reading the acceptance letter.</figcaption></figure><p>When the acceptance came through I was on cloud nine. Nothing beats that feeling of accomplishment and the urge to perform. I soon contacted my mentor and we discussed out a rough timeline of the project. I was also assigned a few tasks, in order to better align the project’s direction.</p>
<p>While I slogged of on this assignment, one error kept bugging me. Initially, I was hesitant to ask my mentor for guidance this early on, but I finally gave way and pinged him. He immediately realised that it was in fact a bug in the latest build. So I fixed the bug, which resulted in making the code a bit faster as a consequence. That first merged PR felt nice in the heart, and I know I'm in for an amazing ride ahead.</p>
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=daa0a8b85b7e" width="1" height="1"></description><category>CTLearn</category><guid>http://openastronomy.org/Universe_OA/posts/2020/05/20200518_1259_sahilyadav27/</guid><pubDate>Mon, 18 May 2020 11:59:01 GMT</pubDate></item><item><title>In-depth comparison between the optimization algorithms implemented</title><link>http://openastronomy.org/Universe_OA/posts/2019/08/20190805_1950_jredondopizarro/</link><dc:creator>Juan Redondo</dc:creator><description><p>On this post I am going to show a comparison between the different optimization algorithms implemented till now in CTLearn, all of them applied to the same telescope and with the same configuration. This is useful to find the similarities and differences between optimization methods, as well as chec <a class="reference external" href="https://medium.com/@jrpg1996/in-depth-comparison-between-the-optimization-algorithms-implemented-31f3d8ea80af?source=rss-d64822a82f52------2">...READ MORE...</a></p></description><category>CTLearn</category><guid>http://openastronomy.org/Universe_OA/posts/2019/08/20190805_1950_jredondopizarro/</guid><pubDate>Mon, 05 Aug 2019 18:50:27 GMT</pubDate></item><item><title>Implementation of Gaussian Processes based Bayesian Optimization</title><link>http://openastronomy.org/Universe_OA/posts/2019/07/20190722_0953_jredondopizarro/</link><dc:creator>Juan Redondo</dc:creator><description><p>During this weeks I have been working mainly on the implementation of the Gaussian Processes based Bayesian Optimization using the scikit-optimize package (also known as skopt).Regarding the hyperparameter space, this implementation only supports uniform, quniform, loguniform and choice options, whi <a class="reference external" href="https://medium.com/@jrpg1996/implementation-of-gaussian-processes-based-bayesian-optimization-3b319b38812d?source=rss-d64822a82f52------2">...READ MORE...</a></p></description><category>CTLearn</category><guid>http://openastronomy.org/Universe_OA/posts/2019/07/20190722_0953_jredondopizarro/</guid><pubDate>Mon, 22 Jul 2019 08:53:05 GMT</pubDate></item><item><title>[Weeks 3–4] — Making progresses</title><link>http://openastronomy.org/Universe_OA/posts/2019/06/20190623_1001_jredondopizarro/</link><dc:creator>Juan Redondo</dc:creator><description><p>[Weeks 3–4] — Making progressesIn this blog post I will be brief, as I intend to increase my posting frecuency from next week.Achievements completed during this period:Created a new repository within CTLearn-project to keep all the code, documentation, experiments and plots related to my GSoC projec <a class="reference external" href="https://medium.com/@jrpg1996/weeks-3-4-making-progresses-113f24433210?source=rss-d64822a82f52------2">...READ MORE...</a></p></description><category>CTLearn</category><guid>http://openastronomy.org/Universe_OA/posts/2019/06/20190623_1001_jredondopizarro/</guid><pubDate>Sun, 23 Jun 2019 09:01:01 GMT</pubDate></item></channel></rss>