Kornia @ Google Summer of Code 2023

Welcome to the Google Summer of Code 2023 @ Kornia

NOTE

We have been rejected as mentoring org (again)

We still encourage people to reach us and work on this projects by contacting us in slack but we do not have budget to pay anyone.

The Kornia team will work on this projects during the summer, feel free to join and help us.

Table of Contents

• What is Google Summer of Code 🌞 👩‍💻
• How to apply ? 🧑‍🎓
• Project Ideas List 💡 💡
  • Improve documentation and tutorials website
  • Automatic Augmentations API
  • Add distributed execution to Limbus
  • Make kornia fully-runnable on Apple Silicon chips
  • Object Detection API
  • Support to torchdynamo and Triton
  • Expand kornia io with Image, Video, Cameras api
• Timeline 📆
• How do I pass the GSoC evaluations ? 🚀

What is Google Summer of Code

more info: https://developers.google.com/open-source/gsoc/resources/downloads/GSoC2021Presentation.pdf

How to apply ?

1. Are you a >=18; oss beginner or student in an eligible country ? [ READ ] 🧑‍🎓 🗺️
2. GO through the Project Ideas List list below 📄
3. Pre-Apply to this [ FORM ] AND join Kornia Slack [ JOIN ]
4. IF you are contacted by a mentor THEN write the project proposal ✍️
5. ELSE improve your skills and try next year 🔁
6. SUBMIT your project proposal through GSoC website !! (VERY IMPORTANT) ⚠️ ⚠️
7. The project admins will balance the applications ⚖️
8. IF you passed this process THEN Congratulations!! You're in !! 🎉 🎆
9. GO TO How do I pass the GSoC evaluations ? ➡️ 🚀

DISCLAIMERS:

• We won't consider any application from a student that hasn't been contacted by a mentor.
• Projects without a detailed schedule won't be considered. 📆
• The GSoC is a full-time internship; do not expect being contacted or if you are already working.
• A Project failure is not an option; we won't take that risk.
• The final application is on the GSoC site; otherwise your are out.
• Do not open useless pull requests to increase your git history; we know how to detect fake profiles.
• Pre-selected students might expect a screen interview.
• Google pays to the student; not kornia.org.
• If you are not notified by Google; you are not in.
• Read the GSoC student GUIDELINES & FAQ

Project Ideas List

1. IDEA: Improve documentation, tutorials website

   • Description: Review the documentation and the tutorials website. Re-organise existing tutorials, create new examples, improve testing on the tutorials, better link between functions and examples, benchmarks, etc.
   • Expected Outcomes:
     • A new reformatted website for tutorials and examples.
     • Proper benchmarking suite.
   • Resources:
     • https://github.com/kornia/kornia-examples
     • https://github.com/kornia/tutorials
     • https://github.com/kornia/kornia-benchmark
   • Skills Required: PyTorch and basic knowledge about computer vision and Kornia
   • Possible Mentors: Edgar Riba, Farm-ng.
   • Difficulty: Easy
   • Duration: 175 hours

2. IDEA: Add distributed execution to Limbus

   • Description: Limbus is a new open source framework to create Machine Learning pipelines within the context of Deep Learning and writen in terms of differentiable tensors message passing on top of Kornia and PyTorch. Currently Limbus take advantage of asyncio to asyncronously execute the different components of the pipeline, in this project we want to be able to execute those components or maybe sets of components in different machines in a transparent way.
   • Expected Outcomes:
     • A new API allowing distributed execution.
   • Resources:
     • https://github.com/kornia/limbus
     • https://github.com/kornia/limbus-components
     • https://grpc.io/
   • Skills Required: Python and basic knowledge of grpc or similar libraries.
   • Possible Mentors: Luis Ferraz
   • Difficulty: Medium
   • Duration: 175 hours

3. IDEA: Automatic Augmentations API

   • Description: Starting from AutoAugment, automatic augmentation techniques have been proven to be effective across different application domains and tasks. However, most computer vision frameworks only support using the searched augmentation policies for ImageNet despite the differences across different datasets due to its high computational costs. Recent searching algorithms improved the search efficiency with differentiable augmentation techniques. Based on the nature of the differentiable data augmentation provided by Kornia, we aim at integrating multiple augmentation search frameworks to support searching for any given customised datasets. In general, this is a complete R&D project that performs different algorithm implementation and also benchmarking against the reported stats as in the paper.
   • Expected Outcomes:
     • Multiple augmentation search algorithms implementation with elegance: Faster AutoAugment, AugNet, Augerino.
     • Benchmark the implementation against the reported stats.
   • Resources:
     • moskomule/dda: Differentiable Data Augmentation Library (github.com)
     • cedricrommel/augnet: Code of "Deep invariant networks with differentiable augmentation layers" (github.com)
     • g-benton/learning-invariances: Codebase for Learning Invariances in Neural Networks (github.com) *[1911.06987] Faster AutoAugment: Learning Augmentation Strategies using Backpropagation (arxiv.org)
     • [2202.02142] Deep invariant networks with differentiable augmentation layers (arxiv.org)
   • Skills Required: PyTorch and decent knowledge about computer vision and AutoAugment Family
   • Possible Mentors: Jian Shi, Miquel A. Farre
   • Difficulty: Medium
   • Duration: 175 hours

4. IDEA: Make kornia fully-runnable on Apple Silicon chips

   • Description: Apple Silicon devices become more and more used in by computer vision developers. However, there Pytorch does not fully support them https://github.com/pytorch/pytorch/issues?q=is%3Aopen+is%3Aissue+label%3A%22module%3A+mps%22 For the most operations, though, it is possible to write a custom implementation of the kornia functions, which would work together with current PyTorch version
   • Expected Outcomes:
     • All kornia tests would be passing on M1 device
   • Resources:
     • https://github.com/kornia/kornia/issues/1717
   • Skills Required:
     • Basic PyTorch and Python programming skill
     • Access to Apple Silicon device (MacBook M1/M2)
   • Possible Mentors: Dmytro Mishkin
   • Difficulty: Medium
   • Duration: 175 hours

5. IDEA: Object Detection API

   • Description: Choose and implement from scratch an Object Detection API taking as reference a reference state of the art method suitable for production environments. Run some evaluations and provide training code with robust api, tests, and docs.
   • Expected Outcomes:
     • A high level api to perform object detection using the kornia ecosystem packages e.g augmentations, losses or metrics.
     • Tutorials and documentation about how to use the api for inference and training.
     • Improve other needed components of the library like losses or metrics.
   • Resources:
     • Example to use as a reference for api: https://kornia.readthedocs.io/en/latest/contrib.html#face-detection
   • Skills Required:
     • Knowledge of state of the art for object detection in industry
     • Access to GPU to train and run benchmarks
     • Experience in user API design
   • Possible Mentors: João Gustavo A. Amorim
   • Difficulty: Medium
   • Duration: 175 hours

6. IDEA: Support to torchdynamo and Triton

   • Description: Replace in kornia the support of torchscript by torchdynamo to make the library faster in production environments. Learn more: https://github.com/kornia/kornia/issues/2200
   • Expected Outcomes:
     • Full support of torchdynamo without changing the api of existing functions
     • Propose some algorithms to be implemented in Triton, e.g bilateral filer, connected components, etc
     • Run benchmarks, write blog post and tutorials
   • Resources:
     • https://github.com/kornia/kornia-benchmark/tree/master/dynamo
     • https://github.com/pytorch/torchdynamo
     • https://openai.com/blog/triton/
   • Skills Required:
     • Knowledge of some PyTorch internals
     • Knowledge of cuda programming with Triton
     • Access to GPU to test and benchmark
   • Possible Mentors: João Gustavo A. Amorim , Edgar
   • Difficulty: Medium
   • Duration: 175 hours

7. IDEA: Expand kornia io with Image, Video, Cameras api

   • Description: We want to give more support image and video reading, and support to cameras via kornia rust project. Rust is a modern and safe language to get performant code and very user-friendly for Pythonists.
   • Expected Outcomes:
     • Expanded python apis for image/video encoding and decoding.
     • New api for Cameras, e.g Webcams or Oak-D (Stereo)
     • Implement with Rust and exposed in Python support the DlPack protocol.
   • Resources:
     • https://github.com/kornia/kornia-rs
     • https://github.com/kornia/dlpack-rs
     • https://github.com/kornia/kornia/tree/master/kornia/io
   • Skills Required:
     • Knowledge of Rust and the PyO3 bindings library
     • Knowledge about DlPack
     • Experience with camera drivers
   • Possible Mentors: Edgar
   • Difficulty: Medium/Difficult
   • Duration: 175 hours

How do I pass the GSoC evaluations ?

The program duration is ~2months; we will be flexible but there are some major RULES.

‼️ To PASS the mid-term and final evaluations you MUST have a MERGED pull-request ➡️ you'll get your cut 😎

• Google pays you IF ONLY IF you pass the evaluations
• The mentors will evaluate you based on the performance during the project.

Timeline

The Full Program Timeline: https://summerofcode.withgoogle.com/programs/2023

• Org Applications Open - [ January 23, 2023 ] 🎬 ✔️
• Org Application Deadline - [ February 07 2023 ] 📅
• Org Notification Date - [ February 21, 2023 ] 🙏** WE ARE OUT :(**