[C4GT] Ingestion speedup and refactoring

[techsavvyash]

Project Details

cQube is envisioned as a ready-to-use / pre-packaged, configurable, and extendable DPG solution to enable observability and action towards effective policy implementation in education and other sectors, involving various stakeholders across govt, society, and private sectors.

Features to be implemented

cQube POCs are implemented in this repository and require improvements in the data ingestion process to achieve speedup and optimised usage of hardware resources, along with this, the code needs refactoring and restructuring and a comprehensive testing suite.

The tasks are tracked via the following issues:

• OpenTelemetry #88
• Test Cases #79
• Better .temp file management #77
• Event Upsert #75
• Dimension Upsert #74
• Persist configs for CSV Ingestion #73
• Allow for update in config. #72
• Define errors and add them to output folder. #71
• Dataset Ingestion/Update through a config. #70
• Expose Ingestion Progress #69
• Manage OOM Errors when ingesting large files > 1 GB #68
• Make docker-compose.yml parametric with env variables. #67
• Override global config by program config if available. #66
• Implement - quoteChar #65
• Implement - csvDelimeter #64
• Implement - caseSenstiveFKSearch #63
• Implement - onlyCreateWhitelisted #62

Learning Path

Complexity

Medium

Skills Required

Typescrcipt, NestJS, Hasura.

Name of Mentors:

@techsavvyash

Project size

8 Weeks

Product Set Up

See the setup here

Acceptance Criteria

• Unit Test Cases using Jest
• e2e Test Cases using Jest

C4GT

This issue is nominated for Code for GovTech (C4GT) 2023 edition.
C4GT is India's first annual coding program to create a community that can build and contribute to global Digital Public Goods. If you want to use Open Source GovTech to create impact, then this is the opportunity for you! More about C4GT here: https://codeforgovtech.in/

[Sukriti-m]

Hey @techsavvyash

I am highly interested in this project. As a junior undergraduate in Computer Science and Engineering, specializing in Backend Web Development, I am excited about contributing to this initiative. With a keen interest in driving effective policy implementation in the education sector, I am drawn to cQube's vision as a configurable and extendable DPG solution.

I believe in the power of collaboration and its impact on the community. I have a proven track record of contributing to open source programs such as GirlScript Summer of Code, Social Summer of Code, and many more. In fact, I was one of the top contributors in GSSoC'22. These experiences have equipped me with valuable skills in collaborative development, project management, and technical documentation.

I would appreciate it if you could provide guidance on how I can get started with the project. Should I directly submit a proposal or is there any specific process to follow? I am eager to dive into the work and make a meaningful contribution.

I look forward to the possibility of joining your esteemed team and making a meaningful impact in the project. I will be grateful for the opportunity to collaborate with mentors and contribute to this project.

Best Regards,
Sukriti Maurya

[heyhimansh]

Hey @techsavvyash , I would like to contribute to this project . Looking forward to write a proposal for that ..

[techsavvyash]

Hey @Sukriti-m @heyhimansh
Really glad to see your enthusiasm for the project. Looking forward to seeing you guys contribute. To get started if you feel you understand the project enough, you can go ahead and draft the proposal, but I would suggest that you guys go ahead try to clone the repository, try setting up the codebase and hack around understanding the code to get a better understanding, this is not required but would be good to do. You can reach out to us in case you face any issues either here or via our discord community here

[lata-11]

Hey @techsavvyash. I am looking forward to submit a proposal for this project. I would love to work on this project.

[techsavvyash]

Hey @lata-11, we are excited to have you here, please post whatever questions you might have here or on the C4GT discord.

[Mounika-Pavanthi]

I am excited to contribute to the cQube project and be a part of the team driving effective policy implementation in the education sector. With my expertise in TypeScript, NestJS, and Hasura, I am well-equipped to improve the data ingestion process, refactor and restructure code, and develop a comprehensive testing suite. I am committed to delivering high-quality code and ensuring the project's success. Please consider my involvement in the cQube project, as I am eager to contribute to the development of Digital Public Goods and make a positive impact in the education sector.

[techsavvyash]

Hey @Mounika-Pavanthi,
Excited to see your enthusiasm for the project, Kindly go through some GFIs to understand the project better and write up a proposal, the proposal submission period ends on 11th June. In case of any queries feel free to create a thread on the C4GT discord server.
Thanks