Repositories list

• pymetagen

  Public
  Metadata Generator

  pythonclimetadata+ 7csvsql-queryparquetmetadata-extractionparquet-toolspyarrowpolars

  Python

  •

  MIT License

  •0•0•1•0•Updated Jun 11, 2025Jun 11, 2025

• kafka-streams-example

  Public
  Java

  •0•0•0•1•Updated Jun 7, 2025Jun 7, 2025

• tomlparams

  Public
  TomlParams: TOML-based parameter files made better

  Python

  •

  MIT License

  •0•9•4•0•Updated Jun 3, 2025Jun 3, 2025

• ntu-ktp-data-quality

  Public
  This repository is part of the Knowledge Transfer Partnership (KTP) between Nottingham Trent University (NTU) and Bigspark. The aim of this project is to address data quality issues in large datasets specifically in Finance using advanced techniques for error detection, error correction, duplicate detection, and beyond.

  Python

  •

  MIT License

  •0•0•0•0•Updated Feb 21, 2025Feb 21, 2025

• dbt-airflow-dapr-docker

  Public
  Python

  •

  Apache License 2.0

  •1•0•0•0•Updated Dec 16, 2024Dec 16, 2024

• docformatter

  Public
  Formats docstrings to follow PEP 257

  Python

  •

  MIT License

  •82•0•0•0•Updated Dec 3, 2024Dec 3, 2024

• data-engineering-blueprints

  Public
  Patterns and concepts for building resilient data pipelines in Python and Scala

  0•5•0•0•Updated Aug 27, 2024Aug 27, 2024

• genai-presidio

  Public
  Repository for PII Anonymizer code package and sample FastAPI API to use it to talk to LLM

  Jupyter Notebook

  •0•0•0•0•Updated Jun 21, 2024Jun 21, 2024

• nuxtjs-template

  Public template
  JavaScript

  •0•0•0•0•Updated Jan 21, 2024Jan 21, 2024

• aws-sso-sync

  Public
  sso-sync tool to help with the SCIM setup for bigspark.

  Go

  •

  Apache License 2.0

  •0•0•0•0•Updated Oct 26, 2023Oct 26, 2023

• test_glue_

  Public
  To test glue job

  Python

  •2•0•0•0•Updated Aug 1, 2023Aug 1, 2023

• test_glue

  Public
  To test glue job

  Python

  •2•0•0•0•Updated Jul 14, 2023Jul 14, 2023

• ai-hackathon

  Public
  General Purpose repo for NW AI Hackathon 2023

  Jupyter Notebook

  •

  Apache License 2.0

  •2•0•0•1•Updated Apr 20, 2023Apr 20, 2023

• streamsets_json_schema_validator_processor

  Public
  A streamsets dc sample processor for validation records with a specified JSON schema

  Java

  •

  Apache License 2.0

  •0•1•0•1•Updated Apr 14, 2023Apr 14, 2023

• bigspark-challenge-frontend

  Public
  Shell

  •0•0•0•0•Updated Dec 1, 2022Dec 1, 2022

• vcs_demoo

  Public
  0•0•0•0•Updated Sep 12, 2022Sep 12, 2022

• incubator-livy

  Public
  Mirror of Apache livy (Incubating)

  Scala

  •

  Apache License 2.0

  •610•0•0•0•Updated Jul 22, 2022Jul 22, 2022

• bigspark-challenge-aws

  Public
  1•0•0•0•Updated Jun 30, 2022Jun 30, 2022

• barcode-server

  Public archive
  Java

  •0•0•0•0•Updated Apr 20, 2022Apr 20, 2022

• mwaa-local

  Public
  Shell

  •

  MIT No Attribution

  •0•0•0•0•Updated Mar 25, 2022Mar 25, 2022

• snowpark-scala-data-profiling

  Public
  Data profiling example using Snowflake sample datasets and Scala

  Jupyter Notebook

  •

  Apache License 2.0

  •0•0•0•0•Updated Feb 15, 2022Feb 15, 2022

• sparkMeasure

  Public
  This is the development repository of SparkMeasure, a tool for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark task and stage metrics data.

  Scala

  •

  Apache License 2.0

  •151•0•0•0•Updated Feb 1, 2022Feb 1, 2022

• bigspark-challenge-spark

  Public
  Shell

  •4•0•0•0•Updated Feb 1, 2022Feb 1, 2022

• jvm-profiler

  Public
  JVM Profiler Sending Metrics to Kafka, Console Output or Custom Reporter

  Java

  •

  Other

  •342•0•0•0•Updated Dec 2, 2021Dec 2, 2021

• emr-uber-profiler-notebooks

  Public
  emr-uber-profiler-notebooks

  0•0•0•0•Updated Nov 19, 2021Nov 19, 2021

• bc-git-integration-push

  Public
  Java

  •21•0•0•0•Updated Sep 9, 2021Sep 9, 2021

• deequ

  Public
  Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

  Scala

  •

  Apache License 2.0

  •555•0•0•0•Updated Jul 8, 2021Jul 8, 2021

• tutorials

  Public
  StreamSets Tutorials

  Java

  •

  Apache License 2.0

  •192•0•0•0•Updated Apr 19, 2021Apr 19, 2021

• ibm-mq-streamsets

  Public
  Java

  •0•0•0•0•Updated Mar 16, 2021Mar 16, 2021

• roadmap.sh

  Public
  JavaScript

  •

  Other

  •1•0•0•0•Updated Jan 10, 2021Jan 10, 2021