Data Everything

Data Ingestion:

• LinkedIn’s Gobblin https://gobblin.apache.org/
• Uber’s Marmaray https://eng.uber.com/marmaray-hadoop-ingestion-open-source/

Metadata

• https://engineering.linkedin.com/blog/2020/datahub-popular-metadata-architectures-explained
• LinkedIn’s Datahub https://engineering.linkedin.com/blog/2019/data-hub
• Uber’s Databook https://eng.uber.com/metadata-insights-databook/
• Netflix’s Metacat https://netflixtechblog.com/metacat-making-big-data-discoverable-and-meaningful-at-netflix-56fb36a53520

Data Privacy

• Uber Architecture https://www.infoq.com/articles/data-driven-privacy-architecture/
• Facebook https://engineering.fb.com/2020/07/21/security/data-classification-system/

Data Misc

• https://medium.com/airbnb-engineering/on-spark-hive-and-small-files-an-in-depth-look-at-spark-partitioning-strategies-a9a364f908
• https://medium.com/bigid-on-id-privacy/two-heads-are-better-than-one-when-data-discovery-meets-data-catalogs-bigid-8fc114aa4084

Subject Inventory(Source of truth):

• https://medium.com/bigid-on-id-privacy/building-a-single-source-of-data-truth-across-privacy-security-and-governance-bigid-a7dce8dc3360

Privacy Acts (GDPR/CCPA):

• Entity Matching/Resolution:

  • https://users.umiacs.umd.edu/~getoor/Tutorials/ER_VLDB2012.pdf
  • http://pages.cs.wisc.edu/~anhai/papers/magellan-vldb16.pdf
  • https://www.microsoft.com/en-us/research/uploads/prod/2019/04/Auto-EM.pdf

• Right to Delete:

  • https://medium.com/bigid-on-id-privacy/how-to-avoid-the-nightmare-of-data-deletion-rights-c796b991afc8
  • https://blog.twitter.com/engineering/en_us/topics/infrastructure/2020/deleting-data-distributed-throughout-your-microservices-architecture.html

Unstructured Data

Adding context to data:

• POS Tagging:

  • https://medium.com/analytics-vidhya/pos-tagging-using-conditional-random-fields-92077e5eaa31
  • https://medium.com/free-code-camp/an-introduction-to-part-of-speech-tagging-and-the-hidden-markov-model-953d45338f24

• BERT:

  • https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270

• Key word Extraction:

  • https://medium.com/analytics-vidhya/automated-keyword-extraction-from-articles-using-nlp-bfd864f41b34