-
Notifications
You must be signed in to change notification settings - Fork 13
/
real-world-learnings.html
531 lines (499 loc) · 25.3 KB
/
real-world-learnings.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
<!DOCTYPE html>
<html lang="en">
<head>
<title>Data Mesh Architecture: Real World Learnings</title>
<meta charset="utf-8">
<meta name="description" content="Real world learnings implementing data mesh" />
<meta name="keywords" content="data mesh, data mesh architecture, domain-driven data analytics, data analytics, domain-driven design, domain ownership, data as a product, data product, federated governance, self-serve data platform, data platform">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:site" content="@innoq" />
<meta name="twitter:title" content="Data Mesh Architecture: Real World Learnings" />
<meta name="twitter:description" content="Real world learnings implementing data mesh" />
<meta name="twitter:image" content="https://www.datamesh-architecture.com/images/real-world-learnings.png" />
<meta name="twitter:image:alt" content="Real world learnings implementing data mesh" />
<meta property="og:url" content="https://datamesh-architecture.com" />
<meta property="og:title" content="Data Mesh Architecture: Real World Learnings" />
<meta property="og:description" content="Real world learnings implementing data mesh" />
<meta property="og:image" content="https://www.datamesh-architecture.com/images/real-world-learnings.png" />
<link rel="preload" as="font" type="font/woff2" href="https://www.innoq.com/assets/MarkPro-Book.woff2?cachebuster=2" crossorigin="">
<link rel="preload" as="font" type="font/woff2" href="https://www.innoq.com/assets/MarkPro-Bold.woff2?cachebuster=2" crossorigin="">
<link rel="preload" as="font" type="font/woff2" href="https://www.innoq.com/assets/MarkPro-Heavy.woff2?cachebuster=2" crossorigin="">
<link rel="stylesheet" href="../css/style.css" />
<link rel="stylesheet" href="../css/0.9.3_css_bulma.css" />
<link rel="stylesheet" href="../css/font-awesome_6.0.0_css_all.css"/>
<link rel="shortcut icon" type="image/x-icon" href="/favicon.ico">
</head>
<body>
<nav class="navbar is-dark" role="navigation" aria-label="dropdown navigation">
<div class="container">
<div class="navbar-brand">
<span class="navbar-burger" data-target="navbarMenuHeroA">
<span></span>
<span></span>
<span></span>
</span>
</div>
<div id="navbarMenuHeroA" class="navbar-menu">
<div class="navbar-end">
<a href="/#why" class="navbar-item">
Why
</a>
<a href="/#what-is-data-mesh" class="navbar-item">
What
</a>
<div class="navbar-item has-dropdown is-hoverable">
<a href="/#how-to-design-a-data-mesh" class="navbar-link is-arrowless">
How
</a>
<div class="navbar-dropdown" id="navbarMenuArchitectureDropdown">
<a href="/#how-to-design-a-data-mesh" class="navbar-item">Data Mesh Architecture</a>
<hr class="navbar-divider">
<a href="/#data-product" class="navbar-item">Data Product</a>
<a href="/#federated-governance" class="navbar-item">Federated Governance</a>
<a href="/#analytical-data" class="navbar-item">Analytical Data</a>
<a href="/#ingesting" class="navbar-item">Ingesting</a>
<a href="/#clean-data" class="navbar-item">Clean Data</a>
<a href="/#analytics" class="navbar-item">Analytics</a>
<a href="/#data-platform" class="navbar-item">Data Platform</a>
<a href="/#enabling-team" class="navbar-item">Enabling Team</a>
</div>
</div>
<a href="#mesh" class="navbar-item">
Mesh
</a>
<div class="navbar-item has-dropdown is-hoverable">
<div class="navbar-link is-arrowless">
Specifications
</div>
<div class="navbar-dropdown" id="navbarMenuSpecificationsDropdown">
<a href="https://www.dataproduct-specification.com" class="navbar-item">Data Product Specification</a>
<a href="https://www.datacontract-specification.com" class="navbar-item">Data Contract Specification</a>
</div>
</div>
<div class="navbar-item has-dropdown is-hoverable">
<div class="navbar-link is-arrowless">
Open Source
</div>
<div class="navbar-dropdown" id="navbarMenuArchitectureToolsDropdown">
<a href="/data-product-canvas.html" class="navbar-item">Data Product Canvas</a><a href="/data-mesh-canvas.html" class="navbar-item">Data Mesh Canvas</a>
<a href="/fitness-test.html" class="navbar-item">Fitness Test</a><a href="https://cli.datacontract.com" class="navbar-item">Data Contract CLI</a>
<a href="/open-source/aws.html" class="navbar-item">AWS Terraform Modules</a>
<a href="/open-source/gcp.html" class="navbar-item">GCP Terraform Modules</a>
</div>
</div>
<div class="navbar-item has-dropdown is-hoverable">
<a href="/#tech-stacks" class="navbar-link is-arrowless">
Tech Stacks
</a>
<div class="navbar-dropdown" id="navbarMenuTechStackDropdown">
<a href="/tech-stacks/google-cloud-bigquery" class="navbar-item">
Google Cloud BigQuery
</a>
<a href="/tech-stacks/aws-s3-athena" class="navbar-item">
AWS S3 and AWS Athena
</a>
<a href="/tech-stacks/azure-synapse-analytics.html" class="navbar-item">
Azure Synapse Analytics
</a>
<a href="/tech-stacks/dbt-snowflake.html" class="navbar-item">
dbt and Snowflake
</a>
<a href="/tech-stacks/databricks.html" class="navbar-item">
Databricks
</a>
<a href="/tech-stacks/minio-trino.html" class="navbar-item">
MinIO and Trino
</a>
<a href="/tech-stacks/sap.html" class="navbar-item">
SAP
</a>
<a href="/tech-stacks/kafka-risingwave.html" class="navbar-item">
Kafka and RisingWave
</a>
</div>
</div>
<div class="navbar-item has-dropdown is-hoverable">
<a href="#domain-teams-journey" class="navbar-link is-arrowless">
Start the Journey
</a>
<div class="navbar-dropdown is-right" id="navbarMenuTransformationDropdown">
<a href="/#domain-teams-journey" class="navbar-item">
Domain Team’s Journey
</a>
<a href="/#data-teams-journey" class="navbar-item">
Data Team’s Journey
</a>
<a href="/literature.html" class="navbar-item">
Scientific Literature
</a>
<a href="/real-world-learnings.html" class="navbar-item">
Real World Learnings
</a>
<a href="https://data-ai.innoq.com/services/data-mesh?ref=dma-nav" class="navbar-item">
Get Help
</a>
</div>
</div>
<a href="https://www.datamesh-manager.com/?ref=dma-nav" class="navbar-item" style="color: #FF9B66">
Data Mesh Manager
</a>
</div>
</div>
</div>
</nav>
<div class="container">
<section class="section">
<nav class="breadcrumb" aria-label="breadcrumbs">
<ul>
<li><a href="/">Data Mesh Architecture</a></li>
<li class="is-active"><a href="#" aria-current="page">Real World Learnings</a></li>
</ul>
</nav>
<h1 class="title">Real World Learnings</h1>
<div class="content">
<p>
Implementing data mesh is nontrivial and highly dependent on the organization.
Here, we capture real world learnings based on observations and experiences, so you might be able to get something out of it for your own data mesh journey.
</p>
<p>
We want this collection of real world learnings to be a <strong>community effort</strong>. Please share your observations and learnings <a href="https://www.datamesh-architecture.com/#authors">with us</a> so we can put them up here for you, or <a href="https://github.com/datamesh-architecture/datamesh-architecture.com/edit/main/real-world-learnings.html">add them yourself via a pull request yourself</a>.
</p>
</div>
</section>
<section class="section content">
<h4 class="title is-4">Conflicting Goals</h4>
<div class="columns">
<div class="column">
<blockquote>
We are already more than busy implementing the user stories in our ever-growing backlog to meet the roadmap goals.
We can't take on this additional responsibility for analytical data and data products.
<br>
<br>
— <em>Engineering team</em>
</blockquote>
</div>
<div class="column">
<div class="title is-6">Learnings</div>
<ul>
<li>Team takes protective stance on their roadmap-driven goals</li>
<li>Team's current goals don't align with data mesh</li>
<li>Structural and cultural change in the organization necessary</li>
</ul>
</div>
</div>
</section>
<section class="section content">
<h4 class="title is-4">Cognitive Load</h4>
<div class="columns">
<div class="column">
<blockquote>
We are responsible for frontend, backend, databases, operations, and security.
And now we have to perform data analysis and build data products with tools we have no experience with.
<br>
<br>
— <em>Engineering team</em>
</blockquote>
</div>
<div class="column">
<div class="title is-6">Learnings</div>
<ul>
<li>Respect the <a href="https://itrevolution.com/cognitive-load/">cognitive load limit</a> that is individual for every team</li>
<li>Reduce their cognitive load with a supportive enabling team and a developer-friendly self-service data platform</li>
<li>Let them grow in their own pace</li>
</ul>
</div>
</div>
</section>
<section class="section content">
<h4 class="title is-4">It is an Investment</h4>
<div class="columns">
<div class="column">
<blockquote>
It took us 3 sprints to build our first data product.
And after a while, we had to invest even more because of hiccups and things we haven't thought about before.
<br>
<br>
— <em>Engineering team</em>
</blockquote>
</div>
<div class="column">
<div class="title is-6">Learnings</div>
<ul>
<li>Building a data product is a significant investment</li>
<li>It takes time to learn</li>
<li>Hiccups are to be expected as it's something new/different</li>
<li>Maintaining a data product requires continuous care</li>
<li>Enabling team and community of practice can help</li>
</ul>
</div>
</div>
</section>
<section class="section content">
<h4 class="title is-4">Choose the Right Time</h4>
<div class="columns">
<div class="column">
<blockquote>
We are in the middle of a huge transformation from on-premise monolith to cloud-native microservices.
And you come to us with this new hype topic, now?
<br>
<br>
— <em>Engineering team</em>
</blockquote>
</div>
<div class="column">
<div class="title is-6">Learnings</div>
<ul>
<li>Not the right time for this team</li>
<li>Come back when the current transformation comes to an end</li>
</ul>
</div>
</div>
</section>
<section class="section content">
<h4 class="title is-4">Profit Right Away</h4>
<div class="columns">
<div class="column">
<blockquote>
We previously were roadmap-driven by management.
By using the self-service data platform, we learned a lot about our product that sometimes even surprised us.
Based on hypotheses by our product owner and the team, we were able to craft and defend our own roadmap with user stories that had high business potential according to our data.
It feels great to measure the business value you contribute.
<br>
<br>
— <em>Engineering team</em>
</blockquote>
</div>
<div class="column">
<div class="title is-6">Learnings</div>
<ul>
<li>Starting the data mesh journey can quickly result in business value</li>
<li>The availability of a simple-to-use self-service data platform already brings high benefits</li>
<li>Qualify user stories before implementing allows focus</li>
<li>All this is already possible without even consuming or providing data products by the team</li>
</ul>
</div>
</div>
</section>
<section class="section content">
<h4 class="title is-4">Strong Reject</h4>
<div class="columns">
<div class="column">
<blockquote>
We get a lot of feature requests by our business and stakeholders.
After we started to validate those feature requests with data, we could reject at least half of them that weren't worthwhile from a business perspective.
<br>
<br>
— <em>Product owner</em>
</blockquote>
</div>
<div class="column">
<div class="title is-6">Learnings</div>
<ul>
<li>Increased capacity as they reject feature requests</li>
<li>Line of argument is easy with data</li>
</ul>
</div>
</div>
</section>
<section class="section content">
<h4 class="title is-4">Teams are different</h4>
<div class="columns">
<div class="column">
<blockquote>
In an ideal world, each team could develop a data product and profit right away through analytics.
In practice, this is not true for all teams.
Teams may be on <a href="https://www.datamesh-architecture.com/#level-1-operational-database-queries">level 1</a>
and gain good-enough results running queries on operational databases may have no immediate benefit to go the journey to level 4 to make their data available for other teams.
Other teams may not profit from analytics, as their business is driven by fulfilling standardized processes,
or they use standard software that already has good reporting for their domain.
<br>
<br>
— <em>Engineering team</em>
</blockquote>
</div>
<div class="column">
<div class="title is-6">Learnings</div>
<ul>
<li>Teams have different starting points</li>
<li>The return on investment may be unbalanced across the organization</li>
<li>Standard products, commercial off-the-shelf software, and cloud services may already provide good-enough data analytics capabilities</li>
</ul>
</div>
</div>
</section>
<section class="section content">
<h4 class="title is-4">SQL is the Common Ground</h4>
<div class="columns">
<div class="column">
<blockquote>
We knew SQL pretty well from our operational PostgreSQL database.
Our self-service data platform is based on Google BigQuery.
So, after some initial efforts getting data into BigQuery, we were quite at home defining analytical queries and building data products.
We needed to learn some SQL features that we did not use in the past, such as common-table-expressions and window functions, but this was not really an issue.
<br>
<br>
— <em>Engineering team</em>
</blockquote>
</div>
<div class="column">
<div class="title is-6">Learnings</div>
<ul>
<li>Engineers know SQL</li>
<li>SQL-based data platforms lower the barriers</li>
</ul>
</div>
</div>
</section>
<section class="section content">
<h4 class="title is-4">Developer Experience Matters</h4>
<div class="columns">
<div class="column">
<blockquote>
dbt is awesome.
<br>
<br>
— <em>Engineering team</em>
</blockquote>
</div>
<div class="column">
<div class="title is-6">Learnings</div>
<ul>
<li>Optimize the data platform for developer experience</li>
<li>The enabling team can provide best practices and tools that work well</li>
</ul>
</div>
</div>
</section>
<section class="section content">
<h4 class="title is-4">Pricing Drives Architecture</h4>
<div class="columns">
<div class="column">
<blockquote>
We use Athena with S3 for our data mesh.
We were surprised, how much additional know how we had to learn, particularly for performance and cost optimization:
How to efficiently transfer data from Kafka to S3?
Single events that we have in production got expensive and took too long to process.
Building batches increased latency and led to inconsistencies.
JSON as storage format took too much space and costs, so we had to learn Parquet.
With Athena, every query is billed and needs to be considered, e.g., when defining regular data quality checks.
<br>
<br>
— <em>Engineering team</em>
</blockquote>
</div>
<div class="column">
<div class="title is-6">Learnings</div>
<ul>
<li>Substantial efforts needed for data platform specific optimizations</li>
<li>Learning of additional unexpected technologies required</li>
<li>Cloud pricing models have significant impact on data architecture</li>
</ul>
</div>
</div>
</section>
<section class="section content">
<h4 class="title is-4">Chicken or the egg</h4>
<div class="columns">
<div class="column">
<blockquote>
We need data from an upstream team.
They don't get round to provide it.
So we helped ourselves and simply import operational data from a feed they already provided.
<br>
<br>
— <em>Engineering team</em>
</blockquote>
</div>
<div class="column">
<div class="title is-6">Learnings</div>
<ul>
<li>Dependencies can hurt and slow down</li>
<li>Workarounds using operational data can unblock</li>
<li>One can switch from using the workaround to data product once it's available</li>
</ul>
</div>
</div>
</section>
<section class="section content">
<h4 class="title is-4">Trust Your Data</h4>
<div class="columns">
<div class="column">
<blockquote>
We are annoyed.
We need data from an upstream team.
When they finally provided the data product, it had such a low quality that it was incomplete every other day.
We inform them, and they fix it after a few hours.
But it still happens again and again.
<br>
<br>
— <em>Engineering team</em>
</blockquote>
</div>
<div class="column">
<div class="title is-6">Learnings</div>
<ul>
<li>Teams might struggle to provide a high quality data product</li>
<li>Not enough know how and skills</li>
<li>Data without quality doesn't really help</li>
<li>Data quality monitoring is crucial</li>
</ul>
</div>
</div>
</section>
<section class="section content">
<h4 class="title is-4">Wished but Unused</h4>
<div class="columns">
<div class="column">
<blockquote>
We don't see any return of invest.
No one uses our data products, although they were requested by stakeholders.
For us, providing them was a waste of time.
<br>
<br>
— <em>Engineering team</em>
</blockquote>
</div>
<div class="column">
<div class="title is-6">Learnings</div>
<ul>
<li>Data as a product might fail similar to any other product</li>
<li>Own data analysis neglected</li>
</ul>
</div>
</div>
</section>
<section class="section">
<div class="content">
<p>
We want this collection of real world learnings to be a <strong>community effort</strong>. Please share your observations and learnings <a href="https://www.datamesh-architecture.com/#authors">with us</a> so we can put them up here for you, or <a href="https://github.com/datamesh-architecture/datamesh-architecture.com/edit/main/real-world-learnings.html">add them yourself via a pull request yourself</a>.
</p>
</div>
</section>
</div>
<footer class="footer">
<div class="content has-text-centered">
<p>
<a href="https://www.innoq.ai">
<img src="/images/supported-by-innoq--petrol-apricot.svg" alt="Supported by INNOQ" class="footer-logo" width="180" />
</a>
</p>
<p>
<a href="https://www.innoq.com/en/topics/data-mesh-workshop?ref=dma-footer">Workshop</a> 
<a href="https://www.socreatory.com/de/trainings/datamesh?ref=dma-footer">Training</a> 
<a href="https://www.innoq.com/en/impressum/">Legal Notice</a> 
<a href="https://www.innoq.com/en/datenschutz/">Privacy</a>
</p>
</div>
</footer>
<script async defer src="https://scripts.simpleanalyticscdn.com/latest.js"></script>
<noscript><img src="https://queue.simpleanalyticscdn.com/noscript.gif" alt="" referrerpolicy="no-referrer-when-downgrade" /></noscript>
<link rel="stylesheet" href="css/glightbox.css" />
<script src="js/glightbox.js"></script>
<script type="text/javascript">
const lightbox = GLightbox({});
</script>
<script src="js/anchor.min.js"></script>
<script>anchors.add();</script>
</body>
</html>