v0.12.0

Breaking Changes

• MultiConnection was added to all our data-source plugins, all collected data will be wiped during the upgrade in order to ensure data integrity. #636 #1978 #2262
• pipeline.tasks was renamed to pipeline.plan #1951
• pipeline.plan[][].options.tasks was moved to pipeline.plan[][].subtasks #1412
• Separated Data Transformation Rules from Connection and bound to Scope #1548
• Introduced Dal interface for Database decoupling (deprecating *gorm.DB dependency) #2183
• All connection.rate_limit were renamed to connection.rate_limit_per_hour #2360
• All *users tables were renamed to *accounts, and the users table in Domain Layer is now representing Unique Identity #2431

Features

• New User Flow based on Blueprint #1951 #1979 #1980 #1981 #1982 #1986 #1987 #2468 #2472
• Team feature support #2361 #2426 #2427 #2428 #2429 #2430 #2431 #2477
• Support collecting pr/issue comments from Github / Gitlab #1746, #1758, #1760
• Automatic test including unit-test/e2e-test #1754 #1765 #2491 #1824 #1851 #1853
• New Dashboard "Community Overview" and other dashboard updates #1995 #2323 #2328
• Support collecting data from Jenkins Folders #2145
• Skip HTTPS certificate verification #2370
• Separated Github Tokens #2384 #2423
• Collect organization information for Github users #2408
• Skip collecting changelog for epic issues to reduce total time #2434
• Acquire user's consent before db migration #2440
• Collect Github comments/reviews #2452
• Throttling total number of pipelines running in parallel #2473

Others

• Remove outdated field for Jira plugin #1711
• Release scheduler #1772
• Total number of fixed bugs: 43
• Total number of commits: 359
• Files changed: 1288

New Contributors

• @SnowMoon-Dev made their first contribution in #2074
• @moonlzl made their first contribution in #2221
• @long2ice made their first contribution in #2200
• @marcemv90 made their first contribution in #2238
• @krshkodes made their first contribution in #2388
• @yumengwang03 made their first contribution in #2479
• @phzbbbbbbbb made their first contribution in #2492
• @Pranshu-Raj made their first contribution in #2488

Full Changelog: v0.11.0...v0.12.0

v0.11.0

Changelogs:

• Support Postgres
• Support collecting private git repos
• Improve unit test
• Redesign and implement configuration flow

v0.10.1

v0.10.1 is a minor release on top of v0.10.0 with mostly dashboard/docs updates.

Major Changes since Last Release (v0.10.0)

• Support incremental collection for Jira worklog
• Add a dashboard for weekly bug retro use case (grafana/WeeklyBugRetro.json)
• Add a generic Jenkins dashboard (grafana/Jenkins.json)
• Improve the Github basic metrics dashboard (grafana/GithubBasicMetrics.json)
• Update dashboards according to the latest domain layer schema
• Adopt a new file name convention to shorten file names
• Add docs for the advanced mode of Config UI

Full Changelog: v0.10.0...v0.10.1

v0.10.0

Major Changes since Last Release (v0.9.3)

• Pipeline Blueprint allows users to schedule recurring pipelines with cron expression
• Introduce DB migrations to DevLake to provide a smoother upgrade experience
• Add temporal mode for users with heavy data collection workload
• Add dbt plugin for users who want to define their own data transformations
• Store/cache API responses from data sources in the raw data layer for instant replay of data transformations

Full Changelog: v0.9.3...v0.10.0

Bug Fixes & Improvements since Last Beta Release (v0.10.0-beta1)

• Improve documentation for user setup and GitHub data collection process
• Fix incorrect created_date in domain layer
• Fix stuck pipelines due to retries
• Add users for plugin Jira

Full Changelog: v0.10.0-beta1...v0.10.0

New Contributors

• @liangjingyang made their first contribution in https://github.com/merico-dev/lake/pull/377
• @nitinvasanth made their first contribution in https://github.com/merico-dev/lake/pull/526
• @IronCore864 made their first contribution in https://github.com/merico-dev/lake/pull/747
• @ggcannard made their first contribution in https://github.com/merico-dev/lake/pull/752
• @MaximDub made their first contribution in https://github.com/merico-dev/lake/pull/885
• @Juneezee made their first contribution in https://github.com/merico-dev/lake/pull/1013
• @basicthinker made their first contribution in https://github.com/merico-dev/lake/pull/1178
• @daniel-hutao made their first contribution in https://github.com/merico-dev/lake/pull/1193
• @abeizn made their first contribution in https://github.com/merico-dev/lake/pull/1343

v0.10.0-beta1

Major Changes since Last Release (v0.9.3)

• Pipeline Blueprint allows users to schedule recurring pipelines with cron expression
• Introduce DB migrations to DevLake to provide a smoother upgrade experience
• Add temporal mode for users with heavy data collection workload
• Add dbt plugin for users who want to define their own data transformations
• Store/cache API responses from data sources in the raw data layer for instant replay of data transformations

Full Changelog: v0.9.3...v0.10.0-beta1

New Contributors

• @liangjingyang made their first contribution in https://github.com/merico-dev/lake/pull/377
• @nitinvasanth made their first contribution in https://github.com/merico-dev/lake/pull/526
• @IronCore864 made their first contribution in https://github.com/merico-dev/lake/pull/747
• @ggcannard made their first contribution in https://github.com/merico-dev/lake/pull/752
• @MaximDub made their first contribution in https://github.com/merico-dev/lake/pull/885
• @Juneezee made their first contribution in https://github.com/merico-dev/lake/pull/1013
• @basicthinker made their first contribution in https://github.com/merico-dev/lake/pull/1178
• @daniel-hutao made their first contribution in https://github.com/merico-dev/lake/pull/1193
• @abeizn made their first contribution in https://github.com/merico-dev/lake/pull/1343

v0.9.3

Fixed issues found in v0.9.2

1. gitlab api client reporting nil pointer #1450
2. gitlab commits collection not working #1459
3. mysql reporting unexpected EOF due to some operation would hold cursor for a long time #1452
4. convert gitlab_merge_request_commits to domain layer data pull_request_commits #1470

v0.9.2

Fixed issues found in v0.9.1

1. Improved gitextractor performance, from Forever to less than an hour on Clickhouse repo #1435
2. Fixed commit_sha being incorrect from refs table #1441
3. Skipped invalid pull_request_issues records when enriching data from github plugin #1438
4. Enhanced config-ui field behavior on connection settings and other tweaks #1427, #1422, #1436, #1444
5. Added author_id autho_name and description field to pull_request table #1431
6. Fixed pull_request_commits data corruption #1430
7. Fixed connection status on connection page from config-ui

v0.9.1

Fixed issues found in v0.9.0

1. v0.9.0 wouldn't pick up github bug/requirement settings unless devlake got restarted (https://github.com/merico-dev/lake/issues/1394)
2. v0.9.0 github settings - field saving issues (https://github.com/merico-dev/lake/issues/1393)
3. Github configuration: the placeholder of severity is incorrect (https://github.com/merico-dev/lake/issues/1384)

Other improvements:

4. Improve demo dashboards (https://github.com/merico-dev/lake/pull/1398)

v0.9.0

IMPORTANT: DevLake doesn't support Database Schema Migration yet, upgrading an existing instance is likely to break, we recommend that you deploy a new instance instead.

What's Changed

• fix: gitlab provider - disable project id mappings field by @e2corporation in https://github.com/merico-dev/lake/pull/1260
• EE integration bug fix by @mindlesscloud in https://github.com/merico-dev/lake/pull/1132
• feat: collect gitlab tags by @likyh in https://github.com/merico-dev/lake/pull/1267
• fix: create pipeline : optimize advanced menu accessibility by @e2corporation in https://github.com/merico-dev/lake/pull/1268
• refactor(github): improve github collection performance by @warren830 in https://github.com/merico-dev/lake/pull/1253
• fix(plugins): fix concurrent problems by @warren830 in https://github.com/merico-dev/lake/pull/1275
• docs: refine the main readme by @Startrekzky in https://github.com/merico-dev/lake/pull/1277
• docs: update Chinese main README by @Startrekzky in https://github.com/merico-dev/lake/pull/1283
• fix: multithread read empty config by @mappjzc in https://github.com/merico-dev/lake/pull/1263
• Feat add GitHub response field by @mappjzc in https://github.com/merico-dev/lake/pull/1282
• feat(github): extract relation between pr and issue by @warren830 in https://github.com/merico-dev/lake/pull/1285
• fix(jenkins): change delete scope by @warren830 in https://github.com/merico-dev/lake/pull/1290
• fix(github): fix test connection by @warren830 in https://github.com/merico-dev/lake/pull/1289
• fix: spell component by @mappjzc in https://github.com/merico-dev/lake/pull/1291
• feat: add validate for force required field by @mappjzc in https://github.com/merico-dev/lake/pull/1297
• fix: jira settings > enable reset query on multiselect fields by @e2corporation in https://github.com/merico-dev/lake/pull/1296
• fix: create pipeline > restore basic textarea editor in advanced mode by @e2corporation in https://github.com/merico-dev/lake/pull/1295
• fix: jira proxy should be lowercase by @klesh in https://github.com/merico-dev/lake/pull/1292
• Feat creat ref bug stats by @warren830 in https://github.com/merico-dev/lake/pull/1293
• fix(plugins): add support for proxy field in api by @warren830 in https://github.com/merico-dev/lake/pull/1298
• fix(gitlab): fix confusing tables by @warren830 in https://github.com/merico-dev/lake/pull/1300
• feat: reconfigure proxy and extend github settings by @e2corporation in https://github.com/merico-dev/lake/pull/1276
• jira server support by @mindlesscloud in https://github.com/merico-dev/lake/pull/1301
• fix: handle status code 404 by @mindlesscloud in https://github.com/merico-dev/lake/pull/1305
• fix(frontend): modify jira url for issuetyp and field by @warren830 in https://github.com/merico-dev/lake/pull/1309
• fix(jenkins): change DurationSec unit to second by @warren830 in https://github.com/merico-dev/lake/pull/1310
• chore(docs): release v0.9.0 by @warren830 in https://github.com/merico-dev/lake/pull/1313

Full Changelog: merico-dev/lake@v0.8.0...v0.9.0

v0.8.0

IMPORTANT: DevLake doesn't support Database Schema Migration yet, upgrading an existing instance is likely to break, we recommend that you deploy a new instance instead.

What's Changed

• docs: update issue templates by @Startrekzky in https://github.com/merico-dev/lake/pull/1173
• Automated tests by @joncodo in https://github.com/merico-dev/lake/pull/1174
• Update README by @basicthinker in https://github.com/merico-dev/lake/pull/1178
• Fix store pr labels individually by @warren830 in https://github.com/merico-dev/lake/pull/1180
• fix(plugins): pass pointer instead value by @warren830 in https://github.com/merico-dev/lake/pull/1183
• docs: update main README.md by @e2corporation in https://github.com/merico-dev/lake/pull/1152
• docs: fixed instructions for updated installation by @kevin-kline in https://github.com/merico-dev/lake/pull/1154
• refactor(plugins): use cursor to iterate dbdata by @warren830 in https://github.com/merico-dev/lake/pull/1185
• feat: setup multi-selectors for jira issue types w/ api proxy | 1064 by @e2corporation in https://github.com/merico-dev/lake/pull/1177
• Task cancelling for GitHub plugin by @kevin-kline in https://github.com/merico-dev/lake/pull/1179
• WIP by @joncodo in https://github.com/merico-dev/lake/pull/1188
• Feat: Adding Kubernetes deploy support by @daniel-hutao in https://github.com/merico-dev/lake/pull/1193
• refactor(domain layer,plugins): indicate which sub task failed by @warren830 in https://github.com/merico-dev/lake/pull/1181
• docs: update main README for v0.7.0 by @klesh in https://github.com/merico-dev/lake/pull/1197
• docs: update English and Chinese Readmes to v0.7.0 by @Startrekzky in https://github.com/merico-dev/lake/pull/1198
• hotfix: config-ui - fix create pipeline page | 1176 by @e2corporation in https://github.com/merico-dev/lake/pull/1191
• feat: git extractor support private repo by @mindlesscloud in https://github.com/merico-dev/lake/pull/1202
• E2E tests that can be run manually once per iteration by @joncodo in https://github.com/merico-dev/lake/pull/1189
• fix: refdiff rook_pkg_path typo by @klesh in https://github.com/merico-dev/lake/pull/1206
• feat:Encrypt for JiraAuth by @mappjzc in https://github.com/merico-dev/lake/pull/1205
• docs: fix typo for gitextractor plugin by @klesh in https://github.com/merico-dev/lake/pull/1208
• docs: update readme example code by @klesh in https://github.com/merico-dev/lake/pull/1209
• fix(refdiff): fix progress by @warren830 in https://github.com/merico-dev/lake/pull/1210
• Fix jenkins save builds by @warren830 in https://github.com/merico-dev/lake/pull/1204
• feat: create pipeline - advanced pipeline mode | 1127 by @e2corporation in https://github.com/merico-dev/lake/pull/1203
• fix(plugins): change assignee id by @warren830 in https://github.com/merico-dev/lake/pull/1211
• fix: use hooks to handle encryption and decryption by @mappjzc in https://github.com/merico-dev/lake/pull/1212
• Fix empty pipeline stage causing backend to crash by @klesh in https://github.com/merico-dev/lake/pull/1214
• docs: update developer setup procedure by @klesh in https://github.com/merico-dev/lake/pull/1215
• fix(jenkins): fix table definition by @warren830 in https://github.com/merico-dev/lake/pull/1219
• fix: github reporting 'invalid token' for invalid endpoint url by @klesh in https://github.com/merico-dev/lake/pull/1218
• fix(github): make sure of data integration by @warren830 in https://github.com/merico-dev/lake/pull/1224
• fix: remove empty workflow by @klesh in https://github.com/merico-dev/lake/pull/1225
• fix(github): github not collecting pull requests by @klesh in https://github.com/merico-dev/lake/pull/1227
• fix: retry with new nonce by @mindlesscloud in https://github.com/merico-dev/lake/pull/1228
• fix gitextractor arguments validation by @klesh in https://github.com/merico-dev/lake/pull/1236
• fix(refdiff): correct progress by @warren830 in https://github.com/merico-dev/lake/pull/1235
• fix: ae plugin tasks argument not working by @klesh in https://github.com/merico-dev/lake/pull/1237
• fix: collect pipeline find with project_id by @mappjzc in https://github.com/merico-dev/lake/pull/1241
• fix(feishu): decrease ratelimit by @warren830 in https://github.com/merico-dev/lake/pull/1242
• fix(frontend): to run in advanced mode by @warren830 in https://github.com/merico-dev/lake/pull/1240
• Fix jira collect board panic by @mappjzc in https://github.com/merico-dev/lake/pull/1244
• User setup fixes by @klesh in https://github.com/merico-dev/lake/pull/1254
• chore: release v0.8.0 by @klesh in https://github.com/merico-dev/lake/pull/1255
• Hotfix for github PR by @klesh in https://github.com/merico-dev/lake/pull/1259

New Contributors

• @basicthinker made their first contribution in https://github.com/merico-dev/lake/pull/1178
• @daniel-hutao made their first contribution in https://github.com/merico-dev/lake/pull/1193
• @mappjzc made their first contribution in https://github.com/merico-dev/lake/pull/1205

Full Changelog: merico-dev/lake@v0.7.0...v0.8.0