Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate SQL and shadow backend from Uber internal repo #294

Merged
merged 3 commits into from
Jan 12, 2021

Conversation

SQUIDwarrior
Copy link
Contributor

This PR adds a new, SQL-based backend option for the tag datastore. The SQL backend offers increased performance over filesystem based backends for very large Docker registries as it takes advantage of optimizations such as indexing. Block-based storage systems are particularly bad at handling many small files, or walking directory structures. With the SQL backend, when listing the entire Docker catalog you are doing a simple SELECT over an indexed table, rather than a costly filesystem walk.

This also includes a backend abstraction option called a "shadow" backend. This backend is designed for transitioning Kraken to a new backend by allowing admins to specify an "active" and "shadow" backend. Writes are sent
to both backends, but reads only occur from the active. This ensures data consistency between the backends, and
allows the old, proven backed to act as a safety net in the case where the new backend fails or has to be taken
offline due to some unforeseen problems.

@codecov
Copy link

codecov bot commented Jan 7, 2021

Codecov Report

Merging #294 (6070a08) into master (1e07dd7) will increase coverage by 0.12%.
The diff coverage is 69.81%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #294      +/-   ##
==========================================
+ Coverage   64.76%   64.89%   +0.12%     
==========================================
  Files         186      188       +2     
  Lines        7822     8044     +222     
==========================================
+ Hits         5066     5220     +154     
- Misses       2001     2037      +36     
- Partials      755      787      +32     
Impacted Files Coverage Δ
lib/backend/shadowbackend/client.go 64.94% <64.94%> (ø)
lib/backend/sqlbackend/client.go 73.60% <73.60%> (ø)
core/peer_info.go 85.71% <0.00%> (-7.15%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1e07dd7...6070a08. Read the comment docs.

@codygibb
Copy link
Contributor

codygibb commented Jan 8, 2021

(Talked offline, posting for posterity).

Can we add the license to the top of each file?

// Copyright (c) 2016-2020 Uber Technologies, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

Also, can we remove these shell scripts? They seem pretty bespoke and will be difficult to maintain going forward.

@SQUIDwarrior
Copy link
Contributor Author

@codygibb Made updates, let me know if this looks good.

@codygibb codygibb merged commit 645466c into uber:master Jan 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants