Merge develop into 3.0-dqlite #14717
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Represent changes to DeduceOrigin and Resolve charm where an OS is added to the origin if a series is present.
[JUJU-1830] [CONTRIBUTING.md] Add CLA section + other small fixes
juju#14638 This PR sets up a log sender for the model operator. Also as a drive-by, add missing tests for the model operator manifold package. ## Checklist - [x] Code style: imports ordered, good names, simple structure, etc - [x] Comments saying why design decisions were made - [x] Go unit tests, with comments saying what you're testing - [ ] ~[Integration tests](https://github.com/juju/juju/tree/develop/tests), with comments saying what you're testing~ - [ ] ~[doc.go](https://discourse.charmhub.io/t/readme-in-packages/451) added or updated in changed packages~ ## QA steps ```console $ juju add-model t1 --config logging-config="<root>=INFO;juju.worker.caasadmission=TRACE;" Added 't1' model on microk8s/localhost with credential 'microk8s' for user 'admin' $ juju deploy snappass-test Located charm "snappass-test" in charm-hub, revision 9 Deploying "snappass-test" from charm-hub charm "snappass-test", revision 9 in channel stable on focal $ juju debug-log --color --replay -m k1:t1 --include-module=juju.worker.caasadmission model-15bab7c8-bf6b-4ccd-8734-b8140c2ab313: 15:19:25 DEBUG juju.worker.caasadmission installing caas admission handler at /k8s/admission/15bab7c8-bf6b-4ccd-8734-b8140c2ab313 model-15bab7c8-bf6b-4ccd-8734-b8140c2ab313: 15:19:25 INFO juju.worker.caasadmission ensuring model k8s webhook configurations model-15bab7c8-bf6b-4ccd-8734-b8140c2ab313: 15:19:35 DEBUG juju.worker.caasadmission received admission request for snappass-test of /v1, Kind=Service in namespace t1 model-15bab7c8-bf6b-4ccd-8734-b8140c2ab313: 15:19:35 DEBUG juju.worker.caasadmission received admission request for snappass-test of /v1, Kind=Endpoints in namespace t1 model-15bab7c8-bf6b-4ccd-8734-b8140c2ab313: 15:19:35 DEBUG juju.worker.caasadmission received admission request for of discovery.k8s.io/v1, Kind=EndpointSlice in namespace t1 model-15bab7c8-bf6b-4ccd-8734-b8140c2ab313: 15:19:35 DEBUG juju.worker.caasadmission received admission request for snappass-test-endpoints of /v1, Kind=Service in namespace t1 model-15bab7c8-bf6b-4ccd-8734-b8140c2ab313: 15:19:35 DEBUG juju.worker.caasadmission received admission request for snappass-test-endpoints of /v1, Kind=Endpoints in namespace t1 model-15bab7c8-bf6b-4ccd-8734-b8140c2ab313: 15:19:35 DEBUG juju.worker.caasadmission received admission request for of discovery.k8s.io/v1, Kind=EndpointSlice in namespace t1 model-15bab7c8-bf6b-4ccd-8734-b8140c2ab313: 15:19:35 DEBUG juju.worker.caasadmission received admission request for snappass-test-application-config of /v1, Kind=Secret in namespace t1 model-15bab7c8-bf6b-4ccd-8734-b8140c2ab313: 15:19:36 DEBUG juju.worker.caasadmission received admission request for snappass-test of /v1, Kind=ServiceAccount in namespace t1 ... ``` ## Documentation changes No ## Bug reference No
…torage's filesystem's current status
… application's removal
…ontroller func from migration to includes folder.
Sometimes ec2 machine spawns only a private and public ip, so we should not for a third
…age-aws juju#14641 The recent migration of nw-assess-charm-storage tests from python to bash, created a legitimate test failure situation which was easy to miss most of the times, where the model being destroyed had persistent storage dangling and the ci job had to timeout resulting to a test failure in jenkins. This PR destroys the storage when removing the application and also addresses the situation where the test was trying to assert for the size allocated for a rootfs volume before the storage unit had been attached. Instead now we are waiting for the storage unit (rootfs) to be attached before asserting for the allocated size. ## Checklist *If an item is not applicable, use `~strikethrough~`.* - [x] Code style: imports ordered, good names, simple structure, etc - [x] Comments saying why design decisions were made - ~[ ] Go unit tests, with comments saying what you're testing~ - ~[ ] [Integration tests](https://github.com/juju/juju/tree/develop/tests), with comments saying what you're testing~ - ~[ ] [doc.go](https://discourse.charmhub.io/t/readme-in-packages/451) added or updated in changed packages~ ## QA steps Locally you can run ``` ./main.sh -v -p aws -c aws storage run_charm_storage ``` **But most importantly** `test-storage-aws` should pass in jenkins.
…ror_in_juju_login_in_single_char_login juju#14612 Juju login panics when the username is too short. For example character `q` as username. ## Checklist *If an item is not applicable, use `~strikethrough~`.* - [x] Code style: imports ordered, good names, simple structure, etc - [x] Comments saying why design decisions were made - ~[ ] Go unit tests, with comments saying what you're testing~ - ~[ ] [Integration tests](https://github.com/juju/juju/tree/develop/tests), with comments saying what you're testing~ - ~[ ] [doc.go](https://discourse.charmhub.io/t/readme-in-packages/451) added or updated in changed packages~ ## QA steps Logout out of current controller. ```sh juju login Enter username: q ERROR cannot log into controller "local": user name "q" not valid A user name may contain any case alpha-numeric characters, '+', '.', and '-'; '@' to specify an optional domain. The user name and domain must begin and end with alpha-numeric characters. Examples of valid users include bob, Bob@local, bob@somewhere-else, 0-a-f@123 ``` Should not panic. ## Bug reference https://bugs.launchpad.net/juju/+bug/1989182
Presetting default-series does not allow juju to know if the user intentionally set the value or not. Required for changes to deploy, where default-series and the series flag will be handled the same way. Make the default-series config a variable instead of copying the words around as a string.
Part of work to enable default-series to be the same as using the series flag when deploying.
Related to handling default series as if the series flag was used for deploy.
But only if it's been set. Update the error message where used.
Removes any model config default-series value. Part of work to make default-series from model config work like the series flag on deploy.
juju#14708 Currently, the `secret-set` command clears the secret content if no arg is provided. This PR ensures we only set data if the data is provided in the `secret-set` command arg. ```console $ juju exec --unit hello-kubecon/0 'uri=$(secret-add --owner=unit foo=bar); echo $uri;secret-set $uri --label=ccc;' secret:ccv84seffbaqh15v1q30 $ juju show-secret ccv84seffbaqh15v1q30 --reveal ccv84seffbaqh15v1q30: revision: 1 owner: hello-kubecon/0 label: ccc created: 2022-10-06T07:23:32Z updated: 2022-10-06T07:23:32Z content: {} # empty content ``` Drive-by: surface permission denied error for secret get and delete k8s API calls. ## Checklist - [x] Code style: imports ordered, good names, simple structure, etc - [x] Comments saying why design decisions were made - [x] Go unit tests, with comments saying what you're testing - [ ] ~[Integration tests](https://github.com/juju/juju/tree/develop/tests), with comments saying what you're testing~ - [ ] ~[doc.go](https://discourse.charmhub.io/t/readme-in-packages/451) added or updated in changed packages~ ## QA steps ```console $ juju exec --unit hello-kubecon/0 'uri=$(secret-add --owner=unit foo=bar); echo $uri; secret-set $uri --label=aaa;' secret:ccvs28mffbamreliccg0 $ juju exec --unit hello-kubecon/0 'uri=$(secret-add foo=bar); echo $uri; secret-set $uri --label=bbb;' secret:ccvs2buffbamreliccgg $ juju show-secret ccvs28mffbamreliccg0 --reveal ccvs28mffbamreliccg0: revision: 1 owner: hello-kubecon/0 label: aaa created: 2022-10-07T06:03:15Z updated: 2022-10-07T06:03:15Z content: foo: bar $ juju show-secret ccvs2buffbamreliccgg --reveal ccvs2buffbamreliccgg: revision: 1 owner: hello-kubecon label: bbb created: 2022-10-07T06:03:29Z updated: 2022-10-07T06:03:29Z content: foo: bar ``` ## Documentation changes No ## Bug reference No
juju#14707 Merge 2.9 Lots of conflicts - too many to list. juju#14697 juju#14700 juju#14643 juju#14703 juju#14702 juju#14704 juju#14705 ## Checklist - [X] Code style: imports ordered, good names, simple structure, etc - [X] Comments saying why design decisions were made - [X] Go unit tests, with comments saying what you're testing - ~[ ] [Integration tests](https://github.com/juju/juju/tree/develop/tests), with comments saying what you're testing~ - ~[ ] [doc.go](https://discourse.charmhub.io/t/readme-in-packages/451) added or updated in changed packages~ ## QA steps See PRs
This reverts commit 050a6ef. That will be included in a separate PR.
juju#14696 This updates the "juju info" text format output as per my [proposal doc](https://docs.google.com/document/d/1iehQm0R8w4OUJfakY8UXJCIMt073b8r-FW1ECzXcWmc/edit). Specifically, it shows the arch and bases where applicable as follows: - When no --series or --arch is specified, show just the latest revision for each channel (including the architecture for clarity) - When --series is specified, show one line (revision) per arch - When --arch is specified, show a comma-separated list of series (there's only one latest revision per arch) - When both series and arch are specified, show just the latest revision (but filtered by series and arch) This also changes the YAML and JSON output formats in 3.0, mainly so we can include the multiple revisions per track/risk combination (charms normally have one revision per architecture now, though note that old Charmstore charms can be returned from the CharmHub API as arch="all"). This PR also includes the update to "juju info" YAML/JSON output to show bases as "ubuntu:22.04" rather than "ubuntu/22.04". This should merge cleanly once [this commit](juju@3a27999#diff-2e6e3fd3f525cab09fb3666efde3b00ed439ce2d1bb583d14b9f307d96b12c54) is merged into develop. However, work to add `--base` and convert non-YAML/JSON output to show bases is left to a subsequent PR. ## Checklist - [x] Code style: imports ordered, good names, simple structure, etc - [x] Comments saying why design decisions were made - [x] Go unit tests, with comments saying what you're testing - [x] [Integration tests](https://github.com/juju/juju/tree/develop/tests), with comments saying what you're testing - [x] [doc.go](https://discourse.charmhub.io/t/readme-in-packages/451) added or updated in changed packages ## QA steps ```sh $ juju info ubuntu-advantage name: ubuntu-advantage ... channels: | latest/stable: 33 2022-06-13 (33) 1MB armhf xenial, bionic, focal, jammy latest/candidate: ↑ latest/beta: ↑ latest/edge: 38 2022-06-13 (38) 1MB arm64 xenial, bionic, focal, jammy $ juju info ubuntu-advantage --series jammy name: ubuntu-advantage ... channels: | latest/stable: 33 2022-06-13 (33) 1MB armhf latest/candidate: ↑ latest/beta: ↑ latest/edge: 34 2022-06-13 (34) 1MB amd64 38 2022-06-13 (38) 1MB arm64 37 2022-06-13 (37) 1MB armhf 35 2022-06-13 (35) 1MB ppc64el 36 2022-06-13 (36) 1MB s390x $ juju info ubuntu-advantage --arch amd64 name: ubuntu-advantage ... channels: | latest/stable: 16 2022-03-02 (16) 1MB xenial, bionic, focal latest/candidate: ↑ latest/beta: ↑ latest/edge: 34 2022-06-13 (34) 1MB xenial, bionic, focal, jammy $ juju info ubuntu-advantage --series jammy --arch amd64 name: ubuntu-advantage ... channels: | latest/stable: – latest/candidate: – latest/beta: – latest/edge: 34 2022-06-13 (34) 1MB ``` ## Bug reference https://bugs.launchpad.net/juju/+bug/1980181
juju#14715 Do no ops if no secrets need to be removed. ## Checklist - [x] Code style: imports ordered, good names, simple structure, etc - [x] Comments saying why design decisions were made - [x] Go unit tests, with comments saying what you're testing - [ ] ~[Integration tests](https://github.com/juju/juju/tree/develop/tests), with comments saying what you're testing~ - [ ] ~[doc.go](https://discourse.charmhub.io/t/readme-in-packages/451) added or updated in changed packages~ ## QA steps ```console $ juju deploy hello-kubecon Located charm "hello-kubecon" in charm-hub, revision 14 Deploying "hello-kubecon" from charm-hub charm "hello-kubecon", revision 14 in channel stable on ubuntu:20.04 $ juju exec --unit hello-kubecon/0 'uri=$(secret-add foo=bar); echo $uri; secret-set $uri --label=bbb;' secret:cd1m3nmffbau3ku2d650 $ juju secrets ID Owner Rotation Revision Last updated cd1m3nmffbau3ku2d650 hello-kubecon never 1 6 seconds ago $ juju destroy-model -y --debug --force --destroy-storage k1:t1 # at the same time watch model log $ juju debug-log --color --replay ``` ## Documentation changes No ## Bug reference No
dependencies to workers from which these have been removed.
no longer recruited anywhere.
which uses the leadership.Reader from the facade context. This is because we no longer determine application leaders in exported models from the database.
juju#14716 Remove the "centos7" hack used to model centos series. To do this, we now model os name and channel instead of series. In state, this means that the application's series attribute can be removed, and the charm origin platform stores os version (eg "20.04/stable") instead of series. And so we also correctly store centos info - "centos/7" instead of "centos/centos7". In core, we can now remove the helper methods and hacks needed to transform the centos series. Charm origin and platform are mandatory now, so we rely on those being present and remove the various checks to handle them not being specified, For operator k8s charm, we no longer store the series at "kubernetes". We use the underlying ubuntu version like we do for sidecar charms (we use focal). So now charm base is properly surfaced everywhere. Also remove all the 2.9 upgrade steps - these are no longer needed since 3.0 can only upgrade from 2.9.36 and at the time of writing, this is the latest 2.9 version. The steps were removed here to avoid having to waste time updating them to deal with the series->channel change. ## Checklist - [x ] Code style: imports ordered, good names, simple structure, etc - [x] Comments saying why design decisions were made - [x] Go unit tests, with comments saying what you're testing - ~[ ] [Integration tests](https://github.com/juju/juju/tree/develop/tests), with comments saying what you're testing~ - ~[ ] [doc.go](https://discourse.charmhub.io/t/readme-in-packages/451) added or updated in changed packages~ ## QA steps deploy single machine and k8s charms deploy machine and k8s bundles check juju status deploy slurmd (a centos 7 charm) check that model migration works
juju#14698 Here we finish removing all dealings with Mongo as regards leases. Changes included are: - Removal of some dangling trapdoor declarations that are no longer used following juju#14653. - Removal of all state logic for leases, including the Mongo collection. - Removal of the lease notify functionality from workers, which was previously used to update the Mongo collection upon lease claim/expiry. - Removal of state dependencies from lease-related workers that no longer need it. - Population of application leaders in exported models from the Raft-backed lease worker instead of Mongo. ## QA steps - Deploy a multi-unit workload and observe which unit becomes leader. - Run an action on the leader by targeting `<application>/leader`. - Remove the leader unit, observe that a new unit becomes leader, and run the leader action again. - Migrate the model to another controller and check that it completes successfully. ## Documentation changes None. ## Bug reference N/A
jujubot
added a commit
that referenced
this pull request
Feb 10, 2023
#15177 The following brings the 3.0-dqlite feature branch into the develop branch. ### Changes This brings in the dqlite database to sit along side the mongo database. Currently, only leases are implemented in Juju using dqlite, more controller base configuration and data will be subsequently moved over to dqlite once this branch has landed. #### Leases/Raft The whole raft implementation has been removed from Juju completely. This includes the following workers: - raft backstop - raft clusterer - raft log - raft transport - global clock updater In addition, the raft API implementation has also been removed. Instead, the lease has changed to handle the store (dqlite db) directly, improving readability and complexity. ### Jujud The `jujud` agent is now built using musl (specifically musl-gcc). This allows `juju` to be built statically embedding `dqlite` in the same process. There are still some rough edges when building and testing and when this lands, we expect to see some churn to polish any of those issues. Using `go test` is expected to still work as is, this is a last-minute change so that we can utilize sqlite directly for local tests. If you require to test with dqlite (linux only), then running `-tags="dqlite"` with builds/tests/installs is required. All CI jobs are required to run with the dqlite tag. Some notes: 1. `CGO_ENABLED=1` and `CGO_LDFLAGS_ALLOW="(-Wl,-wrap,pthread_create)|(-Wl,-z,now)"` are required if you're using dqlite directly. 2. You are expected to install musl directly on your system if you want to build, using `make musl-install`. This will require sudo. 3. For development purposes we will download dqlite `.a` files from an s3 bucket to facilitate the setup process. The tar file is sha256 summed to ensure no MITM. You can build these locally if you want to bypass s3 using `make dqlite-build-lxd`. This will spin up an lxd container to build. **Do not attempt** to run `make dqlite-build` locally, unless you know what you're doing. 4. To access dqlite from a controller, use `make repl`, this will open up a pseudo repl when you can then explore the database with. `.open <db name>` and then you can use SQL from there. 5. Cross compilation to other architectures can be done using `GOARCH` and `GOOS` before `make install` or `make build`. There are probably some things I've forgotten, expect a discourse post soon, which will highlight the development flow. ---- Two conflicts when merging. The resolution was to bring in the secret backends for the manifold tests and the controller config type changed for `DefaultMigrationMinionWaitMax`. ``` CONFLICT (content): Merge conflict in cmd/jujud/agent/machine/manifolds_test.go CONFLICT (content): Merge conflict in controller/config.go ``` c141b2e (upstream/3.0-dqlite) Merge pull request #15159 from SimonRichardson/system-install-musl-by-default 83656e2 Merge pull request #15156 from SimonRichardson/change-log-ddl 125c19d Fix static-analysis pipeline (#15168) 5abfa24 Merge pull request #15140 from SimonRichardson/allow-testing-on-mac 1dc60f6 (3.0-dqlite) Merge pull request #15153 from SimonRichardson/content-addressable-deps 5a1cd24 Merge pull request #15150 from jack-w-shaw/JUJU-2615_symlink_sudo 4502d63 Merge pull request #15148 from SimonRichardson/better-install-method 88941dd Merge pull request #15134 from SimonRichardson/bootstrap-dqlite-agent-tests 2551ffc Merge pull request #15130 from SimonRichardson/build-jujud-snap 0180a53 (origin/3.0-dqlite, manadart/3.0-dqlite) Merge pull request #15123 from SimonRichardson/fix-manifold-lease-expiry-tests fdf9cc7 Merge pull request #15115 from SimonRichardson/remove-jujud-main-test-file bf58843 Merge pull request #15113 from SimonRichardson/remove-api-raftlease-api-client f9419c0 Merge pull request #15112 from SimonRichardson/fix-initializing-state-twice 334d557 Merge pull request #15108 from SimonRichardson/github-action-go-build 2ee6e1a Merge pull request #15107 from SimonRichardson/cross-building-jujud 5a93305 Merge pull request #15087 from SimonRichardson/ensure-placement-of-file da95dc0 Merge pull request #15086 from SimonRichardson/more-sudo-changes 7b86376 Merge pull request #15085 from SimonRichardson/sudo-apt-get c4d4eb6 Merge pull request #15057 from SimonRichardson/dqlite-local-build 0ac79b3 Merge pull request #15061 from manadart/develop-into-3.0-dqlite adc20f7 Merge pull request #15043 from SimonRichardson/allow-overriding-arch-machine 8c02f22 Merge pull request #15048 from SimonRichardson/static-analysis-fix 4547c06 Merge pull request #15050 from manadart/dqlite-address-option d51b324 Merge pull request #15049 from manadart/dqlite-bootstrap-options 3801b78 Merge pull request #15047 from manadart/develop-into-3.0-dqlite 22d5247 Merge pull request #15037 from SimonRichardson/standardise-dqlite-build 25640a2 Merge pull request #15036 from SimonRichardson/remove-batch-fsm-controller-config dfa4cb1 Merge pull request #15041 from manadart/dqlite-fix-mock caf9481 Merge pull request #15034 from manadart/develop-into-3.0-dqlite c91985d Merge pull request #15035 from SimonRichardson/remove-typed-lease-error 42d17be Merge pull request #15009 from SimonRichardson/allow-repl-via-juju-ssh d798238 Merge pull request #15002 from manadart/dqlite-use-lease-store e4f0d39 Merge pull request #14918 from manadart/3.0-dqlite-lease-store 8315fb7 Merge pull request #14986 from manadart/dqlite-build-from-tags a73b947 Merge pull request #14927 from manadart/3.0-dqlite-lease-store-interface 1657a1d Merge pull request #14910 from manadart/3.0-dqlite-db-supply 27b23f3 Merge pull request #14909 from manadart/3.0-into-3.0-dqlite 6adff35 Merge pull request #14756 from manadart/develop-into-3.0-dqlite 37d81ff Merge pull request #14717 from manadart/develop-into-3.0-dqlite fe2edb8 Merge pull request #14671 from manadart/3.0-simplify-dbaccessor 1a09836 Merge pull request #14604 from manadart/3.0-bootstrap-controller-db 5ad011e Merge pull request #14652 from manadart/develop-into-3.0-dqlite 1c3d250 Merge pull request #14591 from manadart/develop-into-3.0-dqlite 229cd3e Merge pull request #14578 from manadart/3.0-dqlite-simplify 9d715ba Merge pull request #14565 from manadart/develop-into-3.0-dqlite 92ffd88 Merge pull request #14466 from manadart/develop-into-3.0-dqlite 57f67ce Merge pull request #14336 from manadart/develop-into-3.0-dqlite 648d354 Merge pull request #14364 from manadart/update-musl 198621d Merge pull request #14241 from manadart/develop-into-3.0-dqlite 0360db6 Merge pull request #14153 from manadart/develop-into-3.0-dqlite 17950b2 Merge pull request #14053 from manadart/develop-into-3.0-dqlite 9452026 Merge pull request #14016 from manadart/develop-into-3.0-dqlite 741baca Merge pull request #13963 from manadart/develop-into-3.0-dqlite 5449603 Merge pull request #13969 from manadart/dqlite-manifolds 7b612a0 Merge pull request #13944 from SimonRichardson/dqlite-develop
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merge from develop to bring in:
io/ioutilwithioandospackages #14631 from Juneezee/refactor/deprecate-ioutil