Develop into 3.0 dqlite #14153
Merged
Develop into 3.0 dqlite #14153
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
network API. Instead of falling back to the legacy behaviour, we now return a NotSupported error.
a stale version number is that which does not match the current juju agent version number
juju#14049 The first on several PRs to bring resources into line with how facades and api clients are commonly implemented. This PR starts with the api client layer. A lot of the changes are file moves and test fixes to use gomock. Some http helpers defined in api layer but used in apiserver layer were moved across. The agent api client is moved out to the agent sub-package. ## QA steps bootstrap a machine controller deploy the dummy-resource charm use juju resources to see the resource metadata It will result in a hook error because the bar resource is not supplied Attach the missing resource `juju attach-resource dummy-resource bar=baz.txt` The status message now shows the resource has been downloaded with `resource-get` ssh into the charm and inspect the resource content use juju resources to inspect the resource metadata bootstrap a k8s controller deploy the mysql-k8s charm It will stay allocating because there's no oci image in the store Attach the mysql-image `juju attach-resource mysql-k8s mysql-image=mariadb` The charm will complete deployment use juju resources to inspect the resource metadata
juju#14051 Fixes [lp#1958253](https://bugs.launchpad.net/juju/+bug/1958253). We now check whether the file provided to `XXXXXX-image` is json, yaml or binary, and return the corresponding error. I've also rewritten the `TestDeployDockerResource*` tests to be table-driven, which made it easy to add new tests covering my changes. ### QA steps ```sh $ juju deploy kubernetes-dashboard --resource dashboard-image=./invalid.json ERROR resource "dashboard-image": json parsing: ... $ juju deploy kubernetes-dashboard --resource dashboard-image=./invalid.yaml ERROR resource "dashboard-image": yaml: ... $ juju deploy kubernetes-dashboard --resource dashboard-image=./dashboard.tar ERROR resource "dashboard-image": expected json or yaml file ```
Allow for http dump functionality in the juju http package to be used
Fix for LP1940327.
juju#14048 Pick up changes to the juju/http package: http.ProxyFromEnvironment only gathers the environment proxy settings once per process and reuses the values. This is difficult for juju as the agent proxy settings can get changed on the fly. Setup to get the current proxy environment variable when running commands. Added loggers to other places where juju/http client was used, to enable httpdump output to be logged when required. The new version of juju/http adds log output of actual proxy values being used: set the juju model logging-config to `#http=TRACE` or `juju.http.middleware=TRACE` ## QA steps Follow the steps in the bug. # the following will enabling logging of the actual proxy settings found for use. $ juju model-config logging-config="juju.http.middlware=TRACE" -m controller ## Bug reference https://bugs.launchpad.net/juju/+bug/1973738
LP1973811. The key above doesn't exist in the controller config map, so our agent stream handling wasn't proper, nor was the warning output. Set the modelString value if empty or does not exist.
juju#14055 Add two simple examples to the juju status help text that show how to filter by application/unit status. I actually had no idea you could do this... ## Bug reference https://bugs.launchpad.net/juju/+bug/1727639
macos tests require more file descriptors then the default for mongo to behave.
Increase the number of open file descriptors.
juju#14059 Updates the `juju add-unit` help text to clarify the issue in [lp#1971956](https://bugs.launchpad.net/juju/+bug/1971956).
juju#14058 Like what was done for resources, the payload api and cli machinery is refactored to match standard practice. API clients for cli and agents are moved to api package and cli is moved to cmd/juju. A lot of the changes are file renames and edits. ## QA steps deploy a charm with payloads, eg ``` payloads: ingress-resource: type: crd api-token: type: secret ``` ``` $ juju exec --unit test/0 "payload-register crd ingress-resource 1234 tag1,tag2" $ juju payloads [Unit Payloads] Unit Machine Payload class Status Type Id Tags test/0 1 ingress-resource running crd 1234 tag1,tag2 $ juju exec --unit test/0 "payload-status-set ingress-resource 1234 stopped" $ juju payloads [Unit Payloads] Unit Machine Payload class Status Type Id Tags test/0 1 ingress-resource stopped crd 1234 tag1,tag2 ```
The public cloud is no longer available.
It was only used by the rackspace provider, which has also been removed.
The rackspace provider has been removed, thus no longer a valid cloud type. Replace in tests. Resolve compiler warnings.
…r local build juju;
juju#14130 In some previous refactoring of the config commands, I had inadvertently removed stdin support. This PR restores it and adds some unit tests. All config commands now support reading from stdin using the option `--file -`. Concretely, this looks like ``` juju config app1 | juju config app2 --file - juju model-config -m m1 --format=yaml | juju model-config -m m2 --file - --ignore-read-only-fields juju model-defaults -c c1 --format=yaml | juju model-defaults -c c2 --file - juju controller-config -c c1 --format=yaml | juju controller-config -c c2 --file - --ignore-read-only-fields ``` Behind the scenes, there have been some changes. The main one is adding a `ReadFile` method to `ConfigCommandBase`, which child commands can use to process files. This allows us to combine the `setConfig` and `setConfigFile` methods in the child commands, removing a lot of duplicate/similar code.
And change to using it for the userpass tests. Importing the provider allows for it to be registered and found during the tests.
juju#14133 Rackspace is no longer a public cloud. Removal of the rackspace provider, includes removal of the openstack NetworkingDecorator which was only used by rackspace. A bit of cleanup to remove reference to the multi-cloud feature flag in test, it hasn't been used for a while now. A continuation of juju#14127, which removes rackspace from clouds in 2.9 and will be merged into develop. ## QA steps Bootstrap an openstack cloud. The unit tests should be clean. ## Documentation changes We need to note the removal of the rackspace provider in juju 3.0
…flag juju#14101 Add `--no-color` to remove ansicolor codes from juju status output. - ~~[ ] Requires a [pylibjuju](https://github.com/juju/python-libjuju) change~~ - ~~[ ] Added [integration tests](https://github.com/juju/juju/tree/develop/tests) for the PR~~ - ~~[ ] Added or updated [doc.go](https://discourse.charmhub.io/t/readme-in-packages/451) related to packages changed~~ - ~~[ ] Comments answer the question of why design decisions were made~~ ## QA steps ```sh juju bootstrap localhost overlord juju add-model lxd-model juju deploy mediawiki juju deploy mysql --series=trusty juju deploy haproxy juju add-relation mediawiki:db mysql juju add-relation mediawiki haproxy juju expose haproxy juju add-unit -n 2 mediawiki juju status --no-color juju status --no-color --relations ``` *The output should not contain colored text*
…t-interface-file juju#14118 This PR includes changes: - try to read microk8s `kubeconfig` file from the content interface file first; - update microk8s content interface target; - check microk8s addons via k8s API; Notes: depends on canonical/microk8s#3193 ## Checklist - [ ] ~Requires a [pylibjuju](https://github.com/juju/python-libjuju) change~ - [ ] ~Added [integration tests](https://github.com/juju/juju/tree/develop/tests) for the PR~ - [ ] ~Added or updated [doc.go](https://discourse.charmhub.io/t/readme-in-packages/451) related to packages changed~ - [x] Comments answer the question of why design decisions were made ## QA steps ```console $ sudo snap install microk8s --channel=latest/edge/MK-530 $ sudo snap install juju_3.0-beta1_amd64.snap --dangerous $ sudo snap connect juju:dot-local-share-juju $ sudo snap connect juju:peers microk8s:microk8s $ juju bootstrap microk8s k1 --debug ``` ## Documentation changes No ## Bug reference No
…de_default_juju_status juju#14038 *Enable color by default for juju status* ## Checklist - ~[ ] Requires a [pylibjuju](https://github.com/juju/python-libjuju) change~ - [ ] Added [integration tests](https://github.com/juju/juju/tree/develop/tests) for the PR - ~[ ] Added or updated [doc.go](https://discourse.jujucharms.com/t/readme-in-packages/451) related to packages changed~ - [ ] Comments answer the question of why design decisions were made ## QA steps ```sh juju bootstrap localhost overlord juju add-model lxd-model juju deploy mediawiki juju deploy mysql --series=trusty juju deploy haproxy juju add-relation mediawiki:db mysql juju add-relation mediawiki haproxy juju expose haproxy juju add-unit -n 2 mediawiki juju status ``` *Ouptut should be colored*
jujubot
added a commit
that referenced
this pull request
Feb 10, 2023
#15177 The following brings the 3.0-dqlite feature branch into the develop branch. ### Changes This brings in the dqlite database to sit along side the mongo database. Currently, only leases are implemented in Juju using dqlite, more controller base configuration and data will be subsequently moved over to dqlite once this branch has landed. #### Leases/Raft The whole raft implementation has been removed from Juju completely. This includes the following workers: - raft backstop - raft clusterer - raft log - raft transport - global clock updater In addition, the raft API implementation has also been removed. Instead, the lease has changed to handle the store (dqlite db) directly, improving readability and complexity. ### Jujud The `jujud` agent is now built using musl (specifically musl-gcc). This allows `juju` to be built statically embedding `dqlite` in the same process. There are still some rough edges when building and testing and when this lands, we expect to see some churn to polish any of those issues. Using `go test` is expected to still work as is, this is a last-minute change so that we can utilize sqlite directly for local tests. If you require to test with dqlite (linux only), then running `-tags="dqlite"` with builds/tests/installs is required. All CI jobs are required to run with the dqlite tag. Some notes: 1. `CGO_ENABLED=1` and `CGO_LDFLAGS_ALLOW="(-Wl,-wrap,pthread_create)|(-Wl,-z,now)"` are required if you're using dqlite directly. 2. You are expected to install musl directly on your system if you want to build, using `make musl-install`. This will require sudo. 3. For development purposes we will download dqlite `.a` files from an s3 bucket to facilitate the setup process. The tar file is sha256 summed to ensure no MITM. You can build these locally if you want to bypass s3 using `make dqlite-build-lxd`. This will spin up an lxd container to build. **Do not attempt** to run `make dqlite-build` locally, unless you know what you're doing. 4. To access dqlite from a controller, use `make repl`, this will open up a pseudo repl when you can then explore the database with. `.open <db name>` and then you can use SQL from there. 5. Cross compilation to other architectures can be done using `GOARCH` and `GOOS` before `make install` or `make build`. There are probably some things I've forgotten, expect a discourse post soon, which will highlight the development flow. ---- Two conflicts when merging. The resolution was to bring in the secret backends for the manifold tests and the controller config type changed for `DefaultMigrationMinionWaitMax`. ``` CONFLICT (content): Merge conflict in cmd/jujud/agent/machine/manifolds_test.go CONFLICT (content): Merge conflict in controller/config.go ``` c141b2e (upstream/3.0-dqlite) Merge pull request #15159 from SimonRichardson/system-install-musl-by-default 83656e2 Merge pull request #15156 from SimonRichardson/change-log-ddl 125c19d Fix static-analysis pipeline (#15168) 5abfa24 Merge pull request #15140 from SimonRichardson/allow-testing-on-mac 1dc60f6 (3.0-dqlite) Merge pull request #15153 from SimonRichardson/content-addressable-deps 5a1cd24 Merge pull request #15150 from jack-w-shaw/JUJU-2615_symlink_sudo 4502d63 Merge pull request #15148 from SimonRichardson/better-install-method 88941dd Merge pull request #15134 from SimonRichardson/bootstrap-dqlite-agent-tests 2551ffc Merge pull request #15130 from SimonRichardson/build-jujud-snap 0180a53 (origin/3.0-dqlite, manadart/3.0-dqlite) Merge pull request #15123 from SimonRichardson/fix-manifold-lease-expiry-tests fdf9cc7 Merge pull request #15115 from SimonRichardson/remove-jujud-main-test-file bf58843 Merge pull request #15113 from SimonRichardson/remove-api-raftlease-api-client f9419c0 Merge pull request #15112 from SimonRichardson/fix-initializing-state-twice 334d557 Merge pull request #15108 from SimonRichardson/github-action-go-build 2ee6e1a Merge pull request #15107 from SimonRichardson/cross-building-jujud 5a93305 Merge pull request #15087 from SimonRichardson/ensure-placement-of-file da95dc0 Merge pull request #15086 from SimonRichardson/more-sudo-changes 7b86376 Merge pull request #15085 from SimonRichardson/sudo-apt-get c4d4eb6 Merge pull request #15057 from SimonRichardson/dqlite-local-build 0ac79b3 Merge pull request #15061 from manadart/develop-into-3.0-dqlite adc20f7 Merge pull request #15043 from SimonRichardson/allow-overriding-arch-machine 8c02f22 Merge pull request #15048 from SimonRichardson/static-analysis-fix 4547c06 Merge pull request #15050 from manadart/dqlite-address-option d51b324 Merge pull request #15049 from manadart/dqlite-bootstrap-options 3801b78 Merge pull request #15047 from manadart/develop-into-3.0-dqlite 22d5247 Merge pull request #15037 from SimonRichardson/standardise-dqlite-build 25640a2 Merge pull request #15036 from SimonRichardson/remove-batch-fsm-controller-config dfa4cb1 Merge pull request #15041 from manadart/dqlite-fix-mock caf9481 Merge pull request #15034 from manadart/develop-into-3.0-dqlite c91985d Merge pull request #15035 from SimonRichardson/remove-typed-lease-error 42d17be Merge pull request #15009 from SimonRichardson/allow-repl-via-juju-ssh d798238 Merge pull request #15002 from manadart/dqlite-use-lease-store e4f0d39 Merge pull request #14918 from manadart/3.0-dqlite-lease-store 8315fb7 Merge pull request #14986 from manadart/dqlite-build-from-tags a73b947 Merge pull request #14927 from manadart/3.0-dqlite-lease-store-interface 1657a1d Merge pull request #14910 from manadart/3.0-dqlite-db-supply 27b23f3 Merge pull request #14909 from manadart/3.0-into-3.0-dqlite 6adff35 Merge pull request #14756 from manadart/develop-into-3.0-dqlite 37d81ff Merge pull request #14717 from manadart/develop-into-3.0-dqlite fe2edb8 Merge pull request #14671 from manadart/3.0-simplify-dbaccessor 1a09836 Merge pull request #14604 from manadart/3.0-bootstrap-controller-db 5ad011e Merge pull request #14652 from manadart/develop-into-3.0-dqlite 1c3d250 Merge pull request #14591 from manadart/develop-into-3.0-dqlite 229cd3e Merge pull request #14578 from manadart/3.0-dqlite-simplify 9d715ba Merge pull request #14565 from manadart/develop-into-3.0-dqlite 92ffd88 Merge pull request #14466 from manadart/develop-into-3.0-dqlite 57f67ce Merge pull request #14336 from manadart/develop-into-3.0-dqlite 648d354 Merge pull request #14364 from manadart/update-musl 198621d Merge pull request #14241 from manadart/develop-into-3.0-dqlite 0360db6 Merge pull request #14153 from manadart/develop-into-3.0-dqlite 17950b2 Merge pull request #14053 from manadart/develop-into-3.0-dqlite 9452026 Merge pull request #14016 from manadart/develop-into-3.0-dqlite 741baca Merge pull request #13963 from manadart/develop-into-3.0-dqlite 5449603 Merge pull request #13969 from manadart/dqlite-manifolds 7b612a0 Merge pull request #13944 from SimonRichardson/dqlite-develop
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merge from develop to bring in:
juju execless verbose #14099 from barrettj12/exec-quietprinfo.json#14089 from barrettj12/rm-prinfomodel-defaultsto use ConfigCommandBase #14072 from barrettj12/cfgbase-dfltsNo conflicts