Replace remote change op #1166

dbnicholson · 2017-09-13T11:43:28Z

The current add and delete remote change ops don't handle the case of modification of existing remotes. There are 2 cases this PR is trying to cover:

Making modifications to an existing remote. This is the merge operation. It's exposed in the CLI through a new ostree remote modify subcommand.
Setting a remote whether it previously existed or not. This is the replace operation. It's exposed in the CLI through the new --force option for ostree remote add. Here I've allowed replace to succeed even if the remote doesn't exist since it really seemed like a force add is the only place it would be used. Let me know what you think.

One question I had is if the new OstreeRepoRemoteChange enum values should be documented with a Since or something like that.

This also fixes https://bugzilla.gnome.org/show_bug.cgi?id=753373 by merging when the remote exists or adding when it doesn't.

rh-atomic-bot · 2017-09-13T16:35:28Z

☔ The latest upstream changes (presumably 3b315e1) made this pull request unmergeable. Please resolve the merge conflicts.

cgwalters · 2017-09-13T19:00:01Z

Let me look at this after the next release if that's OK.

dbnicholson · 2017-09-13T19:36:31Z

Sure, no rush. I was just scratching an itch since I was looking at all the remote stuff. That said, one of my goals with this is to convert flatpak to do reliable remote modifications, so having it in an ostree release is kind of important to me.

cgwalters

A few minor bits, otherwise looks great!

Also can you make the last commit have Closes: https://github.com/ostreedev/ostree/issues/1111 ?

cgwalters · 2017-09-15T01:51:37Z

tests/test-remotes-config-dir.js

+    'branches': GLib.Variant.new('as', ['master'])
+});
+repo.remote_change(null, OSTree.RepoRemoteChange.MERGE, 'baz',
+                   'http://baz2', opts, null);


One issue this raises is that * @url: URL for remote is not currently (nullable). We could make it so...it'd be one of those subtle IABI breaks that only affects compiled languages.

(The reason to make it nullable is to make it more convenient to change the options without changing the URL)

I hadn't thought of that. The url parameter is pretty odd in this operation. I had some more thoughts about this that I'll follow up on in the general PR conversation.

cgwalters · 2017-09-15T01:52:13Z

src/libostree/ostree-repo.c

+        {
+          g_propagate_error (error, g_steal_pointer (&local_error));
+          return FALSE;
+        }


g_clear_error (&local_error) here.

Originally I had that, but is it necessary here? It's declared with g_autoptr and never reused in this branch. But I can add it to be sure it doesn't leak.

Yes, but at this point the error is set - so we must clear it for the invocation of impl_repo_remote_add() right below, otherwise get could get the overwriting-error error.

The call to impl_repo_remote_add uses error, not local_error. local_error is only used to check if the repo exists. The caller's error is used when adding and local_error will be cleared regardless of what happens there. I don't mind clearing the error, but I don't think it's really necessary.

Ah yes, you're totally right. That said, I still think it's best practice to clear checked errors right after checking them. Imagine if someone later comes and wants to add another checked call, sees there's already a local_error value and doesn't realize it may have been written earlier.

cgwalters · 2017-09-15T01:53:04Z

src/libostree/ostree-repo.c

+        g_key_file_set_string (remote->options, remote->group, "metalink",
+                               url + strlen ("metalink="));
+      else
+        g_key_file_set_string (remote->options, remote->group, "url", url);


Hm, at this point we should extract this to a helper I'd say.

Right, I didn't spend a lot of time thinking about refactoring things. I mostly wanted to get the feature and tests in place to see what you thought. But I'll look through again and see if there are clear winners.

jlebon

Cool stuff! 👍

jlebon · 2017-09-21T14:32:44Z

src/ostree/ot-remote-builtin-modify.c

+  const char *remote_name;
+  char **iter;
+  g_autoptr(GVariantBuilder) optbuilder = NULL;
+  g_autoptr(GVariant) options = NULL;


This is minor, though note that we tend to declare things as they are assigned now. For iteration variables like iter and i, you can just declare in the for loop setup.

I thought about that. I just copied remote-builtin-add, but I can make these more modern-y.

dbnicholson · 2017-09-21T15:15:03Z

Thinking about the (nullable) url param for MERGE some more.

Maybe it would be best to move this out to it's own function like ostree_repo_remote_modify that either has a (nullable) URL parameter or just handles everything through the options a{sv} variant.

Furthermore, I later thought of another operation that would be useful - clearing an option so it's effectively reset to the default. That type of operation would require url to be NULL, and in that case how do you differentiate "I want to remove the url setting" (even though that would be silly) from "this is what the API requires".

So, one idea I had to tackle both of those is a separate function:

/**
 * ostree_repo_remote_modify:
 * @self: Repo
 * @remove: Remove the remote option
 * @name: Name of remote
 * @options: GVariant of type a{sv} when @remove is FALSE, type as when TRUE
 * @cancellable: Cancellable
 * @error: Error
 *
 * Set or remove options from an existing remote.
 *
 * Returns: %TRUE on success, %FALSE on failure
*/
gboolean
ostree_repo_remote_modify (OstreeRepo     *self,
                           gboolean        remove,
                           const char     *name,
                           GVariant       *options,
                           GCancellable   *cancellable,
                           GError        **error);

Any option you want to modify is in the variant. When removing options, the variant is simply as rather than a{sv} since there's no value needed. I'm not sure how I feel about the variant type being different depending on another parameter. What do you think?

cgwalters · 2017-09-25T20:58:01Z

Wouldn't one possibly want to both add/change and remove options at the same time? So perhaps what we want is a{smv} or so? (Can one do that? Huh fun, looks like GLib supports it but the current pygobject bindings can't)

rh-atomic-bot · 2017-10-20T13:20:48Z

☔ The latest upstream changes (presumably 9166605) made this pull request unmergeable. Please resolve the merge conflicts.

cgwalters · 2017-11-03T13:00:23Z

Would definitely like to get this in! Maybe it's simplest to punt on "modify" for now and have callers implement by using replace?

dbnicholson · 2018-05-11T15:16:42Z

Sorry for leaving this so long. My ostree hacking time is at a minimum these days. This is something I'd really like to have since we've had a never ending stream of remote configuration issues in Endless.

Another thought I had later was to just add accessors for an already defined remote like flatpak has. I like their APIs:

FLATPAK_EXTERN FlatpakRemote        *flatpak_installation_get_remote_by_name (FlatpakInstallation *self,
                                                                              const gchar         *name,
                                                                              GCancellable        *cancellable,
                                                                              GError             **error);
FLATPAK_EXTERN FlatpakRemote * flatpak_remote_new (const char    *name);
FLATPAK_EXTERN char *        flatpak_remote_get_url (FlatpakRemote *self);
FLATPAK_EXTERN void          flatpak_remote_set_url (FlatpakRemote *self,
                                                     const char    *url);

I could see something very similar in ostree using the currently experimental OstreeRemote. In addition to some common accessors like (get|set)_url() and (get|set)_gpg_verify(), I'd add a generic option accessor:

gchar *ostree_remote_get_option (OstreeRemote *remote, const gchar *option);
void ostree_remote_set_option (OstreeRemote *remote, const gchar *option, const gchar *value);

I could see possibly mirroring some of the GKeyFile APIs for specific values like (get|set)_boolean_option().

And finally an API to remove an option:

void ostree_remote_remove_option (OstreeRemote *remote, const gchar *option);

After that you'd have to make a call to persist the changes:

gboolean ostree_repo_remote_update (OstreeRepo *repo, OstreeRemote *remote, GCancellable cancellable, GError **error);

What do you think?

matthiasclasen · 2018-12-30T19:10:03Z

This would still be useful for flatpak

dbnicholson · 2019-01-09T17:43:43Z

Would definitely like to get this in! Maybe it's simplest to punt on "modify" for now and have callers implement by using replace?

OK, so as you say maybe it's best to just add the REPLACE change operation and worry about modifying in another PR. That's actually what I really want, but I personally think it would be best to flesh out the OstreeRemote API similar to flatpak but that would require more work..

Add the OSTREE_REPO_REMOTE_CHANGE_REPLACE operation to the OstreeRepoRemoteChange enum. This operation will add a remote or replace an existing one. It respects the location of the remote configuration file when replacing and the remotes config dir settings when adding a new remote.

This uses the OSTREE_REPO_REMOTE_CHANGE_REPLACE operation to add a remote or replace an existing one. This is roughly the opposite of --if-not-exists and will raise an error if both options are passed.

dbnicholson · 2019-01-10T16:57:43Z

I repushed now with just the replace operation (and changed the PR title). I added a fixup for the g_clear_error() @cgwalters requested and added a Since in the enum comment (should have been a fixup, too, sorry). I removed all the modification stuff and will move that to a separate PR after I have a bit more thinking about what the best interface is.

ramcq · 2019-02-08T11:56:26Z

@cgwalters @jlebon This API is causing much sadness in flatpak (see flatpak/flatpak#1665 and flatpak/flatpak#2508). Is it possible to give this PR a nudge forward so that Flatpak has at least one way to do remote modification in the config dir case?

cgwalters · 2019-02-08T14:36:37Z

LGTM, thanks!

@rh-atomic-bot r+ d010d5f

rh-atomic-bot · 2019-02-08T14:36:44Z

⌛ Testing commit d010d5f with merge b33a4e9...

Closes: #1166 Approved by: cgwalters

This uses the OSTREE_REPO_REMOTE_CHANGE_REPLACE operation to add a remote or replace an existing one. This is roughly the opposite of --if-not-exists and will raise an error if both options are passed. Closes: #1166 Approved by: cgwalters

rh-atomic-bot · 2019-02-08T14:54:56Z

☀️ Test successful - status-atomicjenkins
Approved by: cgwalters
Pushing b33a4e9 to master...

dbnicholson force-pushed the remote-change-ops branch from 7582250 to 4525b62 Compare September 13, 2017 17:34

cgwalters added for-after-release and removed for-after-release labels Sep 14, 2017

cgwalters reviewed Sep 15, 2017

View reviewed changes

jlebon reviewed Sep 21, 2017

View reviewed changes

cgwalters mentioned this pull request Sep 26, 2017

rpm-ostree kargs command coreos/rpm-ostree#1013

Closed

This was referenced Jan 26, 2018

Extend API to include ADD_OR_UPDATE operation #1434

Open

Merge remotes with the same name #1430

Open

dbnicholson mentioned this pull request May 11, 2018

Respect core.add-remotes-config-dir configuration option when adding remotes flatpak/flatpak#1665

Open

jlebon mentioned this pull request Aug 1, 2018

src/ostree: Fix split string containing multiple dots '.' #1696

Closed

dbnicholson added 4 commits January 10, 2019 07:10

man: Document remote add --if-not-exists option

40cf3c5

remote-add: Add --force option to add or replace remote

4d0a43e

This uses the OSTREE_REPO_REMOTE_CHANGE_REPLACE operation to add a remote or replace an existing one. This is roughly the opposite of --if-not-exists and will raise an error if both options are passed.

fixup! repo: Add OstreeRepoRemoteChange replace operation

d010d5f

dbnicholson changed the title ~~Merge and replace remote change ops~~ Replace remote change op Jan 10, 2019

dbnicholson force-pushed the remote-change-ops branch from 4525b62 to d010d5f Compare January 10, 2019 16:55

rh-atomic-bot pushed a commit that referenced this pull request Feb 8, 2019

man: Document remote add --if-not-exists option

8431bb5

Closes: #1166 Approved by: cgwalters

rh-atomic-bot closed this in 15ba200 Feb 8, 2019

Cynical-Optimist mentioned this pull request Oct 13, 2020

Updating project.conf with a new url does not cause an update of the ostree config buildstream-migration/bst-staging#217

Closed

BuildStream-Migration-Bot mentioned this pull request Feb 2, 2021

Updating project.conf with a new url does not cause an update of the ostree config apache/buildstream#217

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace remote change op #1166

Replace remote change op #1166

dbnicholson commented Sep 13, 2017

rh-atomic-bot commented Sep 13, 2017

cgwalters commented Sep 13, 2017

dbnicholson commented Sep 13, 2017

cgwalters left a comment

cgwalters Sep 15, 2017

dbnicholson Sep 21, 2017

cgwalters Sep 15, 2017

dbnicholson Sep 21, 2017

cgwalters Sep 25, 2017

dbnicholson Sep 25, 2017

cgwalters Sep 26, 2017

cgwalters Sep 15, 2017

dbnicholson Sep 21, 2017

jlebon left a comment

jlebon Sep 21, 2017

dbnicholson Sep 21, 2017

dbnicholson commented Sep 21, 2017

cgwalters commented Sep 25, 2017

rh-atomic-bot commented Oct 20, 2017

cgwalters commented Nov 3, 2017

dbnicholson commented May 11, 2018

matthiasclasen commented Dec 30, 2018

dbnicholson commented Jan 9, 2019

dbnicholson commented Jan 10, 2019

ramcq commented Feb 8, 2019

cgwalters commented Feb 8, 2019

rh-atomic-bot commented Feb 8, 2019

rh-atomic-bot commented Feb 8, 2019

Replace remote change op #1166

Replace remote change op #1166

Conversation

dbnicholson commented Sep 13, 2017

rh-atomic-bot commented Sep 13, 2017

cgwalters commented Sep 13, 2017

dbnicholson commented Sep 13, 2017

cgwalters left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlebon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbnicholson commented Sep 21, 2017

cgwalters commented Sep 25, 2017

rh-atomic-bot commented Oct 20, 2017

cgwalters commented Nov 3, 2017

dbnicholson commented May 11, 2018

matthiasclasen commented Dec 30, 2018

dbnicholson commented Jan 9, 2019

dbnicholson commented Jan 10, 2019

ramcq commented Feb 8, 2019

cgwalters commented Feb 8, 2019

rh-atomic-bot commented Feb 8, 2019

rh-atomic-bot commented Feb 8, 2019