Fixes #24428,#24424 - UI Modularity streams - Host #7724

parthaa · 2018-09-25T19:48:03Z

This commit contains the UI functionality to list and enable modularity
streams. It displays the list of available streams and provides remote
action templates to the user to enable/disable, install/remove and
lock/unlock streams.

theforeman-bot · 2018-09-25T19:48:17Z

Issues: #24428 #24424

parthaa · 2018-09-27T04:51:48Z

Api Changes

new "name_stream_only" option

$ curl -X GET -s -k -u admin:changeme 'https://localhost/katello/api/v2/module_streams?host_ids[]=3&name_stream_only=1'  --globoff|json_reformat 
{
    "total": 4,
    "subtotal": 4,
    "page": 1,
    "per_page": 20,
    "error": null,
    "search": null,
    "sort": {
        "by": "name",
        "order": "asc"
    },
    "results": [
        {
            "name": "duck",
            "stream": "0",
            "id": "duck:0"
        },
        {
            "name": "kangaroo",
            "stream": "0",
            "id": "kangaroo:0"
        },
        {
            "name": "walrus",
            "stream": "0.71",
            "id": "walrus:0.71"
        },
        {
            "name": "walrus",
            "stream": "5.21",
            "id": "walrus:5.21"
        }
    ]
}

UI Changes

Search info on modules belonging to a host.
New Module Streams page for content hosts
You can select "View Details" to go to Module Details page
You can run various rex actions. You can also choose to "customize" first which should take you to a page like this
If it succeded you should be able to see something like
Click on the host name to find the result

Setup:

Going to have setup Remote execution on the dev environment. Rough instructions if yours is not very old

dnf install rubygem-smart_proxy_dynflow rubygem-smart_proxy_remote_execution_ssh tfm-rubygem-smart_proxy_dynflow_core
create an ssh key for your smart proxy via
sudo -u foreman-proxy ssh-keygen -f ~foreman-proxy/.ssh/id_rsa_foreman_proxy -N '' accept the default options
ssh-copy-id that key over to your content host and restart your smart proxy services
systemctl restart smart_proxy.service smart_proxy_dynflow_core.service
Check Infrastructure -> Smart proxies and make sure you have 'Dynflow,.. SSH' features

Importing Job templates:

start with a rake db:seed this should import a bunch of job templates
Also download the 2 templates https://github.com/theforeman/foreman_remote_execution/blob/master/app/views/templates/ssh/module_action.erb and (https://github.com/theforeman/community-templates/blob/develop/job_templates/module_action_-_ansible_default.erb - if you want to test ansible) and manually import em via -> Hosts -> Job templates -> Import - Upload the templates . Note these should make it by default when foreman-rex rpms get built, but for now its a manual step.

On your consumer
Steps to test

$ vagrant up fedora28
$ vagrant ssh fedora28
$ sudo su -
# install modularity stuff
$ dnf install fedora-repos-modular
$ dnf module list 
# Add following in your /etc/yum.repos.d/hosttools.repo
$ cat host-tools.repo 
[rhel-client]
name = rhel-client
baseurl = https://fedorapeople.org/groups/katello/releases/yum/nightly/client/f28/x86_64/ 
enabled = 1
gpgcheck = 0

$ cat fedora-nightly-copr.repo 
[rpmsoftwaremanagement-dnf-nightly]
name=Copr repo for dnf-nightly owned by rpmsoftwaremanagement
baseurl=https://copr-be.cloud.fedoraproject.org/results/rpmsoftwaremanagement/dnf-nightly/fedora-$releasever-$basearch/
type=rpm-md
skip_if_unavailable=True
gpgcheck=1
gpgkey=https://copr-be.cloud.fedoraproject.org/results/rpmsoftwaremanagement/dnf-nightly/pubkey.gpg
repo_gpgcheck=0
enabled=1
enabled_metadata=1
$ dnf update dnf
$ dnf install subscription-manager katello-host-tools -y
#Install Katello no arch rpm
# subman register with an activation key that enables modular repos

Alternatively you can also artificially add an entry in the db to ContentFacetRepository via rails console to
achieve the same. ::Katello::ContentFacetRepository.create!(:content_facet_id =><..>, :repository_id => <..>)

app/models/katello/module_stream.rb

app/controllers/katello/api/v2/module_streams_controller.rb

johnpmitsch · 2018-09-27T13:43:47Z

@parthaa I'll test functionality further, but a couple questions:

Why not return the actual id and create a new field for the name + stream i.e.

        {
            "id": 20
            "name": "duck",
            "stream": "0",
            "dnf_friendly_name": "duck:0"
        },

(that name was the best I can come up with right now)
I think hammer may require an id

What about this for the action menu? (cc @Rohoover)

initial menu looks like this:

Enable
Disable
Install
Uninstall
Lock
Unlock

Then that opens up a custom modal:
"Would you like to customize the template?"
"Yes, customize" "No, run template"
(wording may need to change, just an example)

parthaa · 2018-09-27T14:02:26Z

Why not return the actual id and create a new field for the name + stream i.e.
        {
            "id": 20
            "name": "duck",
            "stream": "0",
            "dnf_friendly_name": "duck:0"
        },

Remember name and stream are duplicated across many versions. The query generating this uses a group by / distinct. If you argue get the "latest" or the "first" id, I 'd then have to do a secondary query for each of these guys. This output is useful for the host module streams page and what I have there is sufficient.
If you prefer Not calling that field id and instead call it "name_stream" or something, then I am ok with that.

Rohoover · 2018-09-27T14:04:45Z

@johnpmitsch

Let's do it.

parthaa · 2018-09-27T14:07:53Z

@Rohoover @johnpmitsch @akofink just letting y'all know that we don't do this in package and errata install pages, but I m open to it,

johnpmitsch · 2018-09-27T14:08:15Z

@parthaa ok, that makes sense. I think id is a little misleading here since you can't query on it again, so my vote is for name_stream or something like that.

Can you check if the name_stream_only option works in hammer? If not, we should disable it in hammer.

parthaa · 2018-09-27T21:49:13Z

Also added search by module spec

parthaa · 2018-09-27T22:21:47Z

@parthaa ok, that makes sense. I think id is a little misleading here since you can't query on it again, so my vote is for name_stream or something like tha

I called it module_spec

$ curl -X GET -s -k -u admin:changeme 'https://localhost/katello/api/v2/module_streams?host_ids[]=3&name_stream_only=1'  --globoff|json_reformat
{
    "total": 4,
    "subtotal": 4,
    "page": 1,
    "per_page": 20,
    "error": null,
    "search": null,
    "sort": {
        "by": "name",
        "order": "asc"
    },
    "results": [
        {
            "name": "duck",
            "stream": "0",
            "module_spec": "duck:0"
        },
        {
            "name": "kangaroo",
            "stream": "0",
            "module_spec": "kangaroo:0"
        },
        {
            "name": "walrus",
            "stream": "0.71",
            "module_spec": "walrus:0.71"
        },
        {
            "name": "walrus",
            "stream": "5.21",
            "module_spec": "walrus:5.21"
        }
    ]
}

parthaa · 2018-09-27T22:22:22Z

@johnpmitsch @Rohoover

Let's do it.
Need to work on this

Rohoover · 2018-09-28T13:34:34Z

Out of curiosity, does searching by module_name or module_stream do anything?

As a general statement, we have an issue in the UI whereas the syntax for searching doesn't match the column headers. Many users have pointed this out to me.

johnpmitsch · 2018-09-28T13:51:25Z

@Rohoover These are the current fields you can search by, they match the column names in this case. (the column covered up is name)

What @parthaa is adding is a way to search by dnf's syntax, which they call 'module_spec' in their docs, this isn't a column since we don't store that info and create it from the name and stream.

Rohoover · 2018-09-28T14:10:34Z

@johnpmitsch

Great. Thanks!

parthaa · 2018-09-28T16:35:30Z

@johnpmitsch @Rohoover @akofink @jlsherrill @omkarkhatavkar updated the actions part

Rohoover · 2018-10-01T13:26:26Z

@parthaa

When looking at this table, how would you know whether something is unlocked in order to lock it or vice versa? Can we add in the lock icon column to this table or is that info displayed somewhere else. I'd argue the same for other functions. Perhaps I'm missing something...

Rohoover · 2018-10-01T19:46:07Z

+1 to @Rohoover's question. I'd like to see the 'current state' (locked, installed, etc.), and we shouldn't allow users to (e.g.) install an already installed module.

We don't have information about the status of the modules on the system. That hopefully should come in when sub-man sends us that in the future.

What 'is' a locked module? If it is something that cannot be changed, and we do not know it's current status, how do we report to a user when they try to complete an action on a locked item? How would they know to unlock it?

johnpmitsch

I like the UI and workflow a lot! 💯

Each action works well and I'm able to see the changes on my host. Code looks good as well.

One open question about the api docs, let me know if you see the same thing. Since you are using apipie according to the docs I don't think its an error in this change.

johnpmitsch · 2018-10-01T19:51:27Z

+1 to @Rohoover's question. I'd like to see the 'current state' (locked, installed, etc.), and we shouldn't allow users to (e.g.) install an already installed module.

We don't have information about the status of the modules on the system. That hopefully should come in when sub-man sends us that in the future.

What 'is' a locked module? If it is something that cannot be changed, and we do not know it's current status, how do we report to a user when they try to complete an action on a locked item? How would they know to unlock it?

We aren't able to currently get the locked/unlocked status from the host until we get changes from subscription manager. Currently, Katello has no idea about the current status of the modules on the host, it can only send actions to the host. Once we have the sub manager changes, we will add the locked/unlocked status as you have specified in the mockups. (partha correct me if any of this is wrong)

johnpmitsch

missed one thing, sorry to take back the ACK! Can you add tests for the new API params in test/controllers/api/v2/module_streams_controller_test.rb?

parthaa · 2018-10-01T23:56:44Z

missed one thing, sorry to take back the ACK! Can you add tests for the new API params in test/controllers/api/v2/module_streams_controller_test.rb?

added

parthaa · 2018-10-02T13:20:34Z

[test katello]

stbenjam · 2018-10-02T14:49:52Z

app/controllers/katello/remote_execution_controller.rb

@@ -40,6 +40,11 @@ def inputs
          { :errata => params[:name] }
        elsif feature_name == 'katello_service_restart'
          { :helper => params[:name] }
+        elsif feature_name =~ /module_stream/
+          fail HttpErrors::NotFound, _('module streams not found') if params[:module_spec].blank?
+          inputs = { :module_spec => params[:module_spec] }


Could we look into passing in :action too? I think @iNecas originally did this for package actions, where there's a separate template for each thing - install, update, etc, but I feel like the number of templates is getting out of control. I don't quite understand why Katello can't use a single job template for all the features

I don't recall any significant reason to chose one over the other. Should be doable.

parthaa · 2018-10-02T17:40:28Z

@stbenjam updated

stbenjam

From a code-perspective it looks good, just a typo I think. I haven't tested it yet

stbenjam · 2018-10-02T17:45:34Z

lib/katello/plugin.rb

@@ -277,6 +277,9 @@
    RemoteExecutionFeature.register(:katello_group_remove, N_("Katello: Remove Package Group"), :description => N_("Remove package group via Katello interface"), :provided_inputs => ['package'])
    RemoteExecutionFeature.register(:katello_errata_install, N_("Katello: Install Errata"), :description => N_("Install errata via Katello interface"), :provided_inputs => ['errata'])
    RemoteExecutionFeature.register(:katello_service_restart, N_("Katello: Service Restart"), :description => N_("Restart Services via Katello interface"), :provided_inputs => ['helpers'])
+    RemoteExecutionFeature.register(:katello_module_stream_action, N_("Katello: Module Stream Actions"),
+                                    :description => N_("Perform a Imodule stream action via Katello interface"),


Typo here? a Imodule

This commit contains the UI functionality to list and enable modularity streams. It displays the list of available streams and provides remote action templates to the user to enable/disable, install/remove and lock/unlock streams.

parthaa · 2018-10-02T20:24:27Z

[test-katello]

parthaa · 2018-10-02T20:27:28Z

[test katello]

johnpmitsch

works well 👍 I'm still wondering about the api docs, but I don't think that is related to your changes. Other than that everything looks good.

omkarkhatavkar · 2018-10-03T13:13:56Z

@parthaa 👍 Looks Great!

Some Questions/Issues:

curl -X GET -s -k -u admin:changeme 'https://localhost/katello/api/v2/module_streams?host_ids[]=1&name_stream_only=invalid' --globoff|json_reformat shows result, hence no validation for name_stream_only field.
Error message received while searching module streams on content host is that needed ? as others filter never have such error message.
Disabling one stream/version of module disables all other steams/versions. (Seems like that expected behaviour from module stream)
e.g. disabling walrus 5.21 disables even walrus 0.71
Even after finishing sucessful Job Invocations, job is still in pending status after some time it shows failed.
Even after finishing sucessful for Remote Action and Run hosts job, foreman task in pending status, how we are making sure that it was successful.

Still in-progress with more testing.

parthaa · 2018-10-03T14:21:05Z

4. Even after finishing sucessful Job Invocations, job is still in pending status after some time it shows failed.
Probably an issue with remote command setup. Need to figure out. Try running a regular remote command like ls -l via
Hosts -> -> Schedule Remote Job -> choose -> Category = Commands , template ]= Run Command SSh Default and command =ls -l or any bash command
See if you hit the same. If you do not hit any issue with that but hit issues with dnf module stuff then its a dnf module issue.

parthaa · 2018-10-03T14:28:06Z

curl -X GET -s -k -u admin:changeme 'https://localhost/katello/api/v2/module_streams?host_ids[]=1&name_stream_only=invalid' --globoff|json_reformat shows result, hence no validation for name_stream_only field.

it treats an invalid value as off. Thats the case with any boolean params in katello/foreman api. Feel free to file a foreman issue if you want that to raise error

omkarkhatavkar · 2018-10-03T14:46:22Z

This was the Issue No. 2, @parthaa if you want we can make a part of forman issue and work again later.

parthaa · 2018-10-03T14:49:36Z

@omkarkhatavkar can also reproduce this via

$ curl -X GET -s -k -u admin:changeme 'https://localhost/katello/api/v2/module_streams?host_ids[]=3&name_stream_only=1&search=foo' --globoff
{"total":4,"subtotal":null,"page":null,"per_page":null,"error":"Your search query was invalid. Please revise it and try again. The full error has been sent to the application logs.","search":"foo","sort":{"by":null,"order":null},"results":[]}

its the name_stream_only causing the error. Going by the ugly sql error.

2018-10-03T14:37:40 [E|app|51e34] Invalid search: PG::GroupingError: ERROR:  column "katello_module_streams.id" must appear in the GROUP BY clause or be used in an aggregate function
 | LINE 1: ...reams"."name", "katello_module_streams"."stream", "katello_m...
 |                                                              ^
 | : SELECT "katello_module_streams"."name", "katello_module_streams"."stream", "katello_module_streams"."id" AS t0_r0, "katello_module_streams"."name" AS t0_r1, "katello_module_streams"."uuid" AS t0_r2, "katello_module_streams"."version" AS t0_r3, "katello_module_streams"."context" AS t0_r4, "katello_module_streams"."stream" AS t0_r5, "katello_module_streams"."arch" AS t0_r6, "katello_module_streams"."created_at" AS t0_r7, "katello_module_streams"."updated_at" AS t0_r8, "katello_module_streams"."description" AS t0_r9, "katello_module_streams"."summary" AS t0_r10, "katello_repositories"."id" AS t1_r0, "katello_repositories"."name" AS t1_r1, "katello_repositories"."pulp_id" AS t1_r2, "katello_repositories"."created_at" AS t1_r3, "katello_repositories"."updated_at" AS t1_r4, "katello_repositories"."major" AS t1_r5, "katello_repositories"."minor" AS t1_r6, "katello_repositories"."gpg_key_id" AS t1_r7, "katello_repositories"."library_instance_id" AS t1_r8, "katello_repositories"."content_id" AS t1_r9, "katello_repositories"."arch" AS t1_r10, "katello_repositories"."label" AS t1_r11, "katello_repositories"."content_view_version_id" AS t1_r12, "katello_repositories"."relative_path" AS t1_r13, "katello_repositories"."url" AS t1_r14, "katello_repositories"."unprotected" AS t1_r15, "katello_repositories"."content_type" AS t1_r16, "katello_repositories"."product_id" AS t1_r17, "katello_repositories"."environment_id" AS t1_r18, "katello_repositories"."checksum_type" AS t1_r19, "katello_repositories"."docker_upstream_name" AS t1_r20, "katello_repositories"."distribution_version" AS t1_r21, "katello_repositories"."distribution_arch" AS t1_r22, "katello_repositories"."distribution_bootable" AS t1_r23, "katello_repositories"."distribution_family" AS t1_r24, "katello_repositories"."distribution_variant" AS t1_r25, "katello_repositories"."distribution_uuid" AS t1_r26, "katello_repositories"."mirror_on_sync" AS t1_r27, "katello_repositories"."download_policy" AS t1_r28, "katello_repositories"."verify_ssl_on_sync" AS t1_r29, "katello_repositories"."upstream_username" AS t1_r30, "katello_repositories"."upstream_password" AS t1_r31, "katello_repositories"."ostree_upstream_sync_policy" AS t1_r32, "katello_repositories"."ostree_upstream_sync_depth" AS t1_r33, "katello_repositories"."container_repository_name" AS t1_r34, "katello_repositories"."deb_releases" AS t1_r35, "katello_repositories"."deb_components" AS t1_r36, "katello_repositories"."deb_architectures" AS t1_r37, "katello_repositories"."ssl_ca_cert_id" AS t1_r38, "katello_repositories"."ssl_client_cert_id" AS t1_r39, "katello_repositories"."ssl_client_key_id" AS t1_r40, "katello_repositories"."ignore_global_proxy" AS t1_r41, "katello_repositories"."ignorable_content" AS t1_r42, "katello_repositories"."description" AS t1_r43, "katello_repositories"."docker_tags_whitelist" AS t1_r44, "katello_repositories"."source_repo_checksum_type" AS t1_r45 FROM "katello_module_streams" LEFT OUTER JOIN "katello_repository_module_streams" ON "katello_repository_module_streams"."module_stream_id" = "katello_module_streams"."id" LEFT OUTER JOIN "katello_repositories" ON "katello_repositories"."id" = "katello_repository_module_streams"."repository_id" WHERE (("katello_module_streams"."name" ILIKE '%module_spec%' OR "katello_module_streams"."uuid" ILIKE '%module_spec%' OR "katello_module_streams"."stream" ILIKE '%module_spec%' OR "katello_module_streams"."version" ILIKE '%module_spec%' OR "katello_module_streams"."context" ILIKE '%module_spec%' OR "katello_module_streams"."arch" ILIKE '%module_spec%' OR "katello_repositories"."name" ILIKE '%module_spec%')) AND "katello_module_streams"."id" IN ($1, $2, $3, $4, $5, $6, $7) GROUP BY "katello_module_streams"."name", "katello_module_streams"."stream"

omkarkhatavkar · 2018-10-03T15:02:44Z

For Issue No. 2 https://projects.theforeman.org/issues/25116 is created, not a blocker issue
Issue no. 4 and 5 are also getting reproduced using the bash script mentioned on #7724 (comment). Looks to be not related to this PR

parthaa · 2018-10-03T15:07:08Z

@omkarkhatavkar thanks

parthaa · 2018-10-03T20:49:32Z

[test katello]

parthaa · 2018-10-03T21:16:38Z

[test katello]

theforeman-bot added Needs testing Not yet reviewed labels Sep 25, 2018

parthaa force-pushed the content-hosts-modules-ui branch from 8ae3614 to 152c080 Compare September 25, 2018 19:49

akofink self-requested a review September 26, 2018 18:31

parthaa force-pushed the content-hosts-modules-ui branch from 152c080 to ed27bc5 Compare September 27, 2018 04:04

johnpmitsch reviewed Sep 27, 2018

View reviewed changes

app/models/katello/module_stream.rb Show resolved Hide resolved

johnpmitsch reviewed Sep 27, 2018

View reviewed changes

app/controllers/katello/api/v2/module_streams_controller.rb Show resolved Hide resolved

parthaa force-pushed the content-hosts-modules-ui branch from ed27bc5 to 0737619 Compare September 27, 2018 22:20

parthaa force-pushed the content-hosts-modules-ui branch from 0737619 to fcd7c83 Compare September 28, 2018 16:32

parthaa force-pushed the content-hosts-modules-ui branch from fcd7c83 to 9c65d9e Compare September 28, 2018 19:46

This comment has been minimized.

Sign in to view

theforeman-bot added the Waiting on contributor label Sep 28, 2018

parthaa force-pushed the content-hosts-modules-ui branch from a9dd373 to 2943a27 Compare September 28, 2018 20:24

theforeman-bot removed the Waiting on contributor label Sep 28, 2018

parthaa force-pushed the content-hosts-modules-ui branch from 2943a27 to 0f9b279 Compare September 28, 2018 20:51

johnpmitsch approved these changes Oct 1, 2018

View reviewed changes

johnpmitsch suggested changes Oct 1, 2018

View reviewed changes

parthaa force-pushed the content-hosts-modules-ui branch 2 times, most recently from 27995d3 to 02992cc Compare October 1, 2018 23:48

parthaa force-pushed the content-hosts-modules-ui branch from 02992cc to c21bd1d Compare October 2, 2018 14:25

stbenjam reviewed Oct 2, 2018

View reviewed changes

parthaa force-pushed the content-hosts-modules-ui branch from c21bd1d to 9f2cd4e Compare October 2, 2018 17:37

stbenjam reviewed Oct 2, 2018

View reviewed changes

Fixes #24428,#24424 - UI Modularity streams - Host

4112fdd

This commit contains the UI functionality to list and enable modularity streams. It displays the list of available streams and provides remote action templates to the user to enable/disable, install/remove and lock/unlock streams.

parthaa force-pushed the content-hosts-modules-ui branch from 9f2cd4e to 4112fdd Compare October 2, 2018 18:42

johnpmitsch approved these changes Oct 2, 2018

View reviewed changes

Refs #24428 - Remove lock/unlock

1e1d3bd

parthaa merged commit f9fac08 into Katello:master Oct 4, 2018

parthaa deleted the content-hosts-modules-ui branch October 4, 2018 01:05

Fixes #24428,#24424 - UI Modularity streams - Host #7724

Fixes #24428,#24424 - UI Modularity streams - Host #7724

Conversation

parthaa commented Sep 25, 2018

theforeman-bot commented Sep 25, 2018

parthaa commented Sep 27, 2018 • edited Loading

johnpmitsch commented Sep 27, 2018 • edited Loading

parthaa commented Sep 27, 2018

Rohoover commented Sep 27, 2018

parthaa commented Sep 27, 2018

johnpmitsch commented Sep 27, 2018

parthaa commented Sep 27, 2018

parthaa commented Sep 27, 2018

parthaa commented Sep 27, 2018

Rohoover commented Sep 28, 2018

johnpmitsch commented Sep 28, 2018

Rohoover commented Sep 28, 2018

parthaa commented Sep 28, 2018

This comment has been minimized.

Rohoover commented Oct 1, 2018

Rohoover commented Oct 1, 2018

johnpmitsch left a comment

Choose a reason for hiding this comment

johnpmitsch commented Oct 1, 2018

johnpmitsch left a comment

Choose a reason for hiding this comment

parthaa commented Oct 1, 2018

parthaa commented Oct 2, 2018

stbenjam Oct 2, 2018

Choose a reason for hiding this comment

iNecas Oct 2, 2018

Choose a reason for hiding this comment

parthaa commented Oct 2, 2018

stbenjam left a comment

Choose a reason for hiding this comment

stbenjam Oct 2, 2018

Choose a reason for hiding this comment

parthaa commented Oct 2, 2018

parthaa commented Oct 2, 2018

johnpmitsch left a comment

Choose a reason for hiding this comment

omkarkhatavkar commented Oct 3, 2018

parthaa commented Oct 3, 2018

parthaa commented Oct 3, 2018

omkarkhatavkar commented Oct 3, 2018

parthaa commented Oct 3, 2018

omkarkhatavkar commented Oct 3, 2018 • edited Loading

parthaa commented Oct 3, 2018

parthaa commented Oct 3, 2018

parthaa commented Oct 3, 2018

parthaa commented Sep 27, 2018 •

edited

Loading

johnpmitsch commented Sep 27, 2018 •

edited

Loading

omkarkhatavkar commented Oct 3, 2018 •

edited

Loading