Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prevent empty asset names #1912

Merged
merged 3 commits into from
Dec 8, 2018
Merged

Prevent empty asset names #1912

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7fcb9bf
Skip cleanup of assets with empty name
Martchus Dec 7, 2018
1393e21
Prevent registering empty assets
Martchus Dec 7, 2018
71172fc
Test handling of assets with invalid name
Martchus Dec 7, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 13 additions & 7 deletions lib/OpenQA/Schema/Result/Jobs.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1417,26 +1417,32 @@ sub register_assets_from_settings {

# check assets and fix the file names
for my $k (keys %assets) {
my $a = $assets{$k};
if ($a->{name} =~ /\//) {
log_info "not registering asset $a->{name} containing /";
my $asset = $assets{$k};
my ($name, $type) = ($asset->{name}, $asset->{type});
unless ($name && $type) {
log_info 'not registering asset with empty name or type';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would add the asset type here and upgrade the log level to error

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, there is no type - that's the error (in case)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought making a different if or (unless) for $name and $type would be a bit over-engineering for this hopefully rare error case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeed it is, but actually my comment was about knowing the type of asset, rather than knowing the value of $name

delete $assets{$k};
next;
}
my $f_asset = _asset_find($a->{name}, $a->{type}, \@parents);
if ($name =~ /\//) {
log_info "not registering asset $name containing /";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would change this to error too. While it's not fatal, it's something to keep an eye on...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's an error, it should be reported to the client - and raise an exception. Virtualization uses REPO_0=fixed/SLE... because they use REPO_0 in the pxe bootloader

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can try to propagate this error higher in the stack. That would be a little bit more work because it looks like the method is called 2 times and each not that "close to the user".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with @coolo, this is input validation and failures should be reported to the client instead of getting logged.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'll change this. But for me this is not a reason to postpone merging this because we had this logging before, too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on that too @Martchus. Might be worth revisiting input validation in general, since pretty much none of the API controllers do any validation themselves and leave it all to the model layer. Which is quite risky.

delete $assets{$k};
next;
}
my $f_asset = _asset_find($name, $type, \@parents);
unless (defined $f_asset) {
# don't register asset not yet available
delete $assets{$k};
next;
}
$a->{name} = $f_asset;
$asset->{name} = $f_asset;
$updated{$k} = $f_asset;
}

for my $a (values %assets) {
for my $asset (values %assets) {
# avoid plain create or we will get unique constraint problems
# in case ISO_1 and ISO_2 point to the same ISO
my $aid = $self->result_source->schema->resultset('Assets')->find_or_create($a);
my $aid = $self->result_source->schema->resultset('Assets')->find_or_create($asset);
$self->jobs_assets->find_or_create({asset_id => $aid->id});
}

Expand Down
20 changes: 15 additions & 5 deletions lib/OpenQA/Schema/ResultSet/Assets.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,15 +40,19 @@ sub register {
my ($self, $type, $name, $missingok) = @_;
$missingok //= 0;

unless ($name) {
log_warning "attempt to register asset with empty name";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, we had "log_info" in other cases and probably same reasoning as above applies: If it's an user error we should propagate to the user, otherwise info or debug, not warn, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just a sanity check which actually could be removed because the PR actually ensures that this method is not called with an empty name. If that would be true, the user hasn't done anything wrong.

Copy link
Member

@kraih kraih Dec 7, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems very strange that a sanity check would log a warning instread of throwing an exception.

return;
}
our %types = map { $_ => 1 } qw(iso repo hdd other);
unless ($types{$type}) {
log_warning "asset type '$type' invalid";
return;
}
unless (locate_asset $type, $name, mustexist => 1) {
if (!$missingok) {
log_warning "no file found for asset '$name' type '$type'";
}
return 'missing' if ($missingok);

log_warning "no file found for asset '$name' type '$type'";
return;
}

Expand Down Expand Up @@ -205,14 +209,20 @@ END_SQL
# prefetch all assets
my $assets_arrayref = $dbh->selectall_arrayref($prioritized_assets_query);
for my $asset_array (@$assets_arrayref) {
my $id = $asset_array->[0];
my $id = $asset_array->[0];
my $name = $asset_array->[1];
if (!$name) {
OpenQA::Utils::log_warning("asset cleanup: Skipping asset $id because its name is empty.");
next;
}

my $type = $asset_array->[4];
my $fixed = $asset_array->[5];
my $dirname = ($fixed ? $type . '/fixed/' : $type . '/');
my $max_job = $asset_array->[6];
my %asset = (
id => $id,
name => ($dirname . $asset_array->[1]),
name => ($dirname . $name),
t_created => $asset_array->[2],
size => $asset_array->[3],
type => $type,
Expand Down
31 changes: 20 additions & 11 deletions lib/OpenQA/WebAPI/Controller/API/V1/Asset.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,19 +50,28 @@ sub register {

my $type = $self->param('type');
my $name = $self->param('name');
return $self->render(
json => {
error => 'type and name must not be empty'
},
status => 400,
) unless $type && $name;

my $asset = $self->app->schema->resultset("Assets")->register($type, $name);
my $asset = $self->app->schema->resultset('Assets')->register($type, $name);
return $self->render(
json => {
error => 'registering asset failed',
},
status => 400
) unless $asset;

my $status = 200;
my $json = {};
if ($asset) {
$json->{id} = $asset->id;
$self->emit_event('openqa_asset_register', {id => $asset->id, type => $type, name => $name});
}
else {
$status = 400;
}
$self->render(json => $json, status => $status);
$self->emit_event('openqa_asset_register', {id => $asset->id, type => $type, name => $name});
$self->render(
json => {
id => $asset->id,
},
status => 200,
);
}

=over 4
Expand Down
20 changes: 17 additions & 3 deletions lib/OpenQA/WebAPI/Controller/API/V1/Iso.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,8 +416,11 @@ sub schedule_iso {
my ($self, $args) = @_;
# register assets posted here right away, in case no job
# templates produce jobs.
for my $a (values %{parse_assets_from_settings($args)}) {
$self->app->schema->resultset("Assets")->register($a->{type}, $a->{name}, 1);
my $assets = $self->app->schema->resultset('Assets');
for my $asset (values %{parse_assets_from_settings($args)}) {
my ($name, $type) = ($asset->{name}, $asset->{type});
return {error => 'Asset type and name must not be empty.'} unless $name && $type;
return {error => "Failed to register asset $name."} unless $assets->register($type, $name, 1);
}
my $deprioritize = delete $args->{_DEPRIORITIZEBUILD} // 0;
my $deprioritize_limit = delete $args->{_DEPRIORITIZE_LIMIT};
Expand Down Expand Up @@ -604,7 +607,18 @@ sub create {
}
}

my $scheduled_jobs = $self->schedule_iso(\%params);
my $scheduled_jobs = $self->schedule_iso(\%params);
my $error = $scheduled_jobs->{error};
return $self->render(
json => {
error => $error,
count => 0,
ids => [],
failed => {},
},
status => 400,
) if $error;

my $successful_job_ids = $scheduled_jobs->{successful_job_ids};
my $failed_job_info = $scheduled_jobs->{failed_job_info};
my $created_job_count = scalar(@$successful_job_ids);
Expand Down
71 changes: 68 additions & 3 deletions t/37-limit_assets.t
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ use Test::MockModule;
use Test::Output qw(stdout_like);
use OpenQA::Test::Case;
use OpenQA::Task::Asset::Limit;
use OpenQA::Utils;
use OpenQA::WebAPI::Controller::API::V1::Iso;

# allow catching log messages via stdout_like
delete $ENV{OPENQA_LOGFILE};
Expand All @@ -39,6 +41,8 @@ my $test_case = OpenQA::Test::Case->new;
$test_case->init_data;
my $t = Test::Mojo->new('OpenQA::WebAPI');

note("Asset directory: $OpenQA::Utils::assetdir");

# scan initially for untracked assets and refresh
my $schema = $t->app->schema;
$schema->resultset('Assets')->scan_for_untracked_assets();
Expand All @@ -61,6 +65,7 @@ sub prepare_asset_status {

# ignore exact size of untracked assets since it depends on presence of other files (see %ignored_assets)
my $groups = $asset_status->{groups};
is_deeply([sort keys %$groups], [0, 1001, 1002], 'groups present');
ok(delete $groups->{0}->{size}, 'size of untracked assets');
ok(delete $groups->{0}->{picked}, 'untracked assets picked');

Expand Down Expand Up @@ -238,10 +243,70 @@ my %expected_assets_without_max_job = (
},
);

my $empty_asset_id;
subtest 'handling assets with invalid name' => sub {
my $asset_count = $schema->resultset('Assets')->count;

# check whether registering an asset with empty name is prevented
is($schema->resultset('Assets')->register(repo => ''), undef, 'registering an empty asset prevented');

# handling within OpenQA::WebAPI::Controller::API::V1::Iso::schedule_iso
my $iso_api_controller_mock = new Test::MockModule('OpenQA::WebAPI::Controller::API::V1::Iso');
$iso_api_controller_mock->mock(_generate_jobs => sub { return undef; });
$iso_api_controller_mock->mock(emit_event => sub { return undef; });
my $iso_api_controller = OpenQA::WebAPI::Controller::API::V1::Iso->new;
$iso_api_controller->app($t->app);
is_deeply(
$iso_api_controller->schedule_iso({REPO_0 => ''}),
{error => 'Asset type and name must not be empty.'},
'schedule_iso prevents adding assets with empty name',
);
is_deeply(
$iso_api_controller->schedule_iso({REPO_0 => 'invalid'}),
{
successful_job_ids => [],
failed_job_info => [],
},
'schedule_iso allows non-existant assets though',
);

# handling within OpenQA::Schema::Result::Jobs::register_assets_from_settings
my $job = $schema->resultset('Jobs')->first;
my $job_settings = $job->{_settings} = {REPO_0 => ''};
stdout_like(
sub {
$job->register_assets_from_settings();
},
qr/not registering asset with empty name or type/,
'warning on attempt to register asset with empty name/type from settings',
);
$job_settings->{REPO_0} = 'in/valid';
stdout_like(
sub {
$job->register_assets_from_settings();
},
qr/not registering asset in\/valid containing \//,
'warning on attempt to register asset with invalid name from settings',
);
is($schema->resultset('Assets')->count, $asset_count, 'no further assets registered');

# add an asset with empty name nevertheless to test that it is ignored (in subsequent subtest)
my $empty_asset = $schema->resultset('Assets')->create({type => 'repo', name => ''});
ok($empty_asset, 'asset with empty name registered (to test ignoring it)');
$empty_asset_id = $empty_asset->id;
};

subtest 'asset status with pending state, max_job and max_job by group' => sub {
my $asset_status = $schema->resultset('Assets')->status(
compute_pending_state_and_max_job => 1,
compute_max_job_by_group => 1,
my $asset_status;
stdout_like(
sub {
$asset_status = $schema->resultset('Assets')->status(
compute_pending_state_and_max_job => 1,
compute_max_job_by_group => 1,
);
},
qr/Skipping asset $empty_asset_id because its name is empty/,
'warning about skipped asset',
);
my ($assets_with_max_job, $assets_without_max_job) = prepare_asset_status($asset_status);
is_deeply($asset_status->{groups}, \%expected_groups, 'groups');
Expand Down
3 changes: 3 additions & 0 deletions t/api/02-assets.t
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,9 @@ la;
# try to register with invalid type
$ret = $t->post_ok('/api/v1/assets', form => {type => 'foo', name => $iso1})->status_is(400);

# try to register with invalid name
$ret = $t->post_ok('/api/v1/assets', form => {type => 'iso', name => ''})->status_is(400);

# try to register non existing asset
$ret = $t->post_ok('/api/v1/assets', form => {type => 'iso', name => 'foo.iso'})->status_is(400);

Expand Down