ListJobsAsync() returns only us-central1 jobs

[luarvic]

Environment details

• OS: Windows 10 Enterprise
• .NET version: .NET Core 3.1
• Package name and version: Google.Cloud.Dataflow.V1Beta3 v1.0.0-beta02

Steps to reproduce

The below method should return active Dataflow jobs located in a region specified in location parameter, but actually it returns jobs from us-central1 region regardless of the parameter value.

using Google.Cloud.Dataflow.V1Beta3;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using System;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;

namespace Abracadabra.Api.Controllers
{
    internal class GcpController : Controller
    {
        public async Task<IActionResult> GetDataflowJobsAsync(string project, string location, CancellationToken cancellationToken)
        {
            try
            {
                var client = await JobsV1Beta3Client.CreateAsync(cancellationToken);
                var request = new ListJobsRequest
                {
                    ProjectId = project,
                    Filter = ListJobsRequest.Types.Filter.Active,
                    Location = location
                };
                var jobs = client.ListJobsAsync(request);
                var result = await jobs.ToListAsync(cancellationToken);

                return Ok(result);
            }
            catch (Exception e)
            {
                return StatusCode(StatusCodes.Status500InternalServerError, e);
            }
        }
    }
}

[amanda-tarafa]

I'm looking into now. I'll report back here when I know more.

[amanda-tarafa]

According to the API reference docs (scroll down to ListJobs()):

To list the jobs of a project in a region, we recommend using projects.locations.jobs.list with a regional endpoint. To list the all jobs across all regions, use projects.jobs.aggregated. Using projects.jobs.list is not recommended, as you can only get the list of jobs that are running in us-central1.

So the result you are seeing by using JobsV1Beta3Client.ListJobsAsync is expected. You may use JobsV1Beta3Client.AggregatedListJobsAsync to obtain jobs in all locations. As for the other part of the comment, I will ask internally, as I think there's no projects.locations.jobs.list functionality exposed at the moment by the gRPC Dataflow API. I'll get back here as I know more.

(Notice that the library is simply exposing, in and idiomatic manner, the functionality that the API itself exposes, so, if the API is not currently exposing a way to list Jobs per Location, then there's not much we can do on the library side. Still, I'll get clarity on this behaviour, I'll admit the API documentation is confusing).

[luarvic]

Hello @amanda-tarafa

I have tried JobsV1Beta3Client.AggregatedListJobsAsync and it seems to be unimplemented. It throws the following:

Grpc.Core.RpcException: Status(StatusCode="Unimplemented", Detail="AggregatedListJobs method not supported by Router.", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1629903159.218000000","description":"Error received from peer ipv4:142.250.181.202:443","file":"..\..\..\src\core\lib\surface\call.cc","file_line":1067,"grpc_message":"AggregatedListJobs method not supported by Router.","grpc_status":12}")
   at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Google.Api.Gax.Grpc.ResponseAsyncEnumerable`3.ResponseAsyncEnumerator.MoveNextAsync()
   at Google.Api.Gax.Grpc.ResourceEnumerator`2.MoveNextAsync()
   at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|620_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
   at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|620_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36

[amanda-tarafa]

Thanks for that information. I'll include this on my queries. I'll report back here as soon as I know more.

Although we usually recommented the gRPC libraries (Google.Cloud.Dataflow.V1Beta3) over the REST ones (Google.Apis.Dataflow.v1b3), you can try Google.Apis.Dataflow.v1b3 which should work just fine. Here's the List method that will allow you to list jobs per location.

I'll get back here when I know more about Google.Cloud.Dataflow.V1Beta3. Sorry for this.

[luarvic]

Ah, sorry, this error is caused by the line

var result = await jobs.ToListAsync(cancellationToken);

I'll check further and update you.

[amanda-tarafa]

Yes, it is the service failing nonetheless. When you execute var jobs = client.ListJobsAsync(request); (or the Aggregated one) this just creates a sequence, but makes no server calls. It's when you start iterating the sequence, in your case with var result = await jobs.ToListAsync(cancellationToken); that the server calls are performed as needed.

[amanda-tarafa]

Just to say that I'm still looking at this internally with the API team and as soon as I know more, I'll update here.

[amanda-tarafa]

Marking as blocked as we are in talks with the API team. For now, the best answer we have is for you to use Google.Apis.Dataflow.v1b3. I'll update here as soon as I have more information.

[meredithslota]

We've identified similar issues in other repos and have escalated this to the API service team since it's not a language-specific library issue. Here are the related issues:

• ListJobsRequest is not using the 'location' filter python-dataflow-client#31: ListJobsRequest is not using the 'location' filter
• Support for locations in the launch_template method python-dataflow-client#25: Support for locations in the launch_template method
• The location option of launchFlexTemplate is not working nodejs-dataflow#19: The location option of launchFlexTemplate is not working
• https://github.com/googleapis/java-dataflow/issues/64: Dataflow: The location field of LaunchFlexTemplateRequest building is not working
• ListJobsAsync() returns only us-central1 jobs #7038: ListJobsAsync() returns only us-central1 jobs (context from issue is that the location parameter isn't working)

Marking this as a P1 bug w/ external label.

[bbhoss]

I couldn't find an existing issue in the Google Issue Tracker so I created one here. I suggest starring this issue to anyone else that runs across this problem, hopefully it will get some attention from Google soon.

[amanda-tarafa]

@meredithslota I'm assigning to you as I know you are talking to the API team. If there's anything we can do library side, feel free to reassing back to men.

[meredithslota]

The PR to address this (at least partially) has been merged: #7913 — can folks in this thread let me know if this resolves your issue?

[jskeet]

@meredithslota: Unfortunately it doesn't. I'll ping you internally with details.

[jskeet]

@Rosti81: This should now be fixed in a combination of version 1.0.0-beta03 of the library, and some internal changes. Note that the Aggreate RPCs still do not work and are being looked into separately - but the other RPCs should now work appropriately. Please let me know if you have run into any more problems.