BiqQuery API methods return null

[asuresh4]

The following methods of the API ALWAYS return null value. Please have a look. Thanks!
DatasetInfo.getLocation()
DatasetInfo.getCreationTime()
DatasetInfo.getDefaultTableLifetime()
DatasetInfo.getDescription()
DatasetInfo.getLastModified()

This is also related to an initial issue I created: #2482

I created a sample dataset through the UI and added a table to it. I'm able query this relevant information using the gcloud API. On using the Java Library to query information about the dataset using the above methods, I get a null value. However dataset.getDatasetId().getDataset() gives the name of the expected dataset.

Code:

BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();
Page<Dataset> datasets = bigquery.listDatasets();

for (Dataset dataset : datasets.iterateAll()) {
      Map<String, String> props = new HashMap<>();
      String datasetId = dataset.getDatasetId().getDataset();

      dataset.getCreationTime() // This gives null

[asuresh4]

Versions of the API I tried:

Google Cloud 0.21.1-alpha
Google Cloud 0.24.0-alpha
Google Cloud 0.25.0-alpha

[neozwu]

Hi, @asuresh4 I think you are looking for those attributes of a dataset, did you try dataset.getCreationTime(); dataset.getLocation(); dataset.getDefaultTableExpirationMs(); dataset.getDescription(); dataset.getLastModifiedtime();

[asuresh4]

@neozwu Thanks for reaching out. Here's more details about what I've tried.

Full code:

import com.google.api.gax.paging.Page;
import com.google.cloud.MonitoredResource;
import com.google.cloud.bigquery.Dataset;
import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryOptions;
import com.google.common.collect.ImmutableMap;

import java.util.HashMap;
import java.util.Map;

public class BigQueryFetcher implements PropertyFetcher {
    protected MonitoredResource makeResource(String datasetId, String project) {
        return MonitoredResource.of(
            "bigquery_dataset",
            ImmutableMap.of("dataset_id", datasetId, "project_id", project));
    }

    public Map<String, Map<String, String>> getPropertiesByID(String project) {
        // TODO: Check if this is cached automatically by the client lib and do so if not
        BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();

        Map<String, Map<String, String>> propsByID = new HashMap<>();
        Page<Dataset> datasets = bigquery.listDatasets();

        for (Dataset dataset : datasets.iterateAll()) {
            Map<String, String> props = new HashMap<>();
            String datasetId = dataset.getDatasetId().getDataset();
            
//            props.put("location", dataset.getLocation());
//            props.put("created_at", Long.toString(dataset.getCreationTime()));
//            props.put("default_table_ttl", Long.toString(dataset.getDefaultTableLifetime()));
//            props.put("description", dataset.getDescription());
            
            System.out.println(dataset.getDatasetId().getDataset());
            System.out.println(dataset.getLastModified());
            System.out.println(dataset.getLocation());
            System.out.println(dataset.getCreationTime());
            System.out.println(dataset.getDefaultTableLifetime());
            System.out.println(dataset.getDescription());
            
            if (dataset.getLabels() == null) {
                continue;
            }
            // Add all labels
            for (String label : dataset.getLabels().keySet()) {
                props.put(label, dataset.getLabels().get(label));
            }
            
            System.out.format("These are the labels derived from: %s\n", datasetId);
            System.out.println(props);
            
            propsByID.put(UniqueIDCreator.uniqueIdForResource(makeResource(datasetId, project)), props);
        }
        return propsByID;
    }
}

On invoking the above method, the output I get:

test // The name of my dataset
null
null
null
null
null

I did not try: dataset.getDefaultTableExpirationMs() and dataset.getLastModifiedtime(). But I also don't find those methods here: https://googlecloudplatform.github.io/google-cloud-java/0.25.0/apidocs/index.html

[anthmgoogle]

Let's keep iterating to answer the question.

[pongad]

@asuresh4 This is because you are using the listDatasets method. The method can only populate some fields (see doc) since many fields are not populated by the BigQuery API.

If you need the fields, please use the getDataset method separately.