Skip to content
This repository was archived by the owner on Aug 1, 2024. It is now read-only.

Loading data by using CSV files

Jump to bottom
MichaelJWhitney edited this page Jun 27, 2022 · 8 revisions

Developers load the supply chain data so that IBM Intelligence Suite can get up and running and display the relevant data on each dashboard. It is recommended that you use a flat file simple canonical CSV format for your data loads, so that your data is easy to build, consume, and initially load. CSV files are the recommended method because they can used to efficiently set up demo data, daily data integration, and automated data loads for production data.

About this task

During this task, you will put together your ingest data as CSV files and upload it to IBM Cloud Object Storage. Then, the Intelligence Suite will read, process, and transform your data for use in control tower dashboards, work queues, and other supply chain applications.

This process involves the following high-level flow:

  1. Review the CSV data model and create your CSV files. Samples are provided for you to ensure that your data is well-formed.
  2. Provision your IBM Cloud Object Storage bucket. Your account information is used when creating your data import rule.
  3. Create and upload your IBM Cloud Object Storage credentials. Your credentials are used for communication between IBM Cloud Object Storage and IBM Intelligence Suite.
  4. Create your data import rule for your CSV files. The data import rule registers your IBM Cloud Object Storage details so that the Intelligence Suite can monitor your storage bucket for changes. After the data import rule is created, you do not have to repeat this step unless your IBM Cloud Object Storage details change.
  5. Upload your CSV files to IBM Cloud Object Storage. Intelligence Suite monitors your storage bucket and processes the changes when you upload new data. This process transforms your CSV files into data that the system understands. That is, by creating well-formed upsert events that are structured with the relevant object types, events, your data, and any other expected values.
  6. Verify your changes in the Intelligence Suite. After a brief period, go to Data explorer and browse through your data to find the most relevant data objects that match your upload.

Review the data model and create your CSV files

  1. Review the CSV data model. If you are loading data for the first time, the initial data that you load are typically the records that contain a high data ingest priority in the data model, note that priority values 1 and 2 are required. CSV does not currently support priority value 3 attributes. See the sample data csv file below for tested examples.
  2. Put together your ingest data. In this task, CSV files are used, which are then transformed into data that the system understands by using automated processes.

Download the sample CSV files for reference to ensure that your data is well-formed:

Table 1. Sample CSV files

Sample CSV files Description
sample_data.zip An archive of the sample CSV files for the following object types:
  * Catalog
  * DemandPlan
  * Inventory
  * InventoryLot
  * Location
  * LocationGroup
  * Order
  * OrderAllocation
  * OrderLine
  * Organization
  * Product
  * Shipment
  * ShipmentLine
  * SupplyPlan

Note: The object types are named without using spaces or plural forms, for reference, so that they can be correctly identified by the objectType tag in IBM Cloud Object Storage. This tag is set when uploading the CSV files in a later step in this task.

For example, if you are loading inventory data, open the sample inventory CSV file and ensure that your inventory data conforms to the exact specifications of the sample file.

Important: Ensure that each column name in your CSV files exactly matches the fieldName defined in the data model for each data type. Use the sample CSV files as a reference, as they are formatted correctly.

  1. Save your CSV files. They will be uploaded to IBM Cloud Object Storage at a later step in this task. Ensure that the CSV file names do not contain any of the following restricted characters: /":'?<>&*|.

Provision your IBM Cloud Object Storage bucket

Your account information is used when creating your data import rule.

  1. Log in to your IBM Cloud account.
  2. Expand the left menu and click Resource list.
  3. If you do not yet have an object storage service, click Create resource. Then, from the services list, click Object Storage and Create and follow the steps to create your storage service. Once created, go back to the resource list.
  4. From the resource list, expand Storage and click your IBM Cloud Object Storage name.
  5. If you do not yet have a storage bucket, click Create bucket and follow the steps to create your storage bucket. These values will be used in the next step in this task.
  6. Within your bucket click Configuration and review your bucket details information. Make note of the private endpoint and location. These values will be used in the next step in this task.

For more information about IBM Cloud Object Storage general usage, see Getting started with IBM Cloud Object Storage.

Create and upload your IBM Cloud Object Storage credentials

Your credentials are used for communication between IBM Cloud Object Storage and IBM Intelligence Suite.

Create your IBM Cloud Object Storage credentials:

  1. Log in to your IBM Cloud account.
  2. Expand the left menu and click Resource list.
  3. From the resource list, expand Storage and click your IBM Cloud Object Storage name.
  4. From the Buckets view, click your bucket name.
  5. Click Service credentials. Then, click New credential.
  6. Name your credential and assign it the Writer role. Then, click Add.
  7. Expand the newly-created credential and make note of the apikey and resource_instance_id values. These values will be used in the next step in this task.

Upload your IBM Cloud Object Storage credentials:

  1. Using an HTTP tool such as Postman, make the following API call:

Table 2. API call

Parameter Value
URL https://api.ibm.com/infohub/bluerun/metadata/api/v1/na/tenants/{TenantId}/cos/{SecretName}
Method PUT
Content-Type application/json
Body { "value":"{{APIkey}}"}

Create your data import rule for your CSV files

The data import rule registers your IBM Cloud Object Storage details so that the Intelligence Suite can monitor your storage bucket for changes. After the data import rule is created, you do not have to repeat this step unless your IBM Cloud Object Storage details change.

  1. Using an HTTP tool such as Postman, make the following API call:

Table 3. API call

Parameter Value
URL https://api.ibm.com/infohub/bluerun/metadata/api/v1/na/data/import/rules
Method POST
Content-Type application/json
Body see below 
{
  "name": "My first COS CSV bulk import rule",
  "tenantId": "{{YOUR TENANT ID HERE}}",
  "instructions": {
    "transform": {
      "schema": "USE_DEFAULT",
      "deleteTransformFileAfterRoutedToDestination": "true",
      "transformedFilePrefixName": "csvTransform",
      "sourceFormat": "csv"
    },
    "routeTo": {
      "streamingIngest": {
        "priority": "1"
      },
      "destinationType": "streamingIngest"
    },
    "source": {
      "sourceType": "COS",
      "COS": {
        "bucketName": "{{YOUR BUCKET NAME HERE}}",
        "prefixName": "{{YOUR PREFIX NAME HERE}}",
        "endpoint": "{{YOUR COS PRIVATE END POINT HERE}}",
        "instanceId": "{{YOUR COS INSTANCE ID HERE!}}",
        "bucketRegion": "{{YOUR COS BUCKET LOCATION HERE}}",
        "secretName": "{{YOUR SECRET NAME HERE}}",
        "isPublicUrl": "false"
      }
    }
  }
}

Ensure that you enter all the details required in the body's comments, such as the name of the storage bucket and your IBM Cloud Object Storage credentials, as noted in the previous step.

Important: When entering the IBM Cloud Object Storage prefixName, ensure that the value ends in a / to avoid path errors.

Upload your CSV files to IBM Cloud Object Storage

The Intelligence Suite monitors your storage bucket and processes the changes when you upload new data. This process transforms your CSV files into data that the system understands. That is, by creating well-formed upsert events that are structured with the relevant business objects, events, your data, and any other expected values.

  1. Log in to your IBM Cloud account.
  2. Expand the left menu and click Resource list.
  3. From the resource list, expand Storage and click your IBM Cloud Object Storage name.
  4. From the Buckets view, click your bucket name.
  5. Click Upload.
  6. Under Prefix for objects (optional), enter the exact same value that you entered as the prefixName that you used when creating your data import rule, including ending it with a /.
  7. Select your CSV files for upload by using the Drag and drop files (objects) or click to upload form. After you select your CSV files, a file listing is displayed with the file name and size.
  8. Important: Expand Advanced settings (optional) > Tags and click Add.
  9. Add a tag that corresponds with the object type that you are uploading.

Table 4. Tags

Key Value
objectType ObjectTypeName

Where the ObjectTypeName is the name of the high-level object in the data model, without using spaces or plural forms. For example, Inventory or DemandPlan.

  1. Click Upload. The file will be uploaded to IBM Cloud Object Storage and appear under the Buckets > Objects view, so that you can confirm that the upload was successful. Then, the CSV files will be processed, transformed, and reflected in the Intelligence Suite.

Verify your changes in the Intelligence Suite

After a brief period, verify that your data has made its way into the system. Go to Data explorer and browse through your data to find the most relevant data objects that match your upload.

Related Topics

  • Loading data by using JSON files

    As an alternative to using CSV files, you can load your data by using JSON files.

  • Exploring the data model

    The data model demonstrates the relationships between the data objects used by the Intelligence Suite. The data that you load is used to populate the Control Tower dashboards and details pages.

  • Data Import 101 Tutorial

    Learn to import bulk data by creating import rules and by uploading object files to IBM Cloud Object Storage.

  • Data Import 102 Tutorial

    Learn to import bulk CSV data by creating import rules with transform and by uploading CSV object files to IBM Cloud Object Storage.

Parent topic:

Solution guide home

Clone this wiki locally