Skip to content

Releases: cognitedata/toolkit

v0.2.0a5

28 May 09:37
@doctrino doctrino
v0.2.0a5
eacdffd
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.2.0a5 Pre-release
Pre-release

cdf-tk

Added

  • If a resource is referring to another resource, the cdf-tk build will now give a warning if the referred resource
    is not found in the same build. For example, if you have a data model and is missing the space, the build command
    will give a warning that the space required by the data model is missing.
  • The cdf-tk build command will now give warnings on duplicated resource. For example, if you have two files with
    the same externalId in the same module, the build command will give a warning that the externalId is duplicated,
    and that only the first file is used.
  • Support for securityCategories in the auth folder.
  • Added support for resource type DatapointSubscription in the timeseries folder.

Fixed

  • In a function config, if you did not set fileId you would get an error when running cdf-tk deploy,
    Missing required field: 'fileId'.. The fileId is generated automatically when the function is created,
    so it is not necessary to set it in the config file. This is now fixed.
  • If you do cdf-tk init --upgrade, on a pre 0.2.0a3 version, you are met with
    ERROR (ToolkitModuleVersionError): Failed to load previous version, .... This is now fixed.
  • The parameter container.properties.<property>.type.list was required to be set, even thought it is optional
    in the CDF API. This is now fixed.
  • The ExtractionPipelineConfig create, update and delete report numbers were incorrect. This is now fixed.

Improved

  • Gives a more informative error message when the authentication segment of a transformation resource file is
    missing a required field.
  • Transformation queries can be inline, i.e. set in either the Transformation query property in the yaml or
    as a separate file. If set in both, an error is raised because it is ambiguous which query to use.
  • In the cdf-tk pull command, if an error occurs, the temporary directory was not removed. This is now fixed.
  • Improved error message when running cdf-tk deploy/clean before running cdf-tk build.

Changed

  • [BREAKING] In function YAML config externalDataSetId is renamed to dataSetExternalId to be consistent with
    the naming convention used in the rest of the toolkit.

Templates

Added

  • Function used to schedule & trigger workflow
  • Extended cognite_modules/examples/my_example_module with a SecurityCategory case.
  • Extended cognite_modules/examples/my_example_module with a DatapointSubscription case.

Removed

  • The parameter fileId is removed from all function configurations
    (cdf_functions_dummy, cdf_data_pipeline_files_valhall, cdf_data_pipeline_timeseries_valhall,
    and my_example_module) as it is no longer required.
  • In all modules with an extraction_pipelines resource, removed dataSetExternalId and name from all
    ExtractionPipelineConfigs as this is not used and thus only causes confusion.
  • In all modules with a function, renamed externalIdDataSet to dataSetExternalId to be consistent with the
    naming convention used in the Cognite API.
  • In module my_example_module, removed interval and isPaused from the Transformation as these are not used.
    These parameters should only be present in a TransformationSchedule.

Changed

  • Removed schedule from annotation function context:files:oid:fileshare:annotation
  • Add use of map_concatin transformation files_oid_fileshare_file_metadata to keep existing files metadata.
  • Switched to using file.uploaded_time instead of file.last_updated_time since update time
    potentially is updated every time the transformation runs, and don't require a reannotation.

What's Changed

Full Changelog: v0.2.0a4...v0.2.0a5

Contributors

ronpal, haakonvt, and 2 other contributors
Assets 2

v0.1.4

30 Apr 14:52
@doctrino doctrino
v0.1.4
63fcd4a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.1.4 Latest
Latest

cdf-tk

Fixed

  • When running cdf-tk deploy with container YAML config from before cognite-sdk<7.37, the user would get an error
    Missing required field: 'list'. for the direct container properties. This is now fixed.
  • When running cdf-tk init --upgrade, the user would get an error IndexError: list index out of range.
    This is now fixed.

Template

No changes to templates.

What's Changed

Full Changelog: v0.1.3...v0.1.4

Contributors

doctrino
Assets 2

# v0.2.0a4

29 Apr 12:23
@doctrino doctrino
v0.2.0a4
f7b2dfe
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
# v0.2.0a4 Pre-release
Pre-release

cdf-tk

Removed

  • [BREAKING] cognite-tookit no longer supports common_function_code. The code used by functions must be in each
    function directory. The reason for this is that cognite-toolkit is a tool for governance and deployment of
    modules, it is not for development of functions. The common_function_code was a feature to support easier
    development of functions. It is expected that functions are developed in a separate environment and then
    moved to the cognite_modules folder for deployment and promotion between environments.

Changed

  • In config.[env].yaml, in the environment section, name is no longer required. Instead, the [env] part
    of the config.[env].yaml file is used as the name of the environment. This is to avoid redundancy.

Improved

  • When running cdf-tk clean --dry-run the output would show local resources regardless of whether they existed
    in CDF or not. This is now fixed and only resources that exist in CDF are shown in the output.
  • Better error message (no exception raised) if the config file has selected_modules_and_packages, but with no list items.
  • If yaml files are invalid, a link to the API docs for the resource is shown in the error message.

Fixed

  • When deploying a FunctionSchedule that requires an update, the cdf-tk would fail with error
    Failed to update functions.schedules. Error 'FunctionSchedulesAPI' object has no attribute 'update'..
    This is now fixed.
  • When calling cdf-tk init --upgrade, the user is met with a Failed to load previous version, ....
    This is now fixed.
  • When running cdf-tk auth verify --interactive and the user want to create a new group with the necessary
    capabilities, the cdf-tk would successfully create a group, but then raise
    an Error: cognite.client.exceptions.CogniteAPIError: Insufficient access rights. when trying to validate.
    This is now fixed.

Templates

Added

  • Added Workflow with a Function and a Transformation to the cdf_data_pipeline_files_valhall example

Changed

  • The cdf_functions_dummy module now includes codes from the former common_function_code directory.

Fixed

  • In example_pump_data_model, in the Pump.view.yaml the property source used external_id instead of
    externalId.

Details

Full Changelog: v0.1.3...v0.2.0a4

Contributors

ronpal, haakonvt, and 2 other contributors
Assets 2

v0.1.3

29 Apr 07:14
@doctrino doctrino
v0.1.3
1da5f87
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.1.3

cdf-tk

Fixed

  • Functions that are deployed with schedules no longer uses a short-lived session (before: failed after ~an hour).
  • When deploying a FunctionSchedule that requires an update, the cdf-tk would fail with error
    Failed to update functions.schedules. Error 'FunctionSchedulesAPI' object has no attribute 'update'..
    This is now fixed.
  • The dependency cognite-sdk must now be >=7.37.0 to use the cdf-tk.

Templates

Fixed

  • Updated module apm_simple_data_model for cognite-sdk>=7.37, i.e., container properties
    of type direct have now list: false explicitly set.
  • In example_pump_data_model, in the Pump.view.yaml the property source used external_id instead of externalId.

Full Changelog: v0.1.2...v0.1.3

Assets 2

v0.2.0a3

23 Apr 14:36
@doctrino doctrino
v0.2.0a3
b3d5af8
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.2.0a3 Pre-release
Pre-release

cdf-tk

Added

  • Support for the Workflow and WorkflowVersion resource type
  • Support for specifying selected_modules_and_packages as paths and parent paths. For example, you can
    now write cognite_modules/core/cdf_apm_base instead of just cdf_apm_base. This is to support
    modules that have the same name but are in different parent directories. In addition, this also better reflects
    the structure of the cognite_modules and custom_modules folder better.

Fixed

  • Functions that are deployed with schedules no longer uses a short-lived session (before: failed after ~an hour).

Changed

  • [BREAKING] The cdf-tk build will now clean the build directory by default before building the modules to avoid
    unwanted side effects from previous builds. To stop this behavior, use the --no-clean flag.
  • [BREAKING] The _system.yaml is now required to be on the root level when running any cdf-tk command. This
    means that the _system.yaml file must be in the same directory as the config.[env].yaml files. This is to
    support running cdf-tk without the cognite_modules folder.
  • [BREAKING] In the config.[env].yaml files, the modules section is now renamed to variables. This is to
    better reflect the content of this section, which is variables that can be used in the resource files.
  • In addition to cognite_modules and custom_modules, the cognite-toolkit now also support modules as the
    top-level folder for modules. This together with the two changes above, is to have a better support for running
    the cognite-toolkit as a standalone CLI without the cognite_modules.
  • The .gitignore file you get by running cdf-tk init now ignores the /build by default.
  • The dependency cognite-sdk must now be >=7.37.0 to use the cdf-tk.

Templates

Fixed

  • Align tag name in asset hierarchy between the 2 example transformations
  • Added default root asset ID to documents for initial annotation
  • Aligned use of asset external ID across contextualization functions
  • Annotation logic with local time stamp for when to reprocess P&ID files
  • Input to P&ID annotation based on list of synonyms for tag
  • Updated module apm_simple_data_model for cognite-sdk>=7.37, i.e., container properties
    of type direct have now list: false explicitly set.

Added

  • Added Transformation for WorkOrder and WorkItems to OID testdata template
  • Added Workflow with a Function and a Transformation to the custom module example

Details

Full Changelog: v0.1.2...v0.2.0a3

Contributors

ronpal, haakonvt, and 2 other contributors
Assets 2

v0.2.0a2

03 Apr 13:57
@doctrino doctrino
v0.2.0a2
7b5d1b1
Compare
Choose a tag to compare
v0.2.0a2 Pre-release
Pre-release

cdf-tk

Added

  • Variables can now have extra spaces between curly braces and the variable name. For example, {{ my_variable }} is now
    a valid variable. Before this change, you would have to write {{my_variable}}.
  • If an environment variable is not found in a resource file, for example, ${CDF_CLUSTER}, when
    running cdf-tk deploy the user will now get a warning message that the variable is missing. Before this change,
    this would pass silently and potentially cause an error when trying to deploy to CDF that was hard to debug.

Fixed

  • When running cdf-tk with a Token for initialization, the cdf-tk would raise an IndexError. This is now fixed.
  • Container resources that did not have set the optional property usedFor would always be classified as changed,
    when, for example, running cdf-tk deploy --dry-run. This is now fixed.

Changed

  • If two modules have the same name, the cdf-tk build command will now stop and raise an error. Before this change,
    the cdf-tk build command would continue and overwrite the first module with the second module.

Templates

No changes to templates.

Details

Full Changelog: v0.1.2...v0.2.0a2

Contributors

doctrino
Assets 2

v0.2.0a1

20 Mar 17:49
@doctrino doctrino
v0.2.0a1
bee6471
Compare
Choose a tag to compare
v0.2.0a1 Pre-release
Pre-release

v0.2.0a1

cdf-tk

Added

  • Support for interactive login. The user can now set LOGIN_FLOW=interactive in the .env file
    to use interactive login.

Changed

  • The verification of access by the tool is now scoped to the resources that are being deployed instead of
    the entire project. This means that if the user only has access to a subset of the resources in the project,
    the tool will still be able to deploy those resources.

Templates

Added

  • Added functionality for wildcard detection of tags in P&ID
  • Added functionality for multiple overlapping annotation to same tag - related to wildcards

Details

Full Changelog: v0.1.2...v0.2.0a1

Contributors

ronpal, gregertw, and 2 other contributors
Assets 2

v0.1.2

18 Mar 13:50
@doctrino doctrino
v0.1.2
58a0205
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.1.2

cdf-tk

Fixed

  • Running the command cdf-tk auth verify --interactive without a .env would raise a
    AttributeError: 'CDFToolConfig' object has no attribute '_client' error. This is now fixed and instead the user
    gets a guided experience to set up the .env file.

Changed

  • cognite-toolkit have moved the upper bound on the cognite-sdk dependency from 7.27 to 8.0.
  • Creating/Removing spaces no longer requires DataModelingInstances capability.

Templates

No changes to templates.

Details

Full Changelog: v0.1.1...v0.1.2

Contributors

renovate, gregertw, and 2 other contributors
Assets 2

v0.1.1

01 Mar 12:48
@doctrino doctrino
v0.1.1
4ef407f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.1.1

cdf-tk

Fixed

  • When running cdf-tk clean or cdf-tk deploy --drop-data for a data model with more than 10 containers,
    the command would raise an APIError. This is now fixed.
  • A few minor potential AttributeError and KeyError bugs have been fixed.

Templates

No changes to templates.

Full Changelog: v0.1.0...v0.1.1

Assets 2

v0.1.0

29 Feb 12:28
@doctrino doctrino
v0.1.0
d0c0859
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.1.0

cdf-tk

Added

  • Command cdf-tk dump datamodel for dumping data models from CDF into a local folder. The use case for this is to
    dump an existing data model from CDF and use it as a basis for building a new custom module with that data model.
  • A Python package API for the cdf-tk. This allows for programmatic access to the cdf-tk functionality. This
    is limited to the build and deploy functionality. You can start by from cognite_toolkit import CogniteToolkit.

Fixed

  • In the function deployment, the hashing function used of the directory was independent of the location of the files
    within the function directory. This caused moving files not to trigger a redeployment of the function. This is now
    fixed.

Changed

  • Removed unused dependencies mypy, pyarrow and chardet from cognite-toolkit package.
  • Lowered the required version of pandas to 1.5.3 in the cognite-toolkit package.

Templates

Changed

  • In the infield section, the infield_apm_app_config.node.yaml was moved from cdf_infield_location to cdf_infield_common
    module. In addition, the module cdf_infield_second_location was added to the infield section. This is to demonstrate
    how multiple locations in Infield should be handled.
  • In the cdf_data_pipeline_files_valhall example, the Cognite Function fu_context_files_oid_fileshare_annotation
    has been renamed to fn_context_files_oid_fileshare_annotation. It has also been split into several files, to be
    easier to understand. It has also been changed to using prints (over logging), as that is unfortunately a hard
    requirement from the API.

Added

  • In the cdf_data_pipeline_files_valhall example, the README.md file has been updated with instructions on how to
    run and test Cognite Functions locally.

New Contributors

Full Changelog: v0.1.0b9...v0.1.0

Contributors

haakonvt
Assets 2