Threadneedle updated: support for REST operation in global SOAP connector

• an operation can now set type: 'REST' for a connector that has global SOAP configuration
• none of the global configurations are inherited for these operations

[Breaking Change] mout.js removal

Due to vulnerabilities with mout.js, the dependency has been removed (and internal code refactored to use code copied over from mout as short-term solution).

However, since falafel exposed mout on the global level, the removal of this dependency is a breaking change for any connectors that was using mout. For connectors upgrading to this version of falafel or later, there are two ways forward:

1. include mout as a dependency in the connector's package.json, assuming mout have patched the vulnerability (fastest fix, but persists dependency on mout)
2. replace all instances of mout with native, custom, or lodash implementations (recommended, but beware lodash methods don't always the same implementations as mout; best to check beforehand the underlying code between the two libraries, instead of assuming parity)

File-Handling: Mime Type Resolving

File Handling

Default Mime Type

For all file-handling methods, defaults mime type to application/octet-stream if content-type isn't provided and cannot be determined from the file name (extension)

Download methods

The download methods in file-handling try to resolve mime type based on content-type in headers if possible, else uses the default specified above

Update threadneedle

• Updated threadneedle to latest version which uses @trayio/needle
• Changed needle dependency to use @trayio/needle
• Resolved NPM audit issues

File uploads update - bucket ACL

• All file uploads are now uploaded with ACL (Access Control List) of 'bucket-owner-full-control'
• An optional flag disableObjectAcl is also introduced to disable this ACL setting, which is part of the file upload params required by falafel.files.streamUpload, falafel.files.streamMPUpload, and falafel.files.upload.

v1.28.0

• The response validation for Raw HTTP has the body size limit increased to ~5.9MB (6080KB) from 960KB.
• Threadneedle version bumped

File Handling updates, Raw HTTP Request updates, and DDL visibility

File Handling updates

• Expiry time in signed URL is now 6 hours, as per PreSignedURL
• Added CONNECTOR_FILE_REGION and CONNECTOR_FILE_BUCKET environment variables support for defining target region and bucket (and also CONNECTOR_FILE_DEV_BUCKET for development)
• Added check for expired file/URL for download methods based on AWS signing, changing error message if detected
• Disabled decoding response in needle by default, but also allowed passing in needle options

Raw HTTP Request updates

• Operation title is now sentence cased
• Schema has been updated where some properties should have been required, such as body
• Configuration option has been added for restricting URL oneOf options
• The response format has been fixed to match the format defined by the schema (breaking change)
  • status_code is now also returned in the output (bug fix)

schema.js support and DDL visibility

• A type property is now included in the connectorsJson for every operation, with values either being public or ddil
• All DDL operations are now detected if their (programmatic) name ends with _ddl.
  • If a schema is not specified, and generic schema is set (using global schema).

The rules for operation types and being added to the connectors.json op list is as follows:

• If no schema is present
  • If operation's name ends with DDL, then it is added to connectors.json as a DDL type operation
  • Else operation is NOT added to connectors.json
• If a schema is present
  • If type is NOT provided, then assume public by default (unless the operation name ends with _ddl)
  • If type is provided
    • If public, add to connectors.json
    • If ddl, add to connectors.json
    • If private, NOT added to connectors.json

Other

• Threadneedle version updated (v1.12.0)
• The dev server can now be disabled via FALAFEL_DISABLE_DEV_SERVER environment variable when in dev mode
• The no trigger code for request.js has been updated to #no_trigger for BE
• Artisan error handling has been updated to improve connector debugging

v1.26.0 release

• File Handling

  • Now reads CONNECTOR_MAX_ALLOCATED_RAM_MB from environment.
  • Optimisation logic has been updated to:
    a) Handle undefined file lengths
    b) Use up to 3/4 (from half) of allocated RAM for AWS SDK's partSize and queueSize

• JSON Schema: titles now capitalise the following words:

  • ID
  • IDs
  • URL

v1.25.1 - Raw HTTP Request fixes and updates

Main updates

• body is now validated and processed only when necessary.
• options processing has been added, which includes setting Content-Type if not defined by evaluating `body.
  • This includes setting json and multipart flags.
• rawHttpRequest schema has been updated.
• More rawHttpRequest utils have been added.

Other

• JSON schema generator now allows additionalProperties: true if specified and object is one of the types when type is an array.

v1.25.0 - Raw HTTP Request

Raw HTTP Request operation

Raw HTTP Request operation functionality has been added, with default configuration and pre-defined schema. See docs.

Utils

Falafel now exposes utility functions via falafel.utils - see docs.

Threadneedle and smartSubstitution

• Threadneedle v1.11.0 is utilised. See release notes.
• Threadneedle's smartSubstitution can be accessed via falafel.utils.threadneedle.smartSubstitution