mimecast: make CEL the default input, deprecate HTTP JSON - #20356
Conversation
efd6
added
enhancement
Elastic Docs Style Checker (Vale)Summary: 1 suggestion found 💡 Suggestions (1): Optional style improvements. Apply when helpful.
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
🚀 Benchmarks reportTo see the full report comment with |
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
|
@efd6 do we have the ability to completely hide the http json/deprecated Mimecast API for any user who doesn't currently have that input/data stream enabled? If not, is that something the Fleet team would have to implement? @nimarezainia for context, as we rethink onboarding I'm trying to avoid cases like this where we show deprecated inputs/datastreams even for new users of the integration. Would be much cleaner if we can just hide/remove these rather than users being presented with all sorts of deprecated options. |
andrewkroh
added
the
documentation
|
@vera-review-bot review |
![vera-review-bot[bot]](https://avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
| # newer versions go on top | ||
| - version: "3.5.0" | ||
| changes: | ||
| - description: Make CEL the default input for all data streams. The HTTPJSON input is deprecated and will be removed in a future release. New installations use the Mimecast v2 API via CEL; existing HTTPJSON configurations continue to work but users should migrate to CEL by provisioning v2 API credentials. |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/mimecast/changelog.yml:4
The 3.5.0 changelog entry only describes the CEL/HTTPJSON default swap, but this PR also drops enabled: false from the cloud_integrated_logs and message_release_logs streams, so those two data streams now start collecting on a fresh install. Please mention that in the changelog.
Details
The diff removes enabled: false from the single cel stream in both packages/mimecast/data_stream/cloud_integrated_logs/manifest.yml and packages/mimecast/data_stream/message_release_logs/manifest.yml. A stream with no enabled key defaults to enabled, so both data streams are now on by default for new installs of the integration.
The PR's own proposed commit message calls this out ("Also enables the two CEL-only data streams (cloud_integrated_logs, message_release_logs) by default"), but the changelog entry does not. The changelog is the user-facing record of what a version change does, and enabling two data streams that previously collected nothing is a visible behaviour change: cloud_integrated_logs polls /siem/v1/batch/events/ci and siem_logs polls /siem/v1/batch/events/cg, so operators need to know both are now scheduled.
Note that cloud_integrated_logs/manifest.yml and message_release_logs/manifest.yml contain only deletions in this diff, so the changelog is the only in-diff place this can be recorded.
Recommendation:
Extend the 3.5.0 entry so the data stream enablement is recorded alongside the input change:
- version: "3.5.0"
changes:
- description: Make CEL the default input for all data streams. The HTTPJSON input is deprecated and will be removed in a future release. New installations use the Mimecast v2 API via CEL; existing HTTPJSON configurations continue to work but users should migrate to CEL by provisioning v2 API credentials.
type: enhancement
link: https://github.com/elastic/integrations/pull/20356
- description: Enable the cloud_integrated_logs and message_release_logs data streams by default. These are CEL-only data streams that were previously shipped disabled.
type: enhancement
link: https://github.com/elastic/integrations/pull/20356🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| ] | ||
| }, | ||
| "source": { | ||
| "ip": "54.236.186.184" |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/mimecast/data_stream/siem_logs/sample_event.json:73
The regenerated sample events replace previously sanitised placeholders (example.com, 8.8.8.8) with routable public IPs and personal-name mailboxes at non-reserved domains. Please re-anonymise them to RFC 5737 addresses and example.com.
Details
The anonymisation conventions for committed fixtures are RFC 5737 ranges (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24) for IPv4 and the reserved example.com/example.org/example.net domains for hosts and mailboxes. Several sample events regress against that in this diff, replacing values that were already compliant:
siem_logs/sample_event.json: addssource.ip/related.ip54.236.186.184(a routable, publicly allocated address), and replacesjohndoe@example.comando365_service_account@example.comwith personal-name mailboxes atdemo-int.elastic.mime-api.comandpacific-concept.b41.one.ttp_url_logs/sample_event.json(lines 90 and 101): replaces8.8.8.8with54.243.138.179andjohndoe@example.com/googlealerts-noreply@google.comwith personal-name mailboxes atthejunglegroup.b41.oneanddemo-int.elastic.mime-api.com.dlp_logs/sample_event.jsonandttp_ap_logs/sample_event.json: same substitution ofexample.commailboxes for personal-name addresses at non-reserved domains.
The same values are mirrored into packages/mimecast/docs/README.md, so they are rendered on the public integration documentation page. These files are regenerated from the mock service under _dev/deploy/docker/files/, so the durable fix is in the mock fixtures rather than by hand-editing the generated sample events.
Recommendation:
Sanitise the mock fixtures so regeneration produces compliant values, e.g. for the siem_logs sender IP:
{
"source": {
"ip": "203.0.113.18"
},
"related": {
"user": [
"chris.germany@example.com",
"dkinney@example.org"
]
}
}Apply the same mapping consistently (one original value -> one placeholder) across siem_logs, ttp_url_logs, dlp_logs and ttp_ap_logs, then re-run the system tests so sample_event.json and docs/README.md are regenerated rather than edited.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| ID`, `Access Key`, and `Secret Key`) should be provided by a Mimecast | ||
| representative for this integration. Under `Advanced options` you can set the | ||
| time interval between two API requests as well as the API URL. A Mimecast | ||
| Authorization parameters for the Mimecast API (`Client ID` and `Client Key`) should |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/mimecast/_dev/build/docs/README.md:14
The v2 section, now the primary configuration instructions, tells users to obtain a Client Key, but the field the UI actually presents is Client Secret. Rename it in the docs so it matches the manifest.
Details
packages/mimecast/manifest.yml declares the v2 credential pair as client_id (title Client ID for v2 API Endpoints) and client_secret (title Client Secret for v2 API Endpoints). The README text names them Client ID and Client Key, so the second field does not match anything a user sees in Fleet.
This wording predates the PR, but the PR promotes this paragraph from the secondary section to the primary onboarding instructions for the now-default input, so it is the first thing a new user reads. The same text is mirrored into the generated packages/mimecast/docs/README.md, which is also in this changeset.
Recommendation:
Match the manifest field titles:
### v2 API Endpoints
Authorization parameters for the Mimecast API (`Client ID` and `Client Secret`) should
be provided by a Mimecast representative for this integration. Under `Advanced options`
you can set the time interval between two API requests as well as the API URL. A Mimecast
representative should also be able to give you this information in case you need
to change the defaults.Then re-run elastic-package build so packages/mimecast/docs/README.md picks up the change.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| ID`, `Access Key`, and `Secret Key`) should be provided by a Mimecast | ||
| representative for this integration. Under `Advanced options` you can set the | ||
| time interval between two API requests as well as the API URL. A Mimecast | ||
| Authorization parameters for the Mimecast API (`Client ID` and `Client Key`) should |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/mimecast/docs/README.md:14
The generated docs/README.md still says Client Key while the source template _dev/build/docs/README.md now says Client Secret - the README was not rebuilt, so run elastic-package build and commit the regenerated file.
Details
packages/mimecast/docs/README.md is generated from packages/mimecast/_dev/build/docs/README.md by elastic-package build. Line 14 of the source template was updated in this PR to read Client Secret (matching the client_secret var in the root manifest), but line 14 of the generated file still reads Client Key. The generated copy therefore diverges from its own source, and it is the copy users actually read. CI rebuilds the package and asserts the working tree is clean, so this divergence fails the build check.
Recommendation:
Regenerate the docs so the built README matches the template:
cd packages/mimecast
elastic-package buildAfter the rebuild, line 14 of packages/mimecast/docs/README.md should read:
Authorization parameters for the Mimecast API (`Client ID` and `Client Secret`) should
be provided by a Mimecast representative for this integration. Under `Advanced options`
you can set the time interval between two API requests as well as the API URL. A Mimecast
representative should also be able to give you this information in case you need
to change the defaults.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| "subject": "Undelivered Mail Returned to Sender", | ||
| "from": { | ||
| "address": [ | ||
| "webmaster@empirepartners.b41.one" |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/mimecast/docs/README.md:397
All 11 sample events embedded in the generated docs/README.md predate the anonymisation pass and still contain the old mock addresses/domains, so the published docs leak the values this PR scrubbed everywhere else - rebuild the package docs from the current sample_event.json files.
Details
The event.ingested stamps show the README was built from an earlier system-test run than the committed sample events: every event block in docs/README.md carries an ingested value between 2026-07-27T03:45Z and 04:02Z, while all 11 data_stream/*/sample_event.json files carry 21:30Z-21:46Z. The later run is the anonymised one, so the generated README was never refreshed after it.
Concretely, data_stream/dlp_logs/sample_event.json now holds webmaster@example.org / vkamins@example.com / <...@mail.example.net>, but docs/README.md lines 397-417 still hold webmaster@empirepartners.b41.one, vkamins@demo-int.elastic.mime-api.com and <ae9f2f0678ed116f-152138@hapi.b41.one>. The same staleness affects the other data streams - e.g. the archive_search block still shows [User : dhamilton@mimecast.local] where data_stream/archive_search_logs/sample_event.json shows [User : user1@example.net].
This has two consequences: the generated file no longer matches its inputs, which fails the repo's post-build clean-tree check; and the pre-anonymisation addresses and domains that this PR deliberately removed from _dev/deploy/docker/files/config.yml and from every sample_event.json remain in the published documentation.
Recommendation:
Rebuild the package so the README picks up the current sample events, then verify no pre-anonymisation values survive:
cd packages/mimecast
elastic-package build
grep -nE 'mime-api\.com|b41\.one|hamilton321|mimecast\.local' docs/README.mdThe grep should return nothing, and each event block in docs/README.md should match its data stream's sample_event.json byte for byte, e.g. for dlp:
"email": {
"direction": "inbound",
"from": {
"address": [
"webmaster@example.org"
]
},
"message_id": "<ae9f2f0678ed116f-152138@mail.example.net>",
"subject": "New CERA.com Coming Soon! - CERA Alert",
"to": {
"address": [
"vkamins@example.com"
]
}
},🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| required: true | ||
| show_user: false | ||
| default: 5m | ||
| - name: initial_interval |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/mimecast/data_stream/siem_logs/manifest.yml:17
initial_interval is now a user-facing option on the default siem_logs stream but the CEL program ignores it — either wire look_back into the request or drop the variable.
Details
This PR promotes the CEL stream to be the default for siem_logs, which brings initial_interval (described as "Initial interval for the first API call. Defaults to 24 hours") with it as a required: true option.
The CEL program does not use it. data_stream/siem_logs/agent/stream/cel.yml.hbs sets look_back: {{initial_interval}}, start_field: dateRangeStartsAt and end_field: dateRangeEndsAt into state (lines 14-17), but the program body never reads state.look_back, state.start_field or state.end_field. The only place a date range is built is the batch-list request, which reads state.?start / state.?end — keys the template never populates:
?"dateRangeStartsAt": state.?start.optMap(start, [start.format("2006-01-02")]),
?"dateRangeEndsAt": state.?end.optMap(end, [end.format("2006-01-02")]),
So on a first run the request carries no date range at all and Mimecast returns its own default window (the program comment on lines 38-42 says that is the "full 7-day window"), regardless of what the user sets. Setting initial_interval: 1h collects seven days; setting 168h collects the same seven days.
The other nine data streams do consume it (state.start_field: state.?cursor.last.orValue((now - duration(state.look_back))...)), so the inconsistency is specific to the two streams that share the batch program: siem_logs and cloud_integrated_logs (whose cel.yml.hbs has the identical unused look_back/start_field/end_field keys). Because cloud_integrated_logs is also being enabled by default here, both default-on batch streams expose an option that does nothing.
Recommendation:
Either populate the date range from look_back so the option takes effect on the first poll:
// in the batch-list request, replace the state.?start / state.?end lookups
state.url.trim_right("/") + state.path + "?" + {
"type": [work_list[0].type],
?"nextPage": work_list[0].?next.optMap(next, [next]),
"dateRangeStartsAt": [
state.?cursor.last.orValue((now - duration(state.look_back)).format("2006-01-02"))
],
?"pageSize": state.?page_size.optMap(size, [string(int(size))]),
}.format_query()
or, if the batch API is intentionally left to its own default window, drop the dead keys from the template and remove the variable from the manifest so the UI does not advertise a setting that has no effect:
# data_stream/siem_logs/agent/stream/cel.yml.hbs
state:
client_id: {{client_id}}
client_secret: {{client_secret}}
page_size: {{batch_size}}
path: /siem/v1/batch/events/cgApply the same change to data_stream/cloud_integrated_logs/agent/stream/cel.yml.hbs and its manifest.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| be provided by a Mimecast representative for this integration. Under `Advanced options` | ||
| you can set the time interval between two API requests as well as the API URL. A Mimecast | ||
| > **Deprecated:** Mimecast API 1.0 is [end-of-life](https://mimecastsupport.zendesk.com/hc/en-us/articles/43572890309651-API-Integrations-API-1-0-End-of-Life-Project-Extension-Aug-2025). | ||
| > New installations should use the v2 API. Existing v1 configurations continue |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/mimecast/_dev/build/docs/README.md:25
The new deprecation note should also tell existing v1 users what happens at cutover — the httpjson streams are now off by default and switching a stream to CEL re-collects the last initial_interval of events.
Details
The note says "Existing v1 configurations continue to work", which is accurate for policies that are simply upgraded, but it omits the two things a v1 user actually needs to know once this ships:
-
The v1 streams are now disabled by default. Every data stream manifest now carries
enabled: falseon itshttpjsonstream (e.g.data_stream/siem_logs/manifest.yml:100). A user who reads this section on a fresh install, enters the four v1 credentials and saves will collect nothing until they also enable each... (deprecated)stream by hand. -
Migrating a stream from httpjson to CEL replays data. The CEL and httpjson streams keep independent cursors, so enabling CEL on a stream that has been running on httpjson starts from a fresh CEL cursor. For the nine non-batch streams that is
now - initial_interval, default24h(state.start_field: state.?cursor.last.orValue((now - duration(state.look_back))...)in eachcel.yml.hbs), so the cutover re-ingests roughly a day of already-collected events; forsiem_logsit is whatever window the Mimecast batch API defaults to. Linked issue 20296 calls this out explicitly ("Switching existing users off HTTPJSON can reset cursors and cause gaps or duplicates at cutover ... plus a documented deprecation path").
Both points belong in this block since it is the only migration guidance in the README.
Recommendation:
Extend the deprecation blockquote with the cutover details:
> **Deprecated:** Mimecast API 1.0 is [end-of-life](https://mimecastsupport.zendesk.com/hc/en-us/articles/43572890309651-API-Integrations-API-1-0-End-of-Life-Project-Extension-Aug-2025).
> New installations should use the v2 API. The v1 (HTTP JSON) streams are now
> disabled by default; to keep using them, enable the `... (deprecated)` stream
> on each data stream you need. Existing v1 configurations continue to work but
> will stop functioning when Mimecast retires API 1.0 application credentials.
> See the [API 1.0 to 2.0 Migration Guide](https://developer.services.mimecast.com/api-1-0-to-2-0-migration-guide)
> for details on provisioning v2 API credentials.
>
> **Migrating a data stream from v1 to v2:** the v1 and v2 streams track
> collection position independently, so enabling the v2 stream starts a fresh
> cursor at `now - Initial Interval` (default `24h`) and re-collects events
> already ingested via v1. Lower `Initial Interval` before the cutover to
> reduce the overlap, or accept the duplicates and de-duplicate downstream.Run elastic-package build afterwards so docs/README.md is regenerated.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
I do not think this would add any value.
| - description: Make CEL the default input for all data streams. The HTTPJSON input is deprecated and will be removed in a future release. New installations use the Mimecast v2 API via CEL; existing HTTPJSON configurations continue to work but users should migrate to CEL by provisioning v2 API credentials. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20356 | ||
| - description: Enable the cloud_integrated_logs and message_release_logs data streams by default. These are CEL-only data streams that were previously shipped disabled. |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/mimecast/changelog.yml:7
Enabling cloud_integrated_logs by default puts it alongside siem_logs, but the two target different Mimecast products — consider leaving it opt-in or documenting which tenants should turn it on.
Details
data_stream/siem_logs/agent/stream/cel.yml.hbs:15 collects from /siem/v1/batch/events/cg and data_stream/cloud_integrated_logs/agent/stream/cel.yml.hbs:15 collects from /siem/v1/batch/events/ci — the Cloud Gateway and Cloud Integrated batch endpoints. The package's own docs reinforce the split: the Cloud Integrated section of _dev/build/docs/README.md:68 links to Mimecast's threatssecurityeventsanddataforci reference. A tenant is normally entitled to one of these two products, which is presumably why cloud_integrated_logs shipped with enabled: false while the Gateway streams were on.
With this change both are on by default, so on a typical install one of the two runs permanently against an endpoint the tenant is not entitled to. The shared batch program treats a non-200/403/404 response as terminal ("events": {"error": {...}}, "want_more": false), so that stream emits an error event on every interval (default 5m) indefinitely.
message_release_logs is a Gateway feature and does not have this problem — enabling it by default is fine.
Note I could not reach Mimecast's developer docs to confirm the entitlement model (the host is not on the fetch allowlist), so this is flagged as a judgement call rather than a certainty; if hybrid Gateway + Cloud Integrated tenants are common, defaulting both on is defensible and only the docs gap remains.
Recommendation:
Either restore the opt-in default for the Cloud Integrated stream:
# packages/mimecast/data_stream/cloud_integrated_logs/manifest.yml
streams:
- input: cel
template_path: cel.yml.hbs
title: Cloud Integrated logs
enabled: false
description: Collect Cloud Integrated Logsor keep it enabled and add the guidance to _dev/build/docs/README.md next to the Cloud Integrated section:
### Cloud Integrated Logs
> Enable this data stream only if your tenant uses Mimecast Email Security,
> Cloud Integrated. Cloud Gateway tenants should use **SIEM Logs** instead and
> leave this data stream disabled — the Cloud Integrated endpoint will return
> errors for a Gateway-only tenant.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
This would diminish the value of the change here; to give a uniform entry.
| {"_offset":74369,"_partition":60,"taggedExternal":"true","taggedMalicious":"false","accountId":"CUSB4A274","aggregateId":"Zyc2YQp5NiebreJzicSY0g_1731827123","messageId":"<69b360ac6c02441e-173123@hapi.b41.one>","processingId":"rdbUbf7dO9buFz-hm-jV4eMdfFO2Qnu6iZy__ONwPng_1731827848","recipients":"max.yzaguirre@demo-int.elastic.mime-api.com","route":"Inbound","senderDomain":"arcadia-group.b41.one","senderEnvelope":"nikep@arcadia-group.b41.one","senderHeader":"nikep@arcadia-group.b41.one","senderIp":"81.2.69.144","subject":"Weekly Report","subtype":null,"timestamp":1731827851036,"type":"spam"} | ||
| {"_offset":75400,"_partition":60,"accountId":"CUSB4A274","aggregateId":"avtest1_1731500000","emailSize":"1234","fileExtension":"zip","fileName":"eicar.zip","md5":"40be8f71c7eb6fc53bba6e2edc1c5af2","messageId":"<test-av-1@hapi.b41.one>","processingId":"avtest1_proc_1731500000","recipients":"bob@demo-int.elastic.mime-api.com","route":"inbound","senderDomain":"evaluation-fuzz.b41.one","senderDomainInternal":"false","senderEnvelope":"alice@evaluation-fuzz.b41.one","senderIp":"81.2.69.144","sha1":"22fd3d86d8ec05c28d42fab2b765ba4e89147552","sha256":"f99da86bb6bd89416ff53fed876f48330fd8b9505a456d9968ae1321e331babb","subject":"Test AV Detection","subtype":null,"timestamp":1731500100000,"type":"av","virusFound":"Malware detected by AV Scan policy: Eicar-Test-Signature"} | ||
| {"_offset":75401,"_partition":60,"accountId":"CUSB4A274","aggregateId":"avtest2_1731600000","emailSize":"5678","fileExtension":"pdf","fileName":"document.pdf","md5":"d41d8cd98f00b204e9800998ecf8427e","messageId":"<test-av-2@demo-int.elastic.mime-api.com>","processingId":"avtest2_proc_1731600000","recipients":"carol@demo-int.elastic.mime-api.com","route":"internal","senderDomain":"demo-int.elastic.mime-api.com","senderDomainInternal":"true","senderEnvelope":"dave@demo-int.elastic.mime-api.com","senderIp":"10.108.135.241","sha1":"da39a3ee5e6b4b0d3255bfef95601890afd80709","sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","subject":"Internal Document","subtype":null,"timestamp":1731600200000,"type":"av","virusFound":null} | ||
| {"Hostname":"mail.example.net","_offset":70901,"_partition":60,"accountId":"CUSB4A274","aggregateId":"x0J66B7HOZCFDsdRtjBrdg","delivered":"true","deliveryAttempts":"1","deliveryErrors":null,"deliveryTime":"1633","destinationIp":"81.2.69.144","direction":"Internal","emailSize":"31363","messageId":"<27856769-1731451446596@usb-mta-24.usb.mimecast.lan>","numberAttachments":"0","processingId":"LcoZUvowTgh0_A1V-OdChmPWPlyI1gOurfQRUFoRMZk","recipients":"veronica.espinoza@example.com","rejectionCode":null,"rejectionInfo":null,"rejectionType":null,"route":"Accept and drop messages","senderEnvelope":"postmaster@example.com","subject":"We found suspicious files in a message","subtype":"true","timestamp":1731451448010,"tlsCipher":null,"tlsUsed":"No","tlsVersion":null,"totalSizeAttachments":"0","type":"delivery"} |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/mimecast/data_stream/siem_logs/_dev/test/pipeline/test-siem-v2-logs.log:11
The re-anonymised SIEM fixture still carries Mimecast internal MTA hostnames and real-looking personal names in the local part — replace those too for consistency with the rest of the pass.
Details
This pass replaced the customer-identifying domains (hamilton321.net, b41.one, demo-int.elastic.mime-api.com) with example.com / example.net / example.org, which is the important part. Two classes of value were left behind on the rewritten lines:
- Internal MTA hostnames inside
messageId. Line 11 sets"Hostname":"mail.example.net"but the same record keeps"messageId":"<27856769-1731451446596@usb-mta-24.usb.mimecast.lan>". Other rewritten lines keepusb-mta-10,usb-mta-22,usb-mta-26,usb-mta-40, and_dev/deploy/docker/files/config.ymlkeepsus-mta-167.us.mimecast.lan. The result is internally inconsistent — the envelope host is synthetic while the Message-ID host is not. - Personal-name local parts.
veronica.espinoza@example.com,richard.costigan@example.com,coo.jeff@example.comanddl-ga-all_enron_worldwide1@example.comstill name individuals; anonymising only the domain does not remove that.config.ymlalso retains"searchText":"example.org AND !Devin"where the domain was substituted but the personal name was not.
This is cosmetic rather than functional — the fixtures and their -expected.json are internally consistent and the pipeline tests will pass either way.
Recommendation:
Substitute the remaining hostnames and local parts, keeping the record shape identical:
{"Hostname":"mail.example.net","messageId":"<27856769-1731451446596@mta-24.example.net>","recipients":"user2@example.com","senderEnvelope":"postmaster@example.com","type":"delivery"}and in _dev/deploy/docker/files/config.yml:
# searchText: "example.org AND !Devin" ->
searchText: "example.org AND !user3"🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
Names here are Mimecast test environment names.
b9d81af to
2eb8b20
Compare
| @@ -1,9 +1,12 @@ | |||
| # newer versions go on top | |||
| - version: "3.5.0" | |||
| - version: "3.6.0" | |||
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/mimecast/changelog.yml:2
This edits the existing 3.5.0 changelog entry into 3.6.0 instead of adding a new entry, so the released 3.5.0 version and its 'Set elastic owner type' change (PR 20381) disappear from the history. Add a new 3.6.0 block above the untouched 3.5.0 block.
Details
On main the top of packages/mimecast/changelog.yml is:
- version: "3.5.0"
changes:- description: Set 'elastic' owner type.
type: enhancement
link: packages: set 'elastic' owner type on 14 security integrations #20381
- description: Set 'elastic' owner type.
I fetched the default-branch copy to confirm this. This PR rewrites that same block in place: the version string is changed to 3.6.0 and the single 3.5.0 change entry is replaced with the two new entries. The result is a changelog that goes 3.6.0 -> 3.4.4, with no 3.5.0 entry at all.
3.5.0 is already published, so its entry must remain. Removing it loses the record of the owner-type change and breaks the contiguous version history that the changelog is meant to provide.
Recommendation:
Keep the 3.5.0 block intact and prepend a new 3.6.0 block:
# newer versions go on top
- version: "3.6.0"
changes:
- description: Make CEL the default input for all data streams. The HTTPJSON input is deprecated and will be removed in a future release. New installations use the Mimecast v2 API via CEL; existing HTTPJSON configurations continue to work but users should migrate to CEL by provisioning v2 API credentials.
type: enhancement
link: https://github.com/elastic/integrations/pull/20356
- description: Enable the cloud_integrated_logs and message_release_logs data streams by default. These are CEL-only data streams that were previously shipped disabled.
type: enhancement
link: https://github.com/elastic/integrations/pull/20356
- version: "3.5.0"
changes:
- description: Set 'elastic' owner type.
type: enhancement
link: https://github.com/elastic/integrations/pull/20381
- version: "3.4.4"
changes:
- description: Apply grace period to threat intel feed end timestamp to avoid Mimecast API rejecting requests due to clock skew.
type: bugfix
link: https://github.com/elastic/integrations/pull/20001🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20381 | ||
| link: https://github.com/elastic/integrations/pull/20356 | ||
| - description: Enable the cloud_integrated_logs and message_release_logs data streams by default. These are CEL-only data streams that were previously shipped disabled. |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: low path: packages/mimecast/changelog.yml:7
Enabling cloud_integrated_logs by default means every new install polls both the Cloud Integrated and Cloud Gateway batch endpoints, and a tenant provisioned for only one will get continuous error events from the other. Consider leaving cloud_integrated_logs disabled by default, or documenting which one to turn off.
Details
cloud_integrated_logs and siem_logs share one CEL program (the program comment in both cel.yml.hbs files says so explicitly). They differ only in state.path: siem_logs uses /siem/v1/batch/events/cg (Cloud Gateway) and cloud_integrated_logs uses /siem/v1/batch/events/ci (Cloud Integrated). Their types defaults are also product specific -- siem_logs defaults to av/delivery/journal/receipt/spam etc., cloud_integrated_logs to entities/mailflow/urlclick.
This PR removes enabled: false from cloud_integrated_logs (and message_release_logs) while also making CEL the default input everywhere, so a fresh install now has both siem_logs and cloud_integrated_logs polling with the same v2 credentials.
In the shared program, a non-200 from the batch list endpoint falls into the error branch, which emits {"events": {"error": {...}}, "want_more": false}. That surfaces as an error event every interval for whichever product the tenant does not have. I could not reach the Mimecast developer portal to confirm the tenant-level exclusivity of the two products, hence the low confidence -- but if they are exclusive, this default is wrong for every installation.
message_release_logs is not affected by this concern; it has no Cloud Gateway/Cloud Integrated counterpart.
Recommendation:
Either keep cloud_integrated_logs opt-in:
# packages/mimecast/data_stream/cloud_integrated_logs/manifest.yml
title: "Cloud Integrated Logs"
type: logs
streams:
- input: cel
template_path: cel.yml.hbs
title: Cloud Integrated logs
enabled: false
description: Collect Cloud Integrated Logsor, if enabling both by default is intentional, say so in the README so users know to disable the one that does not apply to their tenant:
### Cloud Integrated Logs
> Mimecast tenants are provisioned for either Cloud Gateway or Cloud Integrated.
> Enable `cloud_integrated_logs` for Cloud Integrated tenants and `siem_logs` for
> Cloud Gateway tenants; leaving both enabled causes the endpoint your tenant does
> not have to return errors on every polling interval.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| # Mimecast Integration | ||
|
|
||
| The Mimecast integration collects events from the [Mimecast API](https://integrations.mimecast.com/documentation/). | ||
| The Mimecast integration collects events from the [Mimecast API](https://developer.services.mimecast.com/apis). |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/mimecast/_dev/build/docs/README.md:3
The intro link was moved to the v2 developer portal, but the ten per-dataset 'more information' links still point at integrations.mimecast.com, the documentation for the v1 API this PR deprecates. Repoint them at developer.services.mimecast.com.
Details
This PR changes the intro link from https://integrations.mimecast.com/documentation/ to https://developer.services.mimecast.com/apis, and adds a deprecation banner explaining that Mimecast API 1.0 is end-of-life. But integrations.mimecast.com is the API 1.0 documentation portal, and ten dataset descriptions further down the same file still link there:
- line 46 archive_search_logs
- line 57 audit_events
- line 79 dlp_logs
- line 90 message_release_logs
- line 101 siem_logs
- lines 112 and 123 threat_intel_malware_customer / _grid
- line 137 ttp_ap_logs
- line 148 ttp_ip_logs
- line 165 ttp_url_logs
Since CEL/v2 is now the default input for all of these data streams, the reference documentation a reader is sent to describes endpoints they are no longer calling. cloud_integrated_logs (line 68) already uses a developer.services.mimecast.com link, so the target style is established in this file.
Remember to regenerate packages/mimecast/docs/README.md after editing the source template.
Recommendation:
Repoint each dataset link at the v2 developer portal, matching the style already used for cloud_integrated_logs. For example:
### Archive Search Logs
This is the `mimecast.archive_search_logs` dataset. These logs contain Mimecast archive
search logs with the following details: search source, description and detailed
information about the search performed. More information about these logs is available [here](
https://developer.services.mimecast.com/docs/archivesearchlogs/1/routes/api/archive/get-archive-search-logs/post).Verify each replacement path against the v2 portal before committing; if a v2 equivalent does not exist for a given dataset, leave that link and add a short note that it documents the deprecated v1 endpoint.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| - SVR00005 # Kibana version for saved tags. | ||
| - SVR00006 # tags | ||
| - SVR00008 # on_failure event.kind | ||
| - SVR00009 # on_failure error.message No newline at end of file |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/mimecast/validation.yml:7
validation.yml now ends without a trailing newline, which elastic-package format will rewrite. Add the newline so a formatting check does not report the file as unformatted.
Details
The diff for packages/mimecast/validation.yml ends with \ No newline at end of file. The three new exclusion entries were appended without terminating the last line.
elastic-package normalises YAML files in the package and emits a trailing newline, so elastic-package format will rewrite this file and a format check will flag it as unformatted. Every other line in the file is otherwise untouched, so this is a one-character fix.
Recommendation:
Terminate the final line:
errors:
exclude_checks:
- SVR00004 # References in dashboards.
- SVR00005 # Kibana version for saved tags.
- SVR00006 # tags
- SVR00008 # on_failure event.kind
- SVR00009 # on_failure error.message(with a newline after the last entry). Running elastic-package format in packages/mimecast will do this for you.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
1a3ea48 to
b672571
Compare
| name: '{{ IngestPipeline "v2_pipeline" }}' | ||
| if: ctx.mimecast?.timestamp != null | ||
| tag: v1_pipeline | ||
| tag: v2_pipeline |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/mimecast/data_stream/siem_logs/elasticsearch/ingest_pipeline/default.yml:52
The v2 SIEM pipeline this change promotes to the default path renames three field names the v2 API never emits (mimecast.tlsVer, mimecast.Delivered, mimecast.Hld), so tls.version, event.outcome and event.kind stay unpopulated; rename them to the actual v2 keys tlsVersion / delivered / holdReason.
Details
This PR makes the CEL (v2 API) stream the default for siem_logs, so v2_pipeline.yml is now the path every new installation takes. Three of its processors key off field names the v2 batch API does not produce, and the fixtures regenerated in this PR prove each one is a no-op:
-
v2_pipeline.yml:126renamesmimecast.tlsVertotls.version. The v2 payload usestlsVersion(see the added lines in_dev/test/pipeline/test-siem-v2-logs.log, andfields/field.yml:244which declaresmimecast.tlsVersion). Result:tls.versionis never set, and the raw value is left behind undermimecast.tlsVersion(sample_event.json:64;test-siem-v2-logs.log-expected.json:2255). NotetlsCipheron the adjacent processor is spelled correctly and does map totls.cipher. -
v2_pipeline.yml:311renamesmimecast.Deliveredtoevent.outcome. The v2 payload uses lowercasedelivered. Every delivery event in the regenerated expectations therefore falls through toevent.outcome: "unknown"whilemimecast.delivered: "true"remains unmapped (test-siem-v2-logs.log-expected.json:511and:517). -
v2_pipeline.yml:150setsevent.kind: alertwhenctx.mimecast?.Hld != null. NoHldfield exists in v2 process logs; the hold is signalled byaction: "Hld"plusholdReason. Held messages consequently carryevent.action: "Hld"andevent.reason: "Spm"but noevent.kind: "alert"(test-siem-v2-logs.log-expected.json:1750-1759), which weakens alert-based detection on the default path.
Recommendation:
Correct the three field references in data_stream/siem_logs/elasticsearch/ingest_pipeline/v2_pipeline.yml and regenerate the pipeline-test expectations:
# was: field: mimecast.tlsVer
- rename:
field: mimecast.tlsVersion
tag: rename_tls_version
target_field: tls.version
ignore_missing: true
# was: if: ctx.mimecast?.Hld != null && ctx.mimecast.Hld != ''
- set:
field: event.kind
tag: set_event_kind_held
value: alert
if: ctx.mimecast?.holdReason != null && ctx.mimecast.holdReason != ''
# was: field: mimecast.Delivered
- rename:
field: mimecast.delivered
tag: rename_delivered_to_event_outcome
target_field: event.outcome
ignore_missing: trueThe delivered value arrives as the string "true"/"false", so the follow-up set processors need to match strings rather than booleans:
- set:
field: event.outcome
tag: set_event_outcome_success
value: success
if: ctx.event?.outcome instanceof String && ctx.event.outcome.equalsIgnoreCase('true')
- set:
field: event.outcome
tag: set_event_outcome_failure
value: failure
if: ctx.event?.outcome instanceof String && ctx.event.outcome.equalsIgnoreCase('false')🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| - name: alerting | ||
| type: text | ||
| title: Alert Actions | ||
| description: The set of DLP actions that should be classified as an alert. Possible values are delete, hold, bouce, smart_folder, disable_smart_folder, content_expire, meta_expire, stationery, disable_stationery, gcc, secure_delivery, delivery_route, document_policy, disable_document_policy, secure_messaging, disable_secure_messaging_policy, attach_set_policy, remove_email, tag, link, block, none, and notification. |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/mimecast/data_stream/dlp_logs/manifest.yml:36
The DLP Alert Actions description added to the CEL stream misspells the Mimecast action bounce as bouce; fix the spelling so the Fleet UI lists a value users can actually enter.
Details
This line moves the alerting variable description onto the CEL stream, which this PR makes the default and which is therefore the copy users now see in Fleet. The enumerated list of accepted DLP actions contains bouce instead of bounce. Since the description is the only place the accepted values are documented, a user copying the value verbatim configures an action the pipeline will never match.
Recommendation:
Fix the spelling in the alerting variable description:
- name: alerting
type: text
title: Alert Actions
description: The set of DLP actions that should be classified as an alert. Possible values are delete, hold, bounce, smart_folder, disable_smart_folder, content_expire, meta_expire, stationery, disable_stationery, gcc, secure_delivery, delivery_route, document_policy, disable_document_policy, secure_messaging, disable_secure_messaging_policy, attach_set_policy, remove_email, tag, link, block, none, and notification.
multi: true
required: true
show_user: true
default:
- block🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| { | ||
| "senderAddress": "kay.mann@demo-int.elastic.mime-api.com", | ||
| "recipientAddress": "cbrashear@demo-visionary.b41.one", | ||
| "senderAddress": "kay.mann@example.com", |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/mimecast/_dev/deploy/docker/files/config.yml:279
The anonymisation pass replaced the domains in these fixtures but left real personal names in the email local parts (and the tenant account ID in the SIEM fixture); replace the local parts with synthetic names such as alice.johnson/bob.smith and the account ID with a synthetic code.
Details
This PR rewrites the mock service data and the siem_logs pipeline fixture to use RFC 2606 domains (example.com / example.net / example.org) and RFC 5737 addresses, but only the domain part was substituted. The local parts are still recognisable real people's names carried over from the source corpus - kay.mann, zimin.lu, vince.j.kaminski, thomas.bentz, william.mccoy, monika.causholli, craig.dean, mike.a.roberts, gregoryhunt and, in packages/mimecast/data_stream/siem_logs/_dev/test/pipeline/test-siem-v2-logs.log, veronica.espinoza and richard.costigan. One search string in the same file also still carries a bare first name ('... AND !Devin'). The siem_logs fixture additionally retains the tenant-scoped accountId value on every record, which is an organisational identifier. The anonymisation guidance treats usernames, display names and account IDs as identifiers that must be replaced with synthetic values, not merely re-domained, so the pass is incomplete on files this PR is explicitly anonymising.
Recommendation:
Substitute the local parts and the account identifier with synthetic values, keeping the record shape byte-for-byte otherwise:
{
"senderAddress": "alice.johnson@example.com",
"recipientAddress": "bob.smith@example.net",
"subject": "RE",
"eventTime": "2024-11-18T05:11:02+0000",
"route": "outbound",
"policy": "Confidential",
"action": "hold",
"messageId": "<ee36704d0e84b012-226716@mail.example.net>"
}Apply the same mapping consistently across _dev/deploy/docker/files/config.yml and data_stream/siem_logs/_dev/test/pipeline/test-siem-v2-logs.log (plus its -expected.json), and replace the accountId value with a synthetic same-shape code, e.g.:
{
"accountId": "CUSA0A000",
"recipients": "alice.johnson@example.com",
"senderEnvelope": "postmaster@example.com"
}🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| - description: Make CEL the default input for all data streams. The HTTPJSON input is deprecated and will be removed in a future release. New installations use the Mimecast v2 API via CEL; existing HTTPJSON configurations continue to work but users should migrate to CEL by provisioning v2 API credentials. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20356 | ||
| - description: Enable the cloud_integrated_logs and message_release_logs data streams by default. These are CEL-only data streams that were previously shipped disabled. |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/mimecast/changelog.yml:7
Enabling cloud_integrated_logs by default makes every new policy poll both the Cloud Integrated and the Cloud Gateway SIEM endpoints, but a tenant only has one of those products; keep cloud_integrated_logs disabled by default or document which product each stream requires.
Details
packages/mimecast/data_stream/cloud_integrated_logs/agent/stream/cel.yml.hbs requests /siem/v1/batch/events/ci (Mimecast Cloud Integrated) while packages/mimecast/data_stream/siem_logs/agent/stream/cel.yml.hbs requests /siem/v1/batch/events/cg (Mimecast Cloud Gateway). These are separate Mimecast product lines and a tenant is provisioned for one of them. siem_logs is already enabled by default; this PR removes enabled: false from cloud_integrated_logs, so every newly created policy now polls both endpoints and one of the two will never return data for the tenant. message_release_logs (/api/gateway/get-held-release-logs) is Cloud Gateway only and is enabled by default in the same change. Existing policies keep their stored per-stream state on upgrade, so this affects new installations. cloud_integrated_logs was shipped disabled up to now, which suggests the product split was the reason.
Recommendation:
Either keep the Cloud Integrated stream opt-in:
# packages/mimecast/data_stream/cloud_integrated_logs/manifest.yml
streams:
- input: cel
template_path: cel.yml.hbs
title: Cloud Integrated logs
enabled: false
description: Collect Cloud Integrated Logsor, if enabling it by default is deliberate, say so in the changelog entry and in _dev/build/docs/README.md so users know which streams belong to which Mimecast product:
- description: Enable the cloud_integrated_logs and message_release_logs data streams by default. cloud_integrated_logs applies to Mimecast Cloud Integrated tenants; siem_logs and message_release_logs apply to Mimecast Cloud Gateway tenants. Disable the streams that do not apply to your tenant.
type: enhancement
link: https://github.com/elastic/integrations/pull/20356🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
Make the CEL (v2 API) input the default for all data streams by listing it first and enabling it. The HTTP JSON (v1 API) input is now disabled by default and marked as deprecated in stream titles, descriptions, and documentation. Mimecast API 1.0 is end-of-life (ability to create new applications ends December 2026). Existing policies with HTTP JSON enabled are unaffected; the stream definitions remain present for users who have not yet migrated to v2 API credentials. Also enables the two CEL-only data streams (cloud_integrated_logs, message_release_logs) by default. They were previously disabled with no alternative.
On supporting version of kibana this will display a banner indicating deprecation and will not display the deprecated input for new installations (I think this is the wrong behaviour, but it is intended in the kibana change that added it). Validation checks that fail due to the bump in format version are deferred to a follow-up PR.
|
✅ All changelog entries have the correct PR link. |
| - description: Make CEL the default input for all data streams. The HTTPJSON input is deprecated and will be removed in a future release. New installations use the Mimecast v2 API via CEL; existing HTTPJSON configurations continue to work but users should migrate to CEL by provisioning v2 API credentials. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20356 | ||
| - description: Enable the cloud_integrated_logs and message_release_logs data streams by default. These are CEL-only data streams that were previously shipped disabled. |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/mimecast/changelog.yml:7
Enabling cloud_integrated_logs by default makes every new install poll both the Cloud Integrated and Cloud Gateway SIEM endpoints, but a tenant is provisioned for only one; keep cloud_integrated_logs disabled by default (or make the pair mutually exclusive) and document which product each applies to.
Details
This PR removes enabled: false from data_stream/cloud_integrated_logs/manifest.yml while siem_logs is also enabled by default, so a fresh install runs both. The two streams target different Mimecast product lines, not two views of the same data: cloud_integrated_logs/agent/stream/cel.yml.hbs:15 sets path: /siem/v1/batch/events/ci with default types entities, mailflow, urlclick, whereas siem_logs/agent/stream/cel.yml.hbs sets path: /siem/v1/batch/events/cg with default types av, delivery, journal, receipt, spam, etc. The disjoint paths and disjoint event-type vocabularies are the Cloud Integrated and Cloud Gateway products. A tenant on Cloud Gateway has no /ci entitlement and a tenant on Cloud Integrated has no /cg entitlement, so on every new install one of the two streams issues a failing request each interval (default 5m) and, per the shared CEL program's non-200 branch, emits an error event and halts with want_more: false. message_release_logs is unaffected by this — it is a Gateway held-message endpoint and pairs with siem_logs. Note the two streams were shipped enabled: false for exactly this reason, not merely because they lacked an HTTPJSON alternative.
Recommendation:
Keep cloud_integrated_logs opt-in and leave message_release_logs enabled, so the default policy is coherent for a Cloud Gateway tenant:
# packages/mimecast/data_stream/cloud_integrated_logs/manifest.yml
title: "Cloud Integrated Logs"
type: logs
streams:
- input: cel
template_path: cel.yml.hbs
title: Cloud Integrated logs
enabled: false
description: Collect Cloud Integrated Logs. Enable this only on Mimecast Cloud Integrated tenants; Cloud Gateway tenants should use the SIEM Logs data stream instead.and narrow the changelog entry to match:
- description: Enable the message_release_logs data stream by default. cloud_integrated_logs remains disabled by default because it applies only to Mimecast Cloud Integrated tenants, whereas siem_logs applies to Cloud Gateway tenants.
type: enhancement
link: https://github.com/elastic/integrations/pull/20356🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| Authorization parameters for the Mimecast API (`Client ID` and `Client Key`) should | ||
| be provided by a Mimecast representative for this integration. Under `Advanced options` | ||
| you can set the time interval between two API requests as well as the API URL. A Mimecast | ||
| > **Deprecated:** Mimecast API 1.0 is [end-of-life](https://mimecastsupport.zendesk.com/hc/en-us/articles/43572890309651-API-Integrations-API-1-0-End-of-Life-Project-Extension-Aug-2025). |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/mimecast/_dev/build/docs/README.md:24
The new deprecation notice tells v1 users to migrate to the v2 CEL input but does not say the CEL stream starts with an empty cursor, so the cutover re-ingests up to initial_interval of events; add a migration subsection describing the duplicate/gap window and how to control it.
Details
The HTTPJSON and CEL streams are separate Filebeat inputs with separate state stores, so enabling the CEL stream does not inherit the HTTPJSON cursor. In data_stream/archive_search_logs/agent/stream/cel.yml.hbs:71 the start time is state.?cursor.last.orValue((now - duration(state.look_back)).format(time_layout.RFC3339)) — with no saved cursor it falls back to look_back, which is bound to the initial_interval var (default 24h). The same pattern is used by the shared program in dlp_logs, message_release_logs, ttp_ap_logs, ttp_ip_logs and ttp_url_logs; the siem_logs/cloud_integrated_logs program similarly seeds a fresh work_list with no last_tokens. So a user who follows this PR's advice and switches a data stream from HTTPJSON to CEL re-collects up to 24 hours of events already indexed by the HTTPJSON stream (duplicates), and if HTTPJSON is disabled more than initial_interval before CEL is enabled, the intervening window is never collected (gap). Neither the README nor the changelog entry mentions this, and linked issue #20296 explicitly calls out cursor continuity at cutover as the main risk of this migration.
Recommendation:
Add a migration subsection under the v1 deprecation notice in _dev/build/docs/README.md (then rebuild docs/README.md):
### Migrating from the v1 (HTTPJSON) input to the v2 (CEL) input
The v1 and v2 inputs keep separate collection cursors, so the v2 input does not
resume where the v1 input stopped. When you enable a v2 data stream for the
first time it starts from `now - Initial Interval` (default `24h`).
To migrate a data stream:
1. Note the time at which you disable the v1 stream.
2. Enable the v2 stream and set `Initial Interval` to cover the gap between that
time and now. Values shorter than the gap lose events; longer values
re-collect events the v1 stream already indexed.
3. Expect duplicate documents for the overlap window. `event.id` is stable per
record for most data streams, so duplicates can be filtered at query time.
Run both inputs in parallel only if you accept full duplication for the overlap.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
Review summaryIssues found across the latest commits dd8dee5…7039598 (51 commits) — 2 medium
Issues found across earlier commits a16ff9c — 2 medium
Issues found across earlier commits 3afbf5f…c555423 (3 commits) — 1 medium, 1 low
Issues found across earlier commits dc194c5…b9d81af (13 commits) — 1 high, 1 medium, 2 low
Issues found across earlier commits 6dd99a5 — 3 medium, 1 low
Issues found across earlier commits 005f16a — 2 high
Issues found across earlier commits 7e9b9bd — 2 medium, 1 low
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
💚 Build Succeeded
History
cc @efd6 |
chrisberkhout
left a comment
chrisberkhout
left a comment
There was a problem hiding this comment.
Good. A bit noisy with the data changes but good to get them done as well.
|
Tick the box to add this pull request to the merge queue (same as
|
|
Package mimecast - 3.6.0 containing this change is available at https://epr.elastic.co/package/mimecast/3.6.0/ |
5 participants
Proposed commit message
Note
The HTTP JSON input config options will be removed when the Mimecast v1 API endpoints actually reach EOL.
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots