From 388f398d0cc74e327c924e2b1bded00e986f9c79 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Magnus=20Hartvig=20Gr=C3=B8nbech?= Date: Wed, 29 Jul 2026 09:37:01 +0200 Subject: [PATCH 1/3] [Shopify] Refresh connector documentation for changes since April Brings the Shopify Connector and Test app documentation up to date with the 48 commits merged since the docs were last refreshed (baseline 3227a356, "Remove B2B Plus-only plan restriction"). Documentation only. No AL source changes. App documentation: - Unlisted product status and the new "Shpfy Create Prod. Status Unlisted" implementation of ICreateProductStatusValue - HS code and country of origin, carried on the variant and gated by the Shop's "Sync HS Code and Country" setting - Order tax lines can now hang off order shipping charges, and market-driven shipping methods are fetched through new GraphQL queries - Return-with-exchange refunds: the "Is Exchange Item" order line flag and the synthetic negative refund lines that keep the credit memo total right - Credit memo creation is skipped while a refund transaction is pending - Order mapping keeps a manually set Sell-to Customer No. when Bill-to mapping fails, and creates a BC customer when Shopify has no default address - Bulk price sync surfaces skipped records and the sent JSONL, and decides bulk versus single on the count of changed prices - Corrected the GraphQL rate limiter wait calculation and the query count - "Auto Create Catalog" renamed to "Auto Create B2B Catalog", now always visible with plan validation on enable - Belgium Enterprise No. tax ID mapping and ISO country code tax area filtering Test documentation: - Corrected a stale claim that 3 tests were disabled via app-local DisabledTests/*.json. Those files were removed; disabled-test filtering now comes from central DisabledTests folders read by RunTestsInBcContainer.ps1 - Documented the new Variant API and Transactions test coverage, the .resources/Shipping mocks, and the price calculation isolation pattern - Clarified that Shpfy Bulk Op. Subscriber is bound in the webhooks test, not the bulk operations test Marker files updated to d86ecf32. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/Apps/W1/Shopify/App/.docs-updated | 4 +- src/Apps/W1/Shopify/App/CLAUDE.md | 12 +- .../W1/Shopify/App/docs/business-logic.md | 28 ++++- src/Apps/W1/Shopify/App/docs/data-model.md | 17 ++- src/Apps/W1/Shopify/App/docs/extensibility.md | 8 +- src/Apps/W1/Shopify/App/docs/patterns.md | 18 ++- .../W1/Shopify/App/src/Base/docs/CLAUDE.md | 51 ++++++++- .../App/src/Bulk Operations/docs/CLAUDE.md | 12 +- .../Shopify/App/src/Catalogs/docs/CLAUDE.md | 15 +++ .../Shopify/App/src/Companies/docs/CLAUDE.md | 15 ++- .../App/src/Companies/docs/extensibility.md | 28 ++++- .../Shopify/App/src/Customers/docs/CLAUDE.md | 9 ++ .../App/src/Customers/docs/business-logic.md | 18 ++- .../App/src/Customers/docs/extensibility.md | 16 +++ .../W1/Shopify/App/src/GraphQL/docs/CLAUDE.md | 18 ++- .../Shopify/App/src/GraphQL/docs/patterns.md | 26 ++++- .../App/src/Order Refunds/docs/CLAUDE.md | 8 ++ .../docs/CLAUDE.md | 9 +- .../docs/extensibility.md | 4 + .../App/src/Order handling/docs/CLAUDE.md | 17 ++- .../src/Order handling/docs/business-logic.md | 51 +++++++-- .../App/src/Order handling/docs/data-model.md | 30 ++++- .../src/Order handling/docs/extensibility.md | 12 ++ .../Shopify/App/src/Products/docs/CLAUDE.md | 44 +++++++- .../App/src/Products/docs/business-logic.md | 103 +++++++++++++++++- .../App/src/Products/docs/data-model.md | 63 ++++++++++- .../App/src/Products/docs/extensibility.md | 43 +++++++- .../Shopify/App/src/Shipping/docs/CLAUDE.md | 10 +- .../App/src/Transactions/docs/CLAUDE.md | 6 +- src/Apps/W1/Shopify/Test/.docs-updated | 4 +- .../Test/Bulk Operations/docs/CLAUDE.md | 14 ++- src/Apps/W1/Shopify/Test/CLAUDE.md | 23 ++-- .../W1/Shopify/Test/Companies/docs/CLAUDE.md | 9 +- .../W1/Shopify/Test/Products/docs/CLAUDE.md | 12 +- .../W1/Shopify/Test/docs/business-logic.md | 23 +++- src/Apps/W1/Shopify/Test/docs/patterns.md | 42 ++++++- 36 files changed, 717 insertions(+), 105 deletions(-) diff --git a/src/Apps/W1/Shopify/App/.docs-updated b/src/Apps/W1/Shopify/App/.docs-updated index 51bbc28924c..4e998c37321 100644 --- a/src/Apps/W1/Shopify/App/.docs-updated +++ b/src/Apps/W1/Shopify/App/.docs-updated @@ -1,4 +1,4 @@ # Documentation last updated -commit: 343aa21d1e5737d250aabf05cd9233d15c411cdc -date: 2026-04-08 +commit: d86ecf324ac3682e1d2394fb3ba0425a4b95de24 +date: 2026-07-29 scope: full diff --git a/src/Apps/W1/Shopify/App/CLAUDE.md b/src/Apps/W1/Shopify/App/CLAUDE.md index 4cc88de21ae..3ddecc8606c 100644 --- a/src/Apps/W1/Shopify/App/CLAUDE.md +++ b/src/Apps/W1/Shopify/App/CLAUDE.md @@ -19,9 +19,13 @@ All Shopify API communication goes through GraphQL. The `ShpfyCommunicationMgt.C *Updated: 2026-03-24 -- GraphQL resource file refactoring* -Mapping strategies are interface-driven throughout. Customer mapping (`ICustomerMapping`) selects between by-email/phone, by-bill-to, or default-customer strategies. Company mapping (`ICompanyMapping`) can match by email/phone or tax ID. Stock calculation uses `IStockAvailable` and `IStockCalculation` interfaces. Product status on creation, removal actions for blocked items, county resolution, and customer name formatting are all interface-backed enums. The Shop record's enum fields (e.g., `"Customer Mapping Type"`, `"Stock Calculation"`, `"Status for Created Products"`) select which implementation to use at runtime. +Mapping strategies are interface-driven throughout. Customer mapping (`ICustomerMapping`) selects between by-email/phone, by-bill-to, or default-customer strategies. Company mapping (`ICompanyMapping`) can match by email/phone or tax ID. Stock calculation uses `IStockAvailable` and `IStockCalculation` interfaces. Product status on creation, removal actions for blocked items, county resolution, and customer name formatting are all interface-backed enums. The Shop record's enum fields (e.g., `"Customer Mapping Type"`, `"Stock Calculation"`, `"Status for Created Products"`) select which implementation to use at runtime. Newly created Shopify products can now start as Active, Draft, or Unlisted through the same `ICreateProductStatusValue` path. -Sync is incremental via the Synchronization Info table (`ShpfySynchronizationInfo.Table.al`), which stores the last sync timestamp per shop and sync type. An empty/zero date falls back to a sentinel value of `2004-01-01` (see `GetEmptySyncTime()`). Products use hash-based change detection -- the Product table stores `"Image Hash"`, `"Tags Hash"`, and `"Description Html Hash"` fields, computed via a custom hash algorithm in `ShpfyHash`, to avoid unnecessary API calls when nothing has actually changed. +*Updated: 2026-07-29 -- Unlisted added as a built-in created-product status* + +Sync is incremental via the Synchronization Info table (`ShpfySynchronizationInfo.Table.al`), which stores the last sync timestamp per shop and sync type. An empty/zero date falls back to a sentinel value of `2004-01-01` (see `GetEmptySyncTime()`). Products use hash-based change detection -- the Product table stores `"Image Hash"`, `"Tags Hash"`, and `"Description Html Hash"` fields, computed via a custom hash algorithm in `ShpfyHash`, to avoid unnecessary API calls when nothing has actually changed. Variants also carry customs metadata (`"Tariff No."` and `"Country/Region of Origin Code"`), synchronized with Shopify inventory-item HS code and country of origin when the Shop's `"Sync HS Code and Country"` setting is enabled. + +*Updated: 2026-07-29 -- Variant customs metadata now syncs with Shopify inventory items* Records link to BC entities via SystemId (GUID), not Code/No. For example, `Shpfy Product` has an `"Item SystemId"` field linking to BC Items, with `"Item No."` as a FlowField that looks up the human-readable code. This means renumbering items in BC does not break Shopify links. The same pattern applies to variants (`"Item Variant SystemId"`) and customers. B2B support adds companies, company locations, and catalogs with company-specific pricing, all orchestrated through the `src/Companies/` and `src/Catalogs/` modules. @@ -63,8 +67,11 @@ Records link to BC entities via SystemId (GUID), not Code/No. For example, `Shpf - The Shop table is the god object -- nearly every configuration setting lives there, with over 100 fields controlling sync directions, mapping strategies, account mappings, plan-based feature flags, webhook config, and more. The `"Advanced Shopify Plan"` field gates features requiring Plus/Advanced plans (currently staff members). B2B features are now unconditionally available on all plans. - All API calls go through GraphQL, never REST. Queries are `.graphql` resource files in `.resources/graphql/{Area}/`, loaded via `NavApp.GetResourceAsText()` and dispatched through the `ShpfyGraphQLType` enum. - Products use hash-based change detection (`"Image Hash"`, `"Tags Hash"`, `"Description Html Hash"`) via a custom hash algorithm to skip unnecessary API calls when nothing has changed. +- `"Status for Created Products"` supports Active, Draft, and Unlisted. Unlisted is handled by a dedicated `ICreateProductStatusValue` implementation, so product creation does not need a special-case branch for Shopify's visibility state. +- `"Sync HS Code and Country"` defaults on and drives both directions of customs metadata: BC item tariff/country values are sent as Shopify variant inventory-item data, and imported Shopify values can update BC item tariff/country fields. - Records link to BC entities via SystemId (GUID), not Code/No. -- FlowFields like `"Item No."` display the human-readable values via CalcFormula lookup. Renumbering BC items does not break Shopify links. - Orders store every monetary amount in dual currency: shop currency fields (`"Total Amount"`, `"VAT Amount"`) and presentment/customer-facing currency fields (`"Presentment Total Amount"`, `"Presentment VAT Amount"`). The `"Currency Handling"` setting on Shop controls which is used for BC documents. +- Shipping charges have their own `Shpfy Order Shipping Charges` records, and their Shopify `taxLines` are stored as `Shpfy Order Tax Line` rows parented by the shipping line ID. Deleting a shipping charge deletes its tax lines. - Returns and refunds are independent concepts in Shopify's model -- a refund can exist without a return and vice versa. The connector has three processing modes: import only, auto-create credit memo, and manual. - Return-with-exchange flows: Shopify exposes exchange items on `Return.exchangeLineItems`, not on `Refund.refundLineItems`. The connector flags the matching `Shpfy Order Line` rows with `"Is Exchange Item" = true` so they are excluded from the BC sales invoice, and inserts negative-quantity `Shpfy Refund Line` rows so the credit memo total matches `Refund.totalRefundedSet` without a spurious *Refund Account* G/L balancing line. - Fulfillment Orders (requests assigned to a location) are different from Fulfillments (actual shipments). Both have their own header/line tables. @@ -73,4 +80,5 @@ Records link to BC entities via SystemId (GUID), not Code/No. For example, `Shpf - Metafields use a polymorphic owner pattern (`"Parent Table No."` + `"Owner Id"`) to attach to products, variants, customers, or companies. The `IMetafieldOwnerType` interface resolves the table and shop code. - Gift cards appear as both an order line type (when purchased) and a payment method (when redeemed). They have a dedicated G/L account (`"Sold Gift Card Account"`) on the Shop table. - Bulk operations use async GraphQL mutations with webhook callback -- `IBulkOperation` implementations provide the mutation, input JSONL, and revert logic for failures. +- Bulk price sync only uses the bulk path when the number of changed variant price/cost/compare-at updates reaches the threshold. Failed bulk price lines are reverted and logged as skipped records; when logging mode is All, the sent JSONL can be downloaded from the bulk operation. - The empty sync time sentinel is `2004-01-01` (the `GetEmptySyncTime()` method), not `0DT`. Order sync uses the Shop's `"Shop Id"` hash as the sync key (not the Shop Code) so that multiple BC companies connected to the same Shopify shop share the same sync cursor. diff --git a/src/Apps/W1/Shopify/App/docs/business-logic.md b/src/Apps/W1/Shopify/App/docs/business-logic.md index 5942d3a3b97..0424d8845b1 100644 --- a/src/Apps/W1/Shopify/App/docs/business-logic.md +++ b/src/Apps/W1/Shopify/App/docs/business-logic.md @@ -12,18 +12,28 @@ The export flow is driven by `ShpfyProductExport.Codeunit.al`. It iterates all S The update flow relies heavily on hash-based change detection. Before making any API call, the codeunit computes the current hash of the product's tags, HTML description, and images, then compares them against the stored hashes on the Product record. Only fields that have actually changed result in API calls. This is critical because Shopify's GraphQL API has per-store rate limits, and a full product catalog update without change detection would quickly exhaust the budget. -Price sync can run independently of the full product sync via the `OnlyUpdatePrice` flag. When enabled, prices are batched into a bulk mutation (`ShpfyBulkUpdateProductPrice.Codeunit.al`) for efficiency. If the bulk operation fails, it falls back to individual variant price updates, reverting variant changes on failure. Item price sync is skipped when the item's unit of measure is invalid, preventing errors from propagating into Shopify pricing data. +Price sync can run independently of the full product sync via the `OnlyUpdatePrice` flag. When enabled, changed variant price, compare-at price, and unit cost updates are collected for a bulk mutation (`ShpfyBulkUpdateProductPrice.Codeunit.al`). The bulk threshold is based on that changed-update count, not on the total product or variant count scanned. If the bulk operation cannot be started, the connector falls back to individual variant price updates. If Shopify rejects specific bulk lines, the connector reverts those variant changes and logs skipped records with the Shopify error where available. Item price sync is skipped when the item's unit of measure is invalid, preventing errors from propagating into Shopify pricing data. *Updated: 2026-04-08 -- Price sync skipped for invalid unit of measure* +*Updated: 2026-07-29 -- Bulk price sync now counts changed updates and records skipped failures* + The product body HTML is assembled from BC data by `CreateProductBody()` in the export codeunit -- it concatenates extended text, marketing text, and item attributes into an HTML structure. Events fire before and after this assembly (`OnBeforeCreateProductBodyHtml`, `OnAfterCreateProductBodyHtml`), allowing extensions to customize the output. +When a BC item is added to Shopify, `ShpfyCreateProduct` sets the product status through `ICreateProductStatusValue`. The built-in choices are Active, Draft, and Unlisted; all three feed the same product creation flow and convert to Shopify's uppercase GraphQL status value. + +*Updated: 2026-07-29 -- Unlisted added to the product creation status flow* + ### Shopify to BC import The import direction is handled by `ShpfyProductImport.Codeunit.al`. It fetches products from Shopify using timestamp-based incremental sync, creates or updates local Product and Variant records, and optionally creates BC Items. Item creation uses the template system -- the Shop's `"Item Templ. Code"` provides defaults, and the `OnAfterFindItemTemplate` event allows per-product template selection. `ShpfyCreateItem.Codeunit.al` handles the actual item creation. Variant mapping is complex because Shopify variants carry option values (size/color) that may map to BC Item Variants, and the `"UoM as Variant"` setting can make unit-of-measure values appear as variant options. +When `"Sync HS Code and Country"` is enabled, imported Shopify variant inventory-item data can set the BC item's Tariff No. and Country/Region of Origin Code. Existing-item updates resolve the Shopify values through BC Tariff Number and Country/Region records before applying them, so missing or unmapped values are ignored instead of clearing valid BC data. + +*Updated: 2026-07-29 -- HS code and country of origin now flow from Shopify to BC items* + ```mermaid flowchart TD A[Start product sync] --> B{Sync direction?} @@ -54,16 +64,22 @@ Order processing is the most complex flow in the connector. It spans multiple co ### Import -`ShpfyImportOrder.Codeunit.al` takes an order from the staging table and creates the full Order Header and Order Lines. It parses the GraphQL response to populate all address blocks, financial fields (in both currencies), customer IDs, discount codes, attributes, tax lines, and risk assessments. Staff member handling during order import is gated by the Shop's `"Advanced Shopify Plan"` flag (true for Plus, Plus Trial, Development, and Advanced plans). The `OnAfterImportShopifyOrderHeader` and `OnAfterCreateShopifyOrderAndLines` events fire after import. +`ShpfyImportOrder.Codeunit.al` takes an order from the staging table and creates the full Order Header and Order Lines. It parses the GraphQL response to populate all address blocks, financial fields (in both currencies), customer IDs, discount codes, attributes, tax lines, and risk assessments. Order line GraphQL paging now requests 10 line items per page. Staff member handling during order import is gated by the Shop's `"Advanced Shopify Plan"` flag (true for Plus, Plus Trial, Development, and Advanced plans). The `OnAfterImportShopifyOrderHeader` and `OnAfterCreateShopifyOrderAndLines` events fire after import. *Updated: 2026-04-08 -- Staff member gating moved from B2B Enabled to Advanced Shopify Plan* +*Updated: 2026-07-29 -- Order line import page size raised to 10* + ### Mapping Before a sales document can be created, the order must be mapped to BC entities. `ShpfyOrderMapping.Codeunit.al` orchestrates this. The mapping proceeds in sequence: customer mapping, then shipment method, shipping agent, and payment method. Each step has Before/After events that allow extensions to override or supplement the logic. Customer mapping is the most involved step. For B2B orders (where `"Company Id"` is set), the company mapping strategy runs instead. For D2C orders, the `ICustomerMapping` interface dispatches based on the Shop's `"Customer Mapping Type"` enum -- options include By Email/Phone, By Bill-to Info, and Default Customer. If the customer cannot be found and `"Auto Create Unknown Customers"` is enabled, a new BC customer is created from the template. +Shipment method discovery adapts to Shopify's market-driven shipping model. If the shop feature flag `marketDrivenShipping` is true, `ShpfyShippingMethods` reads active option definitions from Markets instead of delivery profiles. Inherited market shipping configurations and carrier-calculated options without static names are skipped because they do not create stable mapping names. + +*Updated: 2026-07-29 -- Shipping method discovery now supports market-driven shipping* + ### Sales document creation `ShpfyProcessOrder.Codeunit.al` creates the BC sales document. It first confirms mapping succeeded, then creates the Sales Header, Sales Lines, and applies global discounts. @@ -93,7 +109,9 @@ The header creation in `CreateHeaderFromShopifyOrder()` is notably manual -- it The `"Create Invoices From Orders"` setting causes fully-fulfilled orders to be created as Sales Invoices instead of Sales Orders. The `"Use Shopify Order No."` setting uses the Shopify order number (e.g., "#1001") as the BC document number, which requires the number series to have Manual Nos. enabled. -Special line types require attention: gift card purchases map to the `"Sold Gift Card Account"` G/L account, tips map to `"Tip Account"`, and shipping charges map to `"Shipping Charges Account"`. These are all configured on the Shop record. +Special line types require attention: gift card purchases map to the `"Sold Gift Card Account"` G/L account, tips map to `"Tip Account"`, and shipping charges map to `"Shipping Charges Account"`. These are all configured on the Shop record. Shipping charges are staged as their own records before sales line creation, and their Shopify tax lines are imported as `Shpfy Order Tax Line` rows parented by the shipping line ID. + +*Updated: 2026-07-29 -- Shipping charge tax lines are now retained during order import* The Shopify Order page includes contact lookup and validation, allowing users to verify and resolve customer contact information directly from the order. @@ -123,6 +141,10 @@ The `IDocumentSource` interface determines which Shopify document (return or ref Refund lines carry a `"Restock Type"` that affects inventory: Return means items go back to stock, Cancel means they were never shipped, NoRestock is purely financial. Lines with non-restock types are mapped to G/L accounts (`"Refund Acc. non-restock Items"`) instead of item lines on the credit memo. +For return-with-exchange cases, order import fetches exchange line items from the order's returns and flags the matching Shopify order lines as exchange items. Sales document creation excludes those order lines from the original invoice. Refund import then fetches the refund's return exchange lines and creates synthetic negative-quantity refund lines, so the credit memo offsets the exchange item without adding a balancing refund-account line. + +*Updated: 2026-07-29 -- Refund-with-exchange handling now uses exchange GraphQL queries* + ## Inventory synchronization Inventory sync pushes stock levels from BC to Shopify. It is always one-directional (BC to Shopify). `ShpfySyncInventory.Codeunit.al` orchestrates the flow. diff --git a/src/Apps/W1/Shopify/App/docs/data-model.md b/src/Apps/W1/Shopify/App/docs/data-model.md index ac27ca4b27d..b811df70464 100644 --- a/src/Apps/W1/Shopify/App/docs/data-model.md +++ b/src/Apps/W1/Shopify/App/docs/data-model.md @@ -28,7 +28,9 @@ The Shop's `GetEmptySyncTime()` returns `2004-01-01` as a sentinel for "never sy A Shopify product is a parent container holding one or more variants. In BC terms, a Product maps to an Item, and Variants map to Item Variants. The Product table stores the Shopify product ID (BigInteger), description, title, status, and crucially three hash fields: `"Image Hash"`, `"Tags Hash"`, and `"Description Html Hash"`. These enable the export flow to skip API calls when nothing has changed. -Each Variant belongs to a Product (via `"Product Id"`) and can map to both an Item and an Item Variant via their respective SystemIds. Variants carry the pricing fields (Price, Compare at Price, Unit Cost) and up to three option name/value pairs that represent Shopify's variant options (size, color, etc.). +Each Variant belongs to a Product (via `"Product Id"`) and can map to both an Item and an Item Variant via their respective SystemIds. Variants carry the pricing fields (Price, Compare at Price, Unit Cost) and up to three option name/value pairs that represent Shopify's variant options (size, color, etc.). Variants also store the customs fields that Shopify exposes on the variant's inventory item: HS code as `"Tariff No."` and origin country as `"Country/Region of Origin Code"`. + +*Updated: 2026-07-29 -- Variant customs fields now participate in product sync* Inventory Items represent the inventory-trackable entries for each variant at each Shopify location. Shop Inventory combines the location mapping with the calculated stock level. @@ -46,6 +48,10 @@ The `"Item SystemId"` field on both Product and Variant is a GUID linking to BC' The HTML description is stored in a Blob field (`"Description as HTML"`) because AL's Text fields cap at 2048 characters and Shopify product descriptions can be substantially longer. The hash is computed when the description is set (via `SetDescriptionHtml()`) and compared during export to avoid unnecessary API updates. +The Shop's `"Sync HS Code and Country"` setting controls whether BC item tariff and country values are copied to Shopify variants, and whether imported Shopify inventory-item values are applied back to BC items. Existing-item updates only apply values that resolve to BC Tariff Number or Country/Region records, so a blank or unmapped Shopify value does not wipe an existing BC value. + +*Updated: 2026-07-29 -- HS code and country of origin sync now works in both directions* + ## Order lifecycle Orders flow through a staging pipeline: first they appear in Orders to Import (lightweight records with just the Shopify order ID and basic metadata), then they are imported into Order Header and Order Lines, and finally processed into BC Sales Orders or Sales Invoices. @@ -54,14 +60,21 @@ The Order Header is one of the largest tables in the connector. It stores comple Order Lines carry the item-level detail including the Shopify variant ID, mapped BC item number, quantities, and dual-currency amounts. Special line types include tips and gift cards, identified by boolean flags. -Order Attributes and Order Line Attributes store Shopify's custom attributes (key-value pairs that merchants can attach to orders). Order Tax Lines track per-tax-rate amounts, including a `"Channel Liable"` flag for marketplace tax collection. +Order Shipping Charges store Shopify shipping lines separately from item lines so the connector can preserve their title, source, discount, and dual-currency amounts before projecting them to BC sales lines. Their tax lines now use the same `Shpfy Order Tax Line` table as order and item-line taxes, parented by the Shopify shipping line ID. + +Order Attributes and Order Line Attributes store Shopify's custom attributes (key-value pairs that merchants can attach to orders). Order Tax Lines track per-tax-rate amounts, including a `"Channel Liable"` flag for marketplace tax collection. The `"Parent Id"` can point at an order, an order line, or a shipping charge; currency formatting resolves through whichever parent exists. + +*Updated: 2026-07-29 -- Shipping charges now retain their linked tax lines* ```mermaid erDiagram ORDERS_TO_IMPORT ||--|| ORDER_HEADER : imports_to ORDER_HEADER ||--o{ ORDER_LINE : contains + ORDER_HEADER ||--o{ ORDER_SHIPPING_CHARGES : charges ORDER_HEADER ||--o{ ORDER_ATTRIBUTE : has ORDER_HEADER ||--o{ ORDER_TAX_LINE : taxed_by + ORDER_LINE ||--o{ ORDER_TAX_LINE : taxed_by + ORDER_SHIPPING_CHARGES ||--o{ ORDER_TAX_LINE : taxed_by ``` The dual currency design deserves attention. Every monetary field on the Order Header exists twice: once in the shop's base currency (`"Total Amount"`, `"Discount Amount"`, etc.) and once in the presentment currency (`"Presentment Total Amount"`, `"Presentment Discount Amount"`, etc.). The Shop's `"Currency Handling"` setting controls which set of amounts is used when creating BC sales documents. The `"Processed Currency Handling"` field on the order captures which mode was actually used, so re-processing uses the same logic. diff --git a/src/Apps/W1/Shopify/App/docs/extensibility.md b/src/Apps/W1/Shopify/App/docs/extensibility.md index b60fb465ea2..176d9617e9a 100644 --- a/src/Apps/W1/Shopify/App/docs/extensibility.md +++ b/src/Apps/W1/Shopify/App/docs/extensibility.md @@ -18,7 +18,9 @@ The order processing flow in `ShpfyProcessOrder.Codeunit.al` and `ShpfyOrderMapp **Sales header creation** -- `OnBeforeCreateSalesHeader` gives you the Shopify Order Header and lets you set `IsHandled` to completely replace header creation logic. `OnAfterCreateSalesHeader` fires after the header is inserted and validated, giving you the Sales Header to modify. -**Sales line creation** -- `OnBeforeCreateSalesLine` and `OnAfterCreateSalesLine` in `ShpfyProcessOrder.Codeunit.al` let you intercept line creation. This is where you would add custom line types, modify quantities, or insert additional lines. +**Sales line creation** -- `OnBeforeCreateSalesLine` and `OnAfterCreateSalesLine` in `ShpfyProcessOrder.Codeunit.al` let you intercept line creation. This is where you would add custom line types, modify quantities, or insert additional lines. Shipping charge sales lines have separate hooks, `OnBeforeCreateShippingCostSalesLine` and `OnAfterCreateShippingCostSalesLine`, because they are staged as `Shpfy Order Shipping Charges` records before they become BC sales lines. + +*Updated: 2026-07-29 -- Shipping charge sales line events are separate from item line events* **Customer mapping override** -- `OnBeforeMapCustomer` on `ShpfyOrderEvents.Codeunit.al` fires before the customer mapping strategy runs. Set `Handled := true` and populate `"Sell-to Customer No."` on the Order Header to bypass the standard mapping entirely. `OnAfterMapCustomer` lets you adjust the result. @@ -40,7 +42,9 @@ Product events live in `ShpfyProductEvents.Codeunit.al` and cover both import an **Price calculation** -- Events in `ShpfyProductPriceCalc.Codeunit.al` let you override how prices and compare-at prices are calculated during export. -**Product status on creation** -- The `"Status for Created Products"` enum uses the `ICreateProductStatusValue` interface. Built-in values are Active and Draft. Extend the enum and implement the interface to add new initial statuses. +**Product status on creation** -- The `"Status for Created Products"` enum uses the `ICreateProductStatusValue` interface. Built-in values are Active, Draft, and Unlisted. The enum is not extensible in the current app, so these are the supported creation statuses exposed by the base connector. + +*Updated: 2026-07-29 -- Unlisted added and enum extensibility verified* **Action for removed products** -- The `"Action for Removed Products"` enum uses `IRemoveProductAction`. Extend to control what happens to the Shopify product when the linked BC item is blocked or the product is deleted locally. diff --git a/src/Apps/W1/Shopify/App/docs/patterns.md b/src/Apps/W1/Shopify/App/docs/patterns.md index 42edc27ae7d..a013fbbd52e 100644 --- a/src/Apps/W1/Shopify/App/docs/patterns.md +++ b/src/Apps/W1/Shopify/App/docs/patterns.md @@ -29,14 +29,16 @@ Queries are stored as `.graphql` resource files under `.resources/graphql/{Area} The `ShpfyGraphQLQueries.Codeunit.al` dispatcher receives a `"Shpfy GraphQL Type"` enum value and a `Dictionary of [Text, Text]` of parameters, loads the query via `NavApp.GetResourceAsText()`, substitutes parameters into the query string, and returns the final query along with the expected cost. -The `"Shpfy GraphQL Type"` enum is `Extensible = false` and uses `{Area}_{QueryName}` naming for its values (e.g., `Products_GetProductById`, `Customers_NextCustomerIds`). There are 143 resource files covering all query types. +The `"Shpfy GraphQL Type"` enum is `Extensible = false` and uses `{Area}_{QueryName}` naming for its values (e.g., `Products_GetProductById`, `Customers_NextCustomerIds`). There are 149 resource files covering all query types, including market-driven shipping method queries and exchange-line queries for orders and refunds. + +*Updated: 2026-07-29 -- GraphQL resource count and new query families updated* Adding a new query requires two steps: 1. Create a `.graphql` file in the appropriate `.resources/graphql/{Area}/` folder 2. Add a corresponding enum value to `"Shpfy GraphQL Type"` -The `ShpfyCommunicationMgt.Codeunit.al` is the single entry point for all API calls. Its `ExecuteGraphQL()` overloads accept either a `"Shpfy GraphQL Type"` (type-safe) or a raw query string (for ad-hoc queries). Before executing, it calls `WaitForRequestAvailable()` on the rate limiter with the expected cost. This layer and the rate limiter are unchanged. +The `ShpfyCommunicationMgt.Codeunit.al` is the single entry point for all API calls. Its `ExecuteGraphQL()` overloads accept either a `"Shpfy GraphQL Type"` (type-safe) or a raw query string (for ad-hoc queries). Before executing, it calls `WaitForRequestAvailable()` on the rate limiter with the expected cost. The communication entry point remains the central place for these calls; rate-limit behavior is described below. The old `IGraphQL` interface pattern (where each query was a codeunit implementing `GetGraphQL()` and `GetExpectedCost()`) was removed in this refactoring. 14 obsolete stub codeunits remain with `ObsoleteState = Pending; ObsoleteTag = '29.0'` for backward compatibility and will be cleaned up in the CLEAN29 cycle. @@ -68,9 +70,11 @@ After each API call, `SetQueryCost()` reads the `restoreRate` and `currentlyAvai WaitTime := (Max(ExpectedCost - LastAvailable, 0) / RestoreRate * 1000) - (CurrentDateTime - LastRequestedOn) ``` -This formula accounts for the time elapsed since the last request (during which budget has been restoring at `RestoreRate` points per second). If `RestoreRate` is zero (no data yet), it defaults to 50 -- Shopify's standard restore rate. +This formula accounts for the time elapsed since the last request (during which budget has been restoring at `RestoreRate` points per second). If `RestoreRate` is zero (no data yet), it defaults to 50 -- Shopify's standard restore rate. The calculated duration is now scheduled from `CurrentDateTime`, not from `LastRequestedOn`, so elapsed time is not subtracted twice and the connector does not wake up too early under sustained throttling. + +The `GoToSleep()` method uses AL's `Sleep()` function with a TryFunction wrapper. The 100ms fallback is reserved for unexpected sleep failures; normal "no wait needed" cases return without sleeping. -The `GoToSleep()` method uses AL's `Sleep()` function with a TryFunction wrapper. If the calculated sleep time causes an error (e.g., negative duration), it falls back to a 100ms sleep. +*Updated: 2026-07-29 -- Rate limiter now waits from the current time after elapsed-time adjustment* The API version is hardcoded as `'2026-07'` in `ShpfyCommunicationMgt.Codeunit.al`. The connector validates the API version expiry date and shows warnings/errors when it is approaching or has passed expiry, pulling the date from Azure Key Vault. @@ -107,7 +111,11 @@ The flow is: The `IBulkOperation` interface requires `RevertFailedRequests()` and `RevertAllRequests()` methods because the bulk operation is async -- by the time results arrive, the original transaction is long committed. If the bulk operation fails entirely, all changes must be reverted. If it partially succeeds, only the failed entries are reverted. -The `ShpfyBulkOperation.Table.al` tracks the state of each bulk operation: type, status (Created, Running, Completed, Failed), shop code, and the request/response data. +The `ShpfyBulkOperation.Table.al` tracks the state of each bulk operation: type, status (Created, Running, Completed, Failed), shop code, and the request/response data. Price updates store the prior variant state as request data so failed lines can be reverted later. When the Shop logging mode is All, the table also stores the exact JSONL sent to Shopify, and the Bulk Operations page exposes it as a download. + +Bulk price update failures are treated as skipped records, not silent reversions. `RevertFailedRequests()` parses Shopify's JSONL result, identifies the successful variant IDs, reverts the rest, and logs the Shopify user error against the linked BC item variant or item where possible. + +*Updated: 2026-07-29 -- Bulk price sync now records sent JSONL and logs failed lines as skipped records* ## Shop-scoped multi-tenancy diff --git a/src/Apps/W1/Shopify/App/src/Base/docs/CLAUDE.md b/src/Apps/W1/Shopify/App/src/Base/docs/CLAUDE.md index c725bde0bb0..be180423c90 100644 --- a/src/Apps/W1/Shopify/App/src/Base/docs/CLAUDE.md +++ b/src/Apps/W1/Shopify/App/src/Base/docs/CLAUDE.md @@ -26,8 +26,13 @@ splitting between background-allowed and foreground-only shops. `Shpfy Installer` sets up retention policies and Cue thresholds on install, and disables all shops on company copy or environment cleanup. +`Shpfy Shop Mgt.` handles user-facing notifications, including the +Belgian localization prompt shown from the Shops list when the tenant is +in the BE application family and the localization extension is missing. -## B2B Enabled → Advanced Shopify Plan migration +*Updated: 2026-07-29 -- Shop Mgt. now owns the Belgian localization prompt* + +## B2B enabled to Advanced Shopify Plan migration The `B2B Enabled` field (117) on the Shop table is obsoleted with `CLEAN29`/`CLEANSCHEMA32` guards. It is replaced by a new field @@ -37,11 +42,15 @@ Development, and Advanced plans. *Updated: 2026-04-08 -- B2B Enabled obsoleted, replaced by Advanced Shopify Plan* `GetShopSettings()` still queries the Shopify plan info but now sets -`Advanced Shopify Plan` instead of `B2B Enabled`. The plan name check -includes "Advanced" in addition to the previous Plus/Development values. +`Advanced Shopify Plan` instead of `B2B Enabled`. It validates the field, +so dependent settings react to plan changes instead of silently keeping +stale values. The plan name check includes "Advanced" in addition to the +previous Plus/Development values. + +*Updated: 2026-07-29 -- plan refresh now runs validation for dependent settings* `ShpfyUpgradeMgt` has a new `HasAdvancedShopifyPlanUpgrade()` procedure -that uses DataTransfer to copy `B2B Enabled` → `Advanced Shopify Plan` +that uses DataTransfer to copy `B2B Enabled` to `Advanced Shopify Plan` (with a source filter on true and a constant value). ## Activities page and Shop Card visibility changes @@ -56,10 +65,42 @@ groups/actions have been removed. The StaffMembers action is now gated on `Advanced Shopify Plan` instead. The "Sync All" action now unconditionally syncs companies and catalog prices regardless of plan. +The `Auto Create Catalog` field is captioned as `Auto Create B2B Catalog` +and is visible in the B2B company synchronization group. Enabling it now +validates the cached `Advanced Shopify Plan` flag and errors unless the +shop is Plus, Plus Trial, Development, or Advanced. If a later plan +refresh clears `Advanced Shopify Plan`, the table validation turns +`Auto Create Catalog` off. + +*Updated: 2026-07-29 -- Auto Create B2B Catalog is visible but plan-validated* + +## Shop settings added for product sync + +`Find Mapping by Barcode` controls the product mapping fallback that uses +item references by barcode after SKU and variant matching fail. It +defaults to true to preserve the legacy fallback, but it can be disabled +when duplicate or unreliable Shopify barcodes would create wrong links. + +`Sync HS Code and Country` controls whether tariff numbers and country of +origin flow between Business Central items and Shopify inventory items. +When importing to BC, the code only applies values that resolve to an +existing Tariff Number or Country/Region; `Shpfy Filter Mgt.` normalizes +tariff comparisons to digits so `6104.43` and `610443` are treated as the +same HS code. + +*Updated: 2026-07-29 -- barcode fallback and HS code/country sync settings added* + ## Things to know - The `Shpfy Cue` table uses FlowFields for role center counts: unmapped customers/products/companies, unprocessed orders/shipments, sync errors. - Empty sync time sentinel is `2004-01-01` (`GetEmptySyncTime()`), not `0DT`. -- Three page extensions embed Shopify Activities into standard role centers. +- Three page extensions embed Shopify Activities into standard role centers + and add Shops, Customers, Companies, Products, Orders, Refunds, Returns, + Gift Cards, Transactions, and Payouts to the Shopify navigation group. - `ShpfyConnectorGuide` and `ShpfyInitialImport` provide first-time setup. +- Order sync refreshes and saves `GetShopSettings()` before retrieving + orders, so plan-gated order details such as staff member fields use the + current Shopify plan instead of a stale cached value. + +*Updated: 2026-07-29 -- role center navigation and order plan refresh notes added* diff --git a/src/Apps/W1/Shopify/App/src/Bulk Operations/docs/CLAUDE.md b/src/Apps/W1/Shopify/App/src/Bulk Operations/docs/CLAUDE.md index ad14830a4c1..2bddab6b76c 100644 --- a/src/Apps/W1/Shopify/App/src/Bulk Operations/docs/CLAUDE.md +++ b/src/Apps/W1/Shopify/App/src/Bulk Operations/docs/CLAUDE.md @@ -6,14 +6,20 @@ Provides an async GraphQL bulk mutation framework for high-volume updates to Sho The flow is: build JSONL input, upload it to a staged URL, submit the bulk mutation referencing that URL, then wait for a webhook callback. `ShpfyBulkOperationMgt.SendBulkMutation` orchestrates this -- it checks that no bulk operation of the same type is already running, calls `ShpfyBulkOperationAPI.CreateBulkOperationMutation` (which creates the upload URL, POSTs the JSONL as multipart/form-data, then sends the GraphQL mutation), and records the operation in the `Shpfy Bulk Operation` table. When Shopify completes the operation, it fires a webhook that calls `ProcessBulkOperationNotification`, which fetches the final status, result URL, and any error codes. -Each concrete operation implements the `Shpfy IBulk Operation` interface, which provides the GraphQL mutation template, JSONL input template, and revert logic. The `OnModify` trigger on the `Shpfy Bulk Operation` table automatically calls `RevertFailedRequests` on completion or `RevertAllRequests` on cancellation/failure -- this restores local records that were optimistically updated before the bulk call. +Each concrete operation implements the `Shpfy IBulk Operation` interface, which provides the GraphQL mutation template, JSONL input template, and revert logic. The `OnModify` trigger on the `Shpfy Bulk Operation` table automatically calls `RevertFailedRequests` on completion or `RevertAllRequests` on cancellation/failure -- this restores local records that were optimistically updated before the bulk call and, for price updates, logs skipped records for the variants Shopify did not confirm. -Two implementations exist today: `ShpfyBulkUpdateProductImage` (updates product media) and `ShpfyBulkUpdateProductPrice` (updates variant prices and costs). The threshold for switching from individual calls to bulk is 100 items. +Two implementations exist today: `ShpfyBulkUpdateProductImage` (updates product media) and `ShpfyBulkUpdateProductPrice` (updates variant prices and costs). Price sync switches to bulk only when there are at least 100 changed variant price requests, not merely 100 variants considered during the sync. + +*Updated: 2026-07-29 -- price bulk threshold, diagnostics, and revert behavior +updated* ## Things to know - Only one bulk operation per type (mutation/query) can run at a time per shop. The module polls the current operation status before starting a new one. - Enabling bulk operations requires a BC-licensed user and registers a webhook subscription on the shop. - The `Shpfy Bulk Operation` table stores request data as a JSONL blob so revert logic can compare against Shopify's JSONL result to identify which items succeeded vs. failed. -- `RevertFailedRequests` for price updates parses the JSONL result line by line, collects successful variant IDs, and reverts only the records that were not in the success list. +- When the shop logging mode is `All`, the exact JSONL sent to Shopify is also stored on the bulk operation and can be downloaded from the Shopify Bulk Operations page. +- `RevertFailedRequests` for price updates parses the JSONL result line by line, collects successful variant IDs, logs skipped records with Shopify error text when available, and reverts only the records that were not in the success list. +- Revert logic tolerates older request data that does not contain `unitCost`; it restores unit cost only when that key is present. +- Price update JSONL sends `compareAtPrice` only when it changed. If the new compare-at price is not greater than the price, it sends `null` instead of `"0"` so Shopify clears the value rather than storing a zero compare-at price. - Adding a new bulk operation type means implementing `Shpfy IBulk Operation` and adding an enum value to `Shpfy Bulk Operation Type`. diff --git a/src/Apps/W1/Shopify/App/src/Catalogs/docs/CLAUDE.md b/src/Apps/W1/Shopify/App/src/Catalogs/docs/CLAUDE.md index a6771819b04..b72484aba72 100644 --- a/src/Apps/W1/Shopify/App/src/Catalogs/docs/CLAUDE.md +++ b/src/Apps/W1/Shopify/App/src/Catalogs/docs/CLAUDE.md @@ -10,11 +10,26 @@ Price sync is driven by `ShpfySyncCatalogPrices`. For each catalog with `Sync Pr The `Shpfy Catalog` table carries full pricing context: Customer Price Group, Customer Discount Group, Gen. Bus. Posting Group, VAT settings, and an optional Customer No. that overrides catalog-level settings with the customer's own price/discount groups. +*Updated: 2026-07-29 -- B2B catalog page behavior and Sync Prices gotcha +updated* + ## Things to know - `Shpfy Catalog Price` is a temporary table -- it holds the Shopify-side prices just long enough to diff against BC calculations during sync. - If a catalog's currency code does not match what Shopify reports, the sync is skipped and a skipped record is logged. - Company catalogs get their own publication and price list created automatically via separate GraphQL calls. +- The B2B Catalogs page shows the company name as a drilldown. It opens the + related Shopify Company Card through the catalog's `Company SystemId`. - Market catalogs track which Shopify markets they serve through `Shpfy Market Catalog Relation`, which is refreshed on every import. - Price updates are batched at 250 variants per GraphQL call to stay within Shopify limits. +- When a user enables `Sync Prices` on a duplicate company catalog line, the + current row is modified before other lines for the same catalog are disabled. + This preserves the user's current selection instead of losing it during the + `ModifyAll`. +- The shop setting is now captioned `Auto Create B2B Catalog`. The field is + visible on the shop card, but enabling it validates that the shop is on a + Shopify Plus, Plus Trial, Development, or Advanced plan. +- The company catalog import query filters active catalogs by `company_id` and + reads only the catalog id, title, and price-list currency. Companies must be + imported first or `Get Catalogs` has no company context to query. - The catalog URL construction differs between unified and non-unified Shopify markets. diff --git a/src/Apps/W1/Shopify/App/src/Companies/docs/CLAUDE.md b/src/Apps/W1/Shopify/App/src/Companies/docs/CLAUDE.md index fde04336e9c..c33b85f5d7d 100644 --- a/src/Apps/W1/Shopify/App/src/Companies/docs/CLAUDE.md +++ b/src/Apps/W1/Shopify/App/src/Companies/docs/CLAUDE.md @@ -21,6 +21,9 @@ mapping resolves through the `ICompanyMapping` interface, selected by the Shop's `ShpfyCreateCustomer.CreateCustomerFromCompany` builds a BC Customer from the company's location data (address, phone, tax ID, payment terms). +*Updated: 2026-07-29 -- company export now resolves ISO country codes before +tax-area lookup, and tax ID mapping is explicitly localization-friendly* + ## Things to know - B2B features (companies, catalogs) are available on all Shopify plans. @@ -45,8 +48,14 @@ company's location data (address, phone, tax ID, payment terms). The mapping codeunit checks at runtime whether the selected implementation supports `IFindCompanyMapping` and falls back to `CompByEmailPhone` if not. - Tax registration ID mapping is itself pluggable via - `Shpfy Tax Registration Id Mapping` interface with two implementations: - `ShpfyTaxRegistrationNo` (matches Registration No.) and - `ShpfyVATRegistrationNo` (matches VAT Registration No.). + `Shpfy Tax Registration Id Mapping` interface. The base app includes + `ShpfyTaxRegistrationNo` (matches Registration Number) and + `ShpfyVATRegistrationNo` (matches VAT Registration No.); localizations can + add enum values that map the Shopify tax ID to a country-specific customer + field, such as the Belgian Enterprise No. +- Company export resolves the BC Country/Region to its ISO code before setting + `Shpfy Company Location`.`Country/Region Code` and before filtering + `Shpfy Tax Area`. This matters for countries where the BC code differs from + Shopify's ISO 3166-1 alpha-2 code. - When creating customers from companies, the county is resolved through the same `ICounty` interface used by the Customers module. diff --git a/src/Apps/W1/Shopify/App/src/Companies/docs/extensibility.md b/src/Apps/W1/Shopify/App/src/Companies/docs/extensibility.md index c6d4682beca..6c8dc59d36c 100644 --- a/src/Apps/W1/Shopify/App/src/Companies/docs/extensibility.md +++ b/src/Apps/W1/Shopify/App/src/Companies/docs/extensibility.md @@ -31,10 +31,21 @@ are matched and updated on BC Customers. Three methods: - `UpdateTaxRegistrationId` -- writes a new tax ID to a BC Customer during company creation -Two implementations: `ShpfyTaxRegistrationNo` (uses Customer."Registration No." -from the Registration No. extension) and `ShpfyVATRegistrationNo` (uses -Customer."VAT Registration No."). Selected via the "Shpfy Comp. Tax Id Mapping" -enum on the Shop. +The base app provides `ShpfyTaxRegistrationNo` (uses +Customer."Registration Number") and `ShpfyVATRegistrationNo` (uses +Customer."VAT Registration No."). Selected via the extensible +"Shpfy Comp. Tax Id Mapping" enum on the Shop. + +*Updated: 2026-07-29 -- tax ID mapping can be extended by localizations such +as the Belgian Enterprise No. mapping* + +Add another country-specific tax ID strategy by extending +`Shpfy Comp. Tax Id Mapping` and implementing +`Shpfy Tax Registration Id Mapping`. The same implementation is used when +matching a Shopify Company Location to an existing BC Customer, creating or +updating the BC Customer from Shopify, and exporting a BC Customer's tax ID +back to the Shopify Company Location. This keeps localization-specific fields +out of the company import/export code. ## Customizing company import @@ -50,5 +61,10 @@ customize company-to-customer creation, implement a custom `ShpfyCompanyExport` pushes BC Customer data to existing Shopify companies. There are no company-specific events -- the export uses the same field-diff -pattern as customer export. Customization is through the interfaces above -or by modifying the company/location data after API retrieval. +pattern as customer export. It resolves `Country/Region`.`ISO Code` before +filtering `Shpfy Tax Area`, so custom county or tax-area data must use +Shopify's ISO country code. Customization is through the interfaces above or +by modifying the company/location data after API retrieval. + +*Updated: 2026-07-29 -- company export tax-area filtering now uses ISO country +codes* diff --git a/src/Apps/W1/Shopify/App/src/Customers/docs/CLAUDE.md b/src/Apps/W1/Shopify/App/src/Customers/docs/CLAUDE.md index 08bf9e44348..8f3b3ad8179 100644 --- a/src/Apps/W1/Shopify/App/src/Customers/docs/CLAUDE.md +++ b/src/Apps/W1/Shopify/App/src/Customers/docs/CLAUDE.md @@ -24,6 +24,9 @@ different orderings (FirstLast, LastFirst, CompanyName, Empty). County resolutio uses `ICounty` (for records) and `ICountyFromJson` (for API responses) to convert between province codes and names. +*Updated: 2026-07-29 -- customer creation and export country-code behavior +updated* + ## Things to know - `Customer SystemId` on `Shpfy Customer` links to the BC Customer, same @@ -34,9 +37,15 @@ convert between province codes and names. - Customer creation uses country-specific templates. `ShpfyCreateCustomer` looks up the `Shpfy Customer Template` table by Shop Code + Country Code, falling back to the Shop's default template. +- When order-time by-email/phone mapping is allowed to create a missing BC + Customer, it prefers the Shopify default address but falls back to the first + available address. A Shopify customer with addresses but no default address + no longer blocks customer creation. - Export (`ShpfyCustomerExport`) splits the BC Customer name back into first/last using the Shop's "Name Source" config. It handles multi-email fields by taking the first email before any semicolon or comma. +- Export resolves the BC Country/Region to its ISO code before setting the + Shopify customer address country and before filtering `Shpfy Tax Area`. - Phone number matching strips all non-digit characters and builds a wildcard filter (`*1*2*3*...`) to handle format differences. - The `Shpfy Customer Address` table stores Shopify addresses. Addresses diff --git a/src/Apps/W1/Shopify/App/src/Customers/docs/business-logic.md b/src/Apps/W1/Shopify/App/src/Customers/docs/business-logic.md index 4900f0d3730..1a04e43b013 100644 --- a/src/Apps/W1/Shopify/App/src/Customers/docs/business-logic.md +++ b/src/Apps/W1/Shopify/App/src/Customers/docs/business-logic.md @@ -13,11 +13,14 @@ flowchart TD D -- yes --> E[ShpfyUpdateCustomer] D -- no --> F[Done] C -- no --> G{Auto Create Unknown Customers?} - G -- yes --> H[Find default address] + G -- yes --> H[Find default address or first address] H --> I[ShpfyCreateCustomer] G -- no --> J[Skip] ``` +*Updated: 2026-07-29 -- customer creation and update fall back to a +non-default address when Shopify has no default address* + `FindMapping` in `ShpfyCustomerMapping` first checks whether the Shopify Customer already has a `Customer SystemId` that resolves to a valid BC Customer. If the linked customer was deleted, it clears the stale SystemId and proceeds with @@ -36,6 +39,12 @@ template exists, a new empty row is inserted (for future configuration) and the Shop's default template is used. The `OnBeforeFindCustomerTemplate` event can override this entirely. +For order-time by-email/phone mapping, `ShpfyCustByEmailPhone` now uses the +default Shopify address when one exists and otherwise uses the first address for +that Shopify customer. `ShpfyUpdateCustomer` uses the same fallback when +updating an existing BC Customer. If there are no addresses at all, creation +still exits without a BC Customer and update raises an error. + ## Export flow `ShpfyCustomerExport` iterates BC Customers matching the report filters. @@ -60,7 +69,12 @@ and Customer Address records. Name splitting respects three configurable sources (Name, Name 2, Contact) and each source can be parsed as FirstAndLastName, LastAndFirstName, or CompanyName. County mapping uses the `Shpfy Tax Area` table to resolve between province codes and names based on the Shop's "County Source" -setting (Code or Name). +setting (Code or Name). The Tax Area lookup uses the ISO code from +`Country/Region`.`ISO Code`, not the BC country/region code itself, and the +same ISO code is written to the Shopify customer address. + +*Updated: 2026-07-29 -- customer export tax-area filtering now uses ISO +country codes* The export uses a diff check (`HasDiff`) comparing every field of the customer and address records before and after filling. Only when something changed does diff --git a/src/Apps/W1/Shopify/App/src/Customers/docs/extensibility.md b/src/Apps/W1/Shopify/App/src/Customers/docs/extensibility.md index a2d9b0d570b..7d360448309 100644 --- a/src/Apps/W1/Shopify/App/src/Customers/docs/extensibility.md +++ b/src/Apps/W1/Shopify/App/src/Customers/docs/extensibility.md @@ -14,6 +14,15 @@ interfaces handle mapping strategy, name formatting, and county resolution. which Customer Template is selected for a given country code. OnBefore can set Handled to bypass the default template lookup. +*Updated: 2026-07-29 -- creation hooks may receive a fallback address when no +default Shopify address exists* + +When customer creation is triggered by order-time by-email/phone mapping, the +Customer Address passed into these hooks is the Shopify default address when +available, otherwise the first address for that Shopify customer. Extensions +that require the Shopify default address should check the address `Default` +flag themselves. + ## Customizing customer mapping - `OnBeforeFindMapping` -- fires before the default email/phone matching in @@ -55,6 +64,13 @@ enum on the Shop. BC County string. Two implementations: `ShpfyCountyCode` (province code) and `ShpfyCountyName` (province name). Selected via the "County Source" enum. +Customer and company export now filter `Shpfy Tax Area` with Shopify's ISO +country code before applying the county source. Custom tax-area records used by +export should therefore be keyed by ISO code. + +*Updated: 2026-07-29 -- tax-area filtering during export uses ISO country +codes* + **ICountyFromJson** -- same as ICounty but operates on a raw JSON address object during API response parsing. Implementations: `ShpfyCountyFromJsonCode` and `ShpfyCountyFromJsonName`. diff --git a/src/Apps/W1/Shopify/App/src/GraphQL/docs/CLAUDE.md b/src/Apps/W1/Shopify/App/src/GraphQL/docs/CLAUDE.md index 69340978224..98960ff86ad 100644 --- a/src/Apps/W1/Shopify/App/src/GraphQL/docs/CLAUDE.md +++ b/src/Apps/W1/Shopify/App/src/GraphQL/docs/CLAUDE.md @@ -11,7 +11,7 @@ Each API operation lives in a `.graphql` resource file under `.resources/graphql/{Area}/{QueryName}.graphql`. Line 1 is a `# cost: N` comment declaring the expected cost points for rate limiting; lines 2+ contain the JSON-encoded GraphQL request body. The enum -`Shpfy GraphQL Type` (`Enums/ShpfyGraphQLType.Enum.al`) maps 143 values +`Shpfy GraphQL Type` (`Enums/ShpfyGraphQLType.Enum.al`) maps 149 values to their resource files -- it is a plain enum (`Extensible = false`) with no `implements` clause. @@ -24,12 +24,16 @@ literal string replacement -- no escaping, no StrSubstNo. Callers format values correctly within the query template itself. Rate limiting is handled by `Shpfy GraphQL Rate Limit`, a SingleInstance -codeunit that tracks Shopify's `restoreRate` and `currentlyAvailable` -from throttle status and sleeps before requests that would exceed budget. +codeunit that tracks Shopify's `restoreRate`, `currentlyAvailable`, and +the time of the last throttle response. Before sleeping it credits the +points restored since that response, then waits only for the remaining +deficit. + +*Updated: 2026-07-29 -- rate limiter now credits elapsed restore time before sleeping* ## Things to know -- 143 `.graphql` resource files replace the former query-builder +- 149 `.graphql` resource files replace the former query-builder codeunits. Only `ShpfyGraphQLQueries` (dispatcher) and `ShpfyGraphQLRateLimit` remain as codeunits in this folder. - The enum is `Extensible = false`. The dispatcher fires no events -- @@ -37,6 +41,10 @@ from throttle status and sleeps before requests that would exceed budget. - Enum values follow `{Area}_{QueryName}` naming (e.g., `Customers_GetCustomerIds`). Resource files live at `.resources/graphql/{Area}/{QueryName}.graphql`. +- New query pairs for market-driven shipping and exchange lines keep the + same convention, including `Shipping_GetMarketShippingMethods`, + `Orders_GetOrderExchangeLineItems`, and `Refunds_GetRefundExchangeLines` + with matching `GetNext...` values for pagination. - 14 obsolete stub codeunits exist for previously-public GQL codeunits (ObsoleteState = Pending). They delegate to the new dispatcher and will be removed after the CLEAN29 transition. @@ -46,3 +54,5 @@ from throttle status and sleeps before requests that would exceed budget. `Customers_GetCustomerIds` / `Customers_GetNextCustomerIds`). - Cost values are hand-tuned integers, not computed. The API version is pinned in `ShpfyCommunicationMgt` (currently `2026-07`), not here. + +*Updated: 2026-07-29 -- query count and new paged query pairs refreshed* diff --git a/src/Apps/W1/Shopify/App/src/GraphQL/docs/patterns.md b/src/Apps/W1/Shopify/App/src/GraphQL/docs/patterns.md index 753936f28f3..01a0d39e5b2 100644 --- a/src/Apps/W1/Shopify/App/src/GraphQL/docs/patterns.md +++ b/src/Apps/W1/Shopify/App/src/GraphQL/docs/patterns.md @@ -15,7 +15,7 @@ That is the entire contract. No codeunit, no interface, no state. ## Enum-to-resource mapping -The `Shpfy GraphQL Type` enum declares 143 values using +The `Shpfy GraphQL Type` enum declares 149 values using `{Area}_{QueryName}` naming (e.g., `Customers_GetCustomerIds`). The enum is `Extensible = false` -- extensions cannot add new query types. @@ -28,6 +28,14 @@ body. Adding a new API operation requires two things: a `.graphql` resource file and a new enum value. +Recent additions for market-driven shipping and exchange-line handling +continue the same convention: `Shipping_GetMarketShippingMethods`, +`Orders_GetOrderExchangeLineItems`, and `Refunds_GetRefundExchangeLines` +each map to a same-named resource file under their area folder, with a +`GetNext...` partner when the Shopify response is paged. + +*Updated: 2026-07-29 -- query enum expanded while preserving the resource naming contract* + ## Parameter substitution Parameters use named `{{placeholder}}` tokens inside the query string. @@ -90,6 +98,22 @@ the restore rate defaults to 50 points/second. There is no automatic cost calculation. If Shopify changes query pricing, the constants need manual adjustment. +## Rate limiter wait calculation + +`ShpfyGraphQLRateLimit` stores Shopify's `currentlyAvailable`, +`restoreRate`, and the time the last throttle response was received. When +the next expected cost would exceed the last known budget, it computes the +missing points, divides by `restoreRate`, and subtracts the elapsed time +since the last response because Shopify has been restoring points during +that interval. Negative results are clamped to zero. + +The wake-up time is based on the current time plus the remaining wait, not +the last response time plus the full restore duration. Otherwise the +elapsed time would be applied twice and the connector could resume before +the missing points are actually restored. + +*Updated: 2026-07-29 -- documented corrected elapsed-time crediting in throttling* + ## Obsolete stub codeunits *Updated: 2026-03-24 -- GraphQL resource file refactoring* diff --git a/src/Apps/W1/Shopify/App/src/Order Refunds/docs/CLAUDE.md b/src/Apps/W1/Shopify/App/src/Order Refunds/docs/CLAUDE.md index 75934750996..a6251671236 100644 --- a/src/Apps/W1/Shopify/App/src/Order Refunds/docs/CLAUDE.md +++ b/src/Apps/W1/Shopify/App/src/Order Refunds/docs/CLAUDE.md @@ -10,15 +10,23 @@ Refund lines are linked back to order lines via `Order Line Id` and carry a `Res If the refund is associated with a return, the module collects return locations from the Returns API to populate the `Location Id` on refund lines, since Shopify does not always include location data directly on the refund. +Return-with-exchange needs a separate import path. Shopify puts the new exchange item on `Refund.return.exchangeLineItems`, not in `Refund.refundLineItems`. `ShpfyRefundsAPI` reads those exchange line items with `Refunds/GetRefundExchangeLines.graphql` and `Refunds/GetNextRefundExchangeLines.graphql`, then creates synthetic `Shpfy Refund Line` rows with a refund line id that is the negated order line id, negative quantity, negative subtotal and tax amounts, `Restock Type = Return`, and `"Is Exchange Item" = true`. Those rows let the credit memo offset the value of the exchange item without relying on the remaining-amount G/L balance line. + +The matching order lines are flagged by order import from `Order.returns.exchangeLineItems` through `Orders/GetOrderExchangeLineItems.graphql`. That keeps exchange items out of the BC sales invoice while the synthetic refund lines put the inventory and refund total back in balance during credit memo creation. + The refund header stores `Shop Code` directly (populated from the communication manager's current shop during import), enabling shop-level filtering without joining through the order header. Currency codes (`Currency Code` and `Presentment Currency Code`) are also stored on the header and translated via `ImportOrder.TranslateCurrencyCode`. Error tracking fields (`Has Processing Error`, `Last Error Description`, `Last Error Call Stack`) support retry/debugging workflows. The `CheckCanCreateDocument` method prevents duplicate processing by checking whether a `Shpfy Doc. Link To Doc.` already exists for the refund. *Updated: 2026-04-08 -- Shop Code, currency codes, error tracking, and CheckCanCreateDocument on refund header* +*Updated: 2026-07-29 -- return-with-exchange refund lines and pending transaction guard* + ## Things to know - A refund can exist without a return -- for example, an appeasement refund or a partial amount adjustment. - The `Can Create Credit Memo` flag prevents double-processing: refunds that reduced order quantities during import are flagged as non-processable. - The `Return Id` on the refund header links to the `Shpfy Return Header` if the refund was created from a return flow. - Refund transactions are cross-updated -- `UpdateTransactions` stamps the `Refund Id` onto the corresponding `Shpfy Order Transaction` records. +- Credit memo creation is blocked while a linked refund transaction is still `Pending`. `HasPendingRefundTransactions` checks `Shpfy Order Transaction` by refund id, type `Refund`, and status `Pending`; manual creation raises an error, and auto-create exits without building the document. - Refund lines reference order lines, not products directly -- item details come from FlowFields that look up the original order line. +- The Refund page has a Transactions action filtered by `Refund Id`; use it to see whether a refund is still pending before retrying credit memo creation. - The `Note` field is stored as a BLOB to accommodate long refund notes. diff --git a/src/Apps/W1/Shopify/App/src/Order Return Refund Processing/docs/CLAUDE.md b/src/Apps/W1/Shopify/App/src/Order Return Refund Processing/docs/CLAUDE.md index 1f61cf10883..bf3cd3b2f0a 100644 --- a/src/Apps/W1/Shopify/App/src/Order Return Refund Processing/docs/CLAUDE.md +++ b/src/Apps/W1/Shopify/App/src/Order Return Refund Processing/docs/CLAUDE.md @@ -8,14 +8,19 @@ The `IReturnRefund Process` interface (implemented via the extensible enum `Shpf - **Default** (`ShpfyRetRefProcDefault`) -- does nothing. Import is not needed, documents cannot be created. This is the "disabled" state. - **Import Only** (`ShpfyRetRefProcImportOnly`) -- returns and refunds are imported from Shopify (so they appear in staging tables and reduce order line quantities), but no credit memos are auto-created. -- **Auto Create Credit Memo** (`ShpfyRetRefProcCrMemo`) -- imports data and auto-creates sales credit memos (or return orders, depending on `Shop."Process Returns As"`). It validates preconditions: the refund must not already be processed, the parent order must exist and must itself be processed. On success, `ShpfyCreateSalesDocRefund` builds the credit memo with lines from refund lines, return lines, refund shipping lines, and a balance line for any remaining amount. +- **Auto Create Credit Memo** (`ShpfyRetRefProcCrMemo`) -- imports data and auto-creates sales credit memos (or return orders, depending on `Shop."Process Returns As"`). It validates preconditions: the refund must not already be processed, the parent order must exist and must itself be processed. Before running creation it also skips refunds with pending Shopify refund transactions, because the final refunded amount is not settled yet. On success, `ShpfyCreateSalesDocRefund` builds the credit memo with lines from refund lines, return lines, refund shipping lines, and a balance line for any remaining amount. The `IDocument Source` interface (implemented via `Shpfy Source Document Type` enum) provides error reporting back to the source record. The Refund implementation (`ShpfyIDocSourceRefund`) writes errors to the refund header. The extended interface `Shpfy Extended IDocument Source` adds call stack capture for deeper diagnostics. +Return-with-exchange is handled as refund-line data, not as a separate processing strategy. The refund import creates negative-quantity `Shpfy Refund Line` rows for `Return.exchangeLineItems`; `CreateSalesLinesFromRefundLines` then creates ordinary item sales lines from them. If an exchange refund line has no `Location Id`, the creator resolves the location from the linked exchange `Shpfy Order Line` before applying the default return location fallback. This keeps the credit memo total aligned with `Refund.totalRefundedSet` and avoids an artificial `Refund Account` balancing line for the exchange value. + +*Updated: 2026-07-29 -- pending transaction skip and exchange-item refund line processing* + ## Things to know - This module is separate from `Order Refunds` and `Order Returns` which hold the data tables. This module holds only the processing strategies and the credit memo creation logic. - Non-restocked items get posted to a different G/L account (`Refund Acc. non-restock Items`) than restocked items (which are returned as inventory items). Cancelled restock types go to the general `Refund Account`. -- The balance line (`CreateSalesLinesFromRemainingAmount`) catches any difference between the sum of the created sales lines and the total refund amount, posting it to `Refund Account`. This handles adjustments, rounding, and partial refunds that do not fully decompose into line items. You can skip this auto-balancing via the `OnBeforeCreateSalesLinesFromRemainingAmount` event. +- The balance line (`CreateSalesLinesFromRemainingAmount`) catches any difference between the sum of the created sales lines and the total refund amount, posting it to `Refund Account`. This handles adjustments, rounding, and partial refunds that do not fully decompose into line items. Return-with-exchange should normally be covered by the negative-quantity exchange refund lines instead of this balance line. You can skip auto-balancing via the `OnBeforeCreateSalesLinesFromRemainingAmount` event. +- Pending refund transactions are a soft skip in auto-create. The record remains unprocessed so a later order/refund sync can retry after Shopify updates the transaction status. - `ShpfyProcessOrders` triggers refund processing after order processing, but only when the shop's strategy is "Auto Create Credit Memo". - Currency handling for refund documents respects `Processed Currency Handling` from the original order, not the current shop setting. This ensures consistency if the shop's currency handling changed between order creation and refund. diff --git a/src/Apps/W1/Shopify/App/src/Order Return Refund Processing/docs/extensibility.md b/src/Apps/W1/Shopify/App/src/Order Return Refund Processing/docs/extensibility.md index 040ef6bc33b..4590691864d 100644 --- a/src/Apps/W1/Shopify/App/src/Order Return Refund Processing/docs/extensibility.md +++ b/src/Apps/W1/Shopify/App/src/Order Return Refund Processing/docs/extensibility.md @@ -8,6 +8,10 @@ The `Shpfy ReturnRefund ProcessType` enum is `Extensible = true`. To add a custo - `CanCreateSalesDocumentFor(SourceDocumentType, SourceDocumentId, ErrorInfo)` -- validate preconditions. Return true to allow creation. On failure, populate the `ErrorInfo` record with an error message; set `Verbosity` to `Error` for hard failures or `Warning` for soft skips. - `CreateSalesDocument(SourceDocumentType, SourceDocumentId)` -- build and return the Sales Header for the credit memo / return order. +The interface surface did not change for pending Shopify refund transactions. The built-in auto-create implementation checks `ShpfyRefundsAPI.HasPendingRefundTransactions` inside `CreateSalesDocument` and exits without creating a document while a linked refund transaction is still `Pending`. Custom implementations should apply the same guard before creating a document from a refund, because Shopify can report an unsettled refunded amount until the transaction succeeds. + +*Updated: 2026-07-29 -- pending refund transactions remain an implementation guard* + ## IDocument Source interface The `Shpfy Source Document Type` enum is also `Extensible = true` and implements `Shpfy IDocument Source`. The interface has a single method, `SetErrorInfo`, which writes error information back to the source document record. The extended interface `Shpfy Extended IDocument Source` adds `SetErrorCallStack` for detailed diagnostics. When your `IReturnRefund Process` implementation calls `CreateSalesDocument`, errors are routed to the appropriate source document through this interface. diff --git a/src/Apps/W1/Shopify/App/src/Order handling/docs/CLAUDE.md b/src/Apps/W1/Shopify/App/src/Order handling/docs/CLAUDE.md index e56f2e2508f..f9f3e243397 100644 --- a/src/Apps/W1/Shopify/App/src/Order handling/docs/CLAUDE.md +++ b/src/Apps/W1/Shopify/App/src/Order handling/docs/CLAUDE.md @@ -6,13 +6,17 @@ Imports Shopify orders into Business Central, maps them to customers and items, The pipeline has three stages. -- **Discovery**: `ShpfyOrdersAPI` queries the Shopify GraphQL API for orders updated since the last sync. It writes lightweight rows into the `Orders to Import` queue table, not full order data. The query distinguishes first-time sync (open orders only) from incremental sync (all orders by updated-at). +- **Discovery**: `ShpfyOrdersAPI` queries the Shopify GraphQL API for orders updated since the last sync. `ShpfySyncOrdersfromShopify` refreshes the cached shop settings before this call, so plan-dependent query fields use the current Shopify plan. It writes lightweight rows into the `Orders to Import` queue table, not full order data. The query distinguishes first-time sync (open orders only) from incremental sync (all orders by updated-at). -- **Import**: `ShpfyImportOrder` fetches the full order JSON for each queued entry, populates `Order Header` and `Order Line` records, and pulls in related data (tax lines, attributes, risks, shipping charges, transactions, fulfillment orders, returns, refunds). It detects conflicts when a previously processed order is edited in Shopify and flags them with `Has Order State Error`. Staff member retrieval in the GraphQL query and salesperson assignment during header population are conditional on `Shop."Advanced Shopify Plan"` (requires Plus/Advanced plans). +- **Import**: `ShpfyImportOrder` fetches the full order JSON for each queued entry, populates `Order Header` and `Order Line` records, and pulls in related data (tax lines, attributes, risks, shipping charges, transactions, fulfillment orders, returns, refunds). Order line GraphQL pages now request 10 line items per call. If refund import is enabled and the Shopify order has returns, it queries `returns.exchangeLineItems` and marks matching `Order Line` records with `Is Exchange Item`. It detects conflicts when a previously processed order is edited in Shopify and flags them with `Has Order State Error`. Staff member retrieval in the GraphQL query and salesperson assignment during header population are conditional on `Shop."Advanced Shopify Plan"` (requires Plus/Advanced plans). *Updated: 2026-04-08 -- staff member gating changed from B2B Enabled to Advanced Shopify Plan* -- **Processing**: `ShpfyProcessOrder` runs mapping then document creation. `ShpfyOrderMapping.DoMapping` resolves Shopify customers to BC customer numbers (with B2B company mapping as a separate path) and maps each order line's variant to an item/variant/UoM. `ShpfyProcessOrder.CreateHeaderFromShopifyOrder` builds a Sales Header with all three address contexts (sell-to, ship-to, bill-to), then `CreateLinesFromShopifyOrder` adds item lines, tip lines (G/L), gift card lines (G/L), shipping charge lines, and a cash rounding line. Global discounts that were not allocated to individual lines are applied as invoice discount. If the shop has "Auto Release Sales Orders" enabled, the document is released immediately. +*Updated: 2026-07-29 -- order sync now refreshes cached plan data, order line pages use 10 items, and exchange item lines are marked during import* + +- **Processing**: `ShpfyProcessOrder` runs mapping then document creation. `ShpfyOrderMapping.DoMapping` resolves Shopify customers to BC customer numbers (with B2B company mapping as a separate path) and maps each order line's variant to an item/variant/UoM. `ShpfyProcessOrder.CreateHeaderFromShopifyOrder` builds a Sales Header with all three address contexts (sell-to, ship-to, bill-to), then `CreateLinesFromShopifyOrder` adds non-exchange item lines, tip lines (G/L), gift card lines (G/L), shipping charge lines, and a cash rounding line. Global discounts that were not allocated to individual lines are applied as invoice discount. If the shop has "Auto Release Sales Orders" enabled, the document is released immediately. + +*Updated: 2026-07-29 -- exchange item order lines are excluded from BC sales document lines* ## Things to know @@ -24,6 +28,13 @@ The pipeline has three stages. *Updated: 2026-04-08 -- contact lookup/validation added to Shopify Order page (PR #7525)* +- B2C mapping no longer overwrites a manually assigned `Sell-to Customer No.` just because bill-to mapping runs. If bill-to mapping cannot find a customer and no shop default customer is available, the existing sell-to customer is copied to bill-to so the order can still map. +- When customer mapping by email or phone is allowed to create an unknown BC customer, a Shopify customer with no default address can still be created from the first available Shopify customer address. If the Shopify customer has no addresses at all, mapping still returns blank. +- `SetAndInsertOrderLines` calls `TempOrderLine.Init()` for each GraphQL line item. Without that reset, a previous tip line could leave the temporary record's `Tip` flag set for the next line. +- Order tax lines can now belong to shipping charges as well as headers and item lines. The shared `Parent Id` points to the Shopify shipping line id for shipping-level taxes. +- `Created At`, `Updated At`, `Closed At`, and `Processed At` on the order header are Shopify timestamps. `Document Date` is the BC document date and is initialized from Shopify `createdAt` during import. - Fulfilled orders become invoices instead of sales orders when `Create Invoices From Orders` is enabled on the shop. - `ShpfyProcessOrders` (plural) is the batch entry point. After processing orders it also processes refunds if the shop uses the "Auto Create Credit Memo" strategy. - `ShpfyOrders` (public API codeunit) exposes `MarkAsPaid` and `CancelOrder` for external callers. + +*Updated: 2026-07-29 -- captured customer mapping preservation, no-default-address customer creation, line reset, shipping tax links, and timestamp semantics* diff --git a/src/Apps/W1/Shopify/App/src/Order handling/docs/business-logic.md b/src/Apps/W1/Shopify/App/src/Order handling/docs/business-logic.md index 2511608c12c..f6b90a00d11 100644 --- a/src/Apps/W1/Shopify/App/src/Order handling/docs/business-logic.md +++ b/src/Apps/W1/Shopify/App/src/Order handling/docs/business-logic.md @@ -4,14 +4,16 @@ ```mermaid flowchart TD - A[ShpfyOrdersAPI.GetOrdersToImport] -->|writes| B[Orders to Import queue] + A0[Refresh cached shop plan with Shop.GetShopSettings] --> A[ShpfyOrdersAPI.GetOrdersToImport] + A -->|writes| B[Orders to Import queue] B -->|each row| C[ShpfyImportOrder] - C -->|creates/updates| D[Order Header + Lines] - D --> E{User or auto-process} + C -->|creates/updates| D[Order Header + Lines + related records] + D --> C1[Mark exchange item order lines when needed] + C1 --> E{User or auto-process} E -->|ShpfyProcessOrders| F[ShpfyProcessOrder.OnRun] F --> G[ShpfyOrderMapping.DoMapping] G -->|success| H[CreateHeaderFromShopifyOrder] - H --> I[CreateLinesFromShopifyOrder] + H --> I[CreateLinesFromShopifyOrder for non-exchange lines] I --> J[ApplyGlobalDiscounts] J --> K{Auto Release?} K -->|yes| L[ReleaseSalesDocument] @@ -19,13 +21,19 @@ flowchart TD G -->|failure| N[Error: not everything mapped] ``` +*Updated: 2026-07-29 -- plan refresh and exchange-item filtering were added to the order pipeline* + ## Phase 1: order discovery `ShpfyOrdersAPI.GetOrdersToImport` paginates through Shopify's GraphQL `orders` connection. On first sync (no prior sync time), it uses `GetOpenOrdersToImport` which returns only open orders. On subsequent syncs it uses `GetOrdersToImport` filtered by `updatedAt` since the last sync time. For each order node it populates or updates a row in `Orders to Import`, computing the `Import Action` (New or Update) by checking whether an `Order Header` already exists. Closed orders are only imported as updates, never as new. +`ShpfySyncOrdersfromShopify` refreshes the cached shop settings with `Shop.GetShopSettings()` before it calls `GetOrdersToImport`. That matters for plan-gated order fields, because the import query later decides whether to request `staffMember` from the refreshed `Advanced Shopify Plan` value. + +*Updated: 2026-07-29 -- order sync refreshes cached Shopify plan data before discovery* + ## Phase 2: order import -`ShpfyImportOrder.ImportOrderAndCreateOrUpdate` does the heavy lifting. It fetches the full order JSON via a second GraphQL call (`GetOrderHeader`), retrieves order lines in a paginated loop (`GetOrderLines` / `GetNextOrderLines`), and populates the staging tables. +`ShpfyImportOrder.ImportOrderAndCreateOrUpdate` does the heavy lifting. It fetches the full order JSON via a second GraphQL call (`GetOrderHeader`), retrieves order lines in a paginated loop (`GetOrderLines` / `GetNextOrderLines`), and populates the staging tables. The order line resource files request `lineItems(first: 10)`, so most orders need fewer GraphQL calls than the previous three-line page size. `SetAndInsertOrderLines` initializes the temporary order line record before every line item; this prevents flags such as `Tip` from carrying over from the previous JSON node. `RetrieveOrderHeaderJson` conditionally includes the `staffMember { id }` field in the GraphQL query parameters based on `Shop."Advanced Shopify Plan"`. When the shop is not on an Advanced/Plus plan, the `StaffMember` parameter is sent as an empty string, omitting staff data from the query. `SetNewOrderHeaderValuesFromJson` (which populates the order header from the JSON response) similarly gates the staff member processing -- it only extracts the staff member ID and calls `SetSalespersonOnOrderHeader` when `Shop."Advanced Shopify Plan"` is true. @@ -35,6 +43,10 @@ For already-processed orders, it runs conflict detection. A conflict is flagged After populating the header and lines, the import calls into related modules: fulfillment orders, shipping charges, transactions, returns (if the return/refund process requires import), and refunds. It then adjusts order line quantities and header amounts by subtracting refund line quantities and amounts. Zero-quantity lines are deleted. If the order is fully fulfilled and paid, and `Archive Processed Orders` is enabled, it closes the order in Shopify via the `CloseOrder` GraphQL mutation. +When refund import is enabled, `MarkExchangeItemOrderLines` runs after related records are imported and before refund quantities are considered. It skips the extra API call for orders whose return status is blank or `No Return`. For orders with returns, it pages through `returns(first: 10)` using `GetOrderExchangeLineItems` / `GetNextOrderExchangeLineItems`, collects the Shopify line item ids under each return's `exchangeLineItems`, and sets `Shpfy Order Line`.`Is Exchange Item` on matching order lines. + +*Updated: 2026-07-29 -- order line pagination, temporary line reset, and exchange item marking changed import behavior* + ## Phase 3: mapping and document creation ### Customer mapping @@ -47,27 +59,42 @@ flowchart TD D -->|yes, with default customer| E[Use default customer from template] D -->|no| F{Customer import = None?} F -->|yes| G[Use shop default customer] - F -->|no| H[CustomerMapping.DoMapping] - H --> I[Resolve sell-to and bill-to separately] - C --> J{Company Location Id set?} - J -->|yes| K[MapSellToBillToCustomersFromCompanyLocation] - J -->|no| L[CompanyMapping.DoMapping] + F -->|no| H{Sell-to customer already set?} + H -->|no| I[Map or create sell-to from sell-to address] + H -->|yes| I2[Keep existing sell-to] + I --> K[Map bill-to from bill-to address] + I2 --> K + K --> L{Bill-to mapped?} + L -->|yes| M[Use mapped bill-to] + L -->|no, default exists| N[Use shop default customer] + L -->|no default| O[Copy sell-to to bill-to] + C --> B2BLoc{Company Location Id set?} + B2BLoc -->|yes| P[MapSellToBillToCustomersFromCompanyLocation] + B2BLoc -->|no| Q[CompanyMapping.DoMapping] ``` -`ShpfyOrderMapping.DoMapping` first resolves the customer (or company for B2B), then iterates order lines. Tip lines require a configured `Tip Account`, gift card lines require a `Sold Gift Card Account`, and regular lines go through `MapVariant` which looks up the Shopify variant, auto-imports the product if needed, and resolves the BC item, variant code, and unit of measure. +`ShpfyOrderMapping.DoMapping` first resolves the customer (or company for B2B), then iterates order lines. For B2C orders, `MapHeaderFields` maps sell-to only when `Sell-to Customer No.` is blank. It then maps bill-to from the bill-to address. If bill-to mapping returns blank and no default customer is configured, the existing sell-to customer is copied to bill-to. This preserves a manually assigned sell-to customer instead of replacing it during a failed bill-to mapping pass. + +When the email/phone customer mapping strategy is allowed to create an unknown customer and the Shopify customer has no default address, the customer is created from the first available Shopify customer address. If the Shopify customer has no address records, mapping still fails with a blank customer number. + +Tip lines require a configured `Tip Account`, gift card lines require a `Sold Gift Card Account`, and regular lines go through `MapVariant` which looks up the Shopify variant, auto-imports the product if needed, and resolves the BC item, variant code, and unit of measure. The mapping also resolves shipping method (from `Shpfy Shipment Method Mapping` by shipping charge title), shipping agent, and payment method (from the order's successful transactions, only if a single payment method is found). +*Updated: 2026-07-29 -- customer mapping now preserves existing sell-to values and handles Shopify customers without a default address* + ### Sales document creation `ShpfyProcessOrder.CreateHeaderFromShopifyOrder` inserts a Sales Header. The document type is Order by default, or Invoice if the order is already fulfilled and `Create Invoices From Orders` is enabled. It copies all three address blocks from the Shopify order, sets currency based on the shop's `Currency Handling`, applies the document date, shipping method, payment method, tax area, and payment terms. It writes a `Doc. Link To Doc.` record to tie the Shopify order to the BC document. If `Order Attributes To Shopify` is enabled, it pushes the BC document number back to Shopify as an order attribute. -`CreateLinesFromShopifyOrder` iterates order lines and creates corresponding sales lines. Tips go to the tip G/L account, gift cards to the sold gift card G/L account, and regular items are validated with location code resolution from `Shpfy Shop Location`. Shipping charges become separate sales lines, either as G/L account lines using the shop's `Shipping Charges Account` or as item charge lines when configured through `Shpfy Shipment Method Mapping`. Item charges are automatically assigned to the item lines. +`CreateLinesFromShopifyOrder` iterates order lines where `Is Exchange Item = false` and creates corresponding sales lines. Tips go to the tip G/L account, gift cards to the sold gift card G/L account, and regular items are validated with location code resolution from `Shpfy Shop Location`. Shipping charges become separate sales lines, either as G/L account lines using the shop's `Shipping Charges Account` or as item charge lines when configured through `Shpfy Shipment Method Mapping`. Item charges are automatically assigned to the item lines. `ApplyGlobalDiscounts` calculates the portion of the order's total discount that was not allocated to individual lines or shipping charges, and applies it as an invoice discount on the sales header. A final cash rounding line is created if the order has a non-zero `Payment Rounding Amount`, posted to the shop's `Cash Roundings Account`. +*Updated: 2026-07-29 -- exchange item order lines are skipped when creating the sales document* + ## Error handling `ShpfyProcessOrders.ProcessShopifyOrder` wraps `ShpfyProcessOrder.Run` in a `if not ... Run` pattern. On failure it captures the error text into the order header's `Error Message`, clears the sales document number, and calls `CleanUpLastCreatedDocument` to delete the partially created sales document. On success it sets `Processed = true` and records the `Processed Currency Handling` so refund processing later knows which currency was used. diff --git a/src/Apps/W1/Shopify/App/src/Order handling/docs/data-model.md b/src/Apps/W1/Shopify/App/src/Order handling/docs/data-model.md index 48d52e2101b..04d2ffeceb3 100644 --- a/src/Apps/W1/Shopify/App/src/Order handling/docs/data-model.md +++ b/src/Apps/W1/Shopify/App/src/Order handling/docs/data-model.md @@ -7,27 +7,45 @@ erDiagram OrderHeader ||--o{ OrderLine : "Shopify Order Id" OrderHeader ||--o{ OrderTaxLine : "Shopify Order Id = Parent Id (header-level)" OrderLine ||--o{ OrderTaxLine : "Line Id = Parent Id (line-level)" + OrderHeader ||--o{ OrderShippingCharges : "Shopify Order Id" + OrderShippingCharges ||--o{ OrderTaxLine : "Shopify Shipping Line Id = Parent Id (shipping-level)" OrdersToImport }o--|| OrderHeader : "Id = Shopify Order Id" OrderHeader ||--o{ OrderAttribute : "Shopify Order Id = Order Id" ``` -## Order Header and Order Line +*Updated: 2026-07-29 -- order tax lines can now be linked to shipping charges* + +## Order header and order line `Shpfy Order Header` (table 30118) is keyed on `Shopify Order Id` (BigInteger). It carries the full snapshot of a Shopify order: three complete address blocks (sell-to, ship-to, bill-to), financial totals, status enums, and the BC-side output fields (`Sales Order No.`, `Sales Invoice No.`, `Sell-to Customer No.`, `Bill-to Customer No.`). -`Shpfy Order Line` (table 30119) is keyed on `(Shopify Order Id, Line Id)`. Each line holds the Shopify product/variant reference plus the mapped BC item (`Item No.`, `Variant Code`, `Unit of Measure Code`). Boolean flags `Gift Card` and `Tip` classify non-inventory lines. The secondary key on `(Shopify Order Id, Gift Card, Tip)` maintains a SIFT index on `Quantity`, which drives the header's `Total Quantity of Items` FlowField (excluding tips and gift cards). +`Shpfy Order Line` (table 30119) is keyed on `(Shopify Order Id, Line Id)`. Each line holds the Shopify product/variant reference plus the mapped BC item (`Item No.`, `Variant Code`, `Unit of Measure Code`). Boolean flags `Gift Card` and `Tip` classify non-inventory lines. `Is Exchange Item` marks the new item in a return-with-exchange; document creation filters those lines out of the BC sales document. The secondary key on `(Shopify Order Id, Gift Card, Tip)` maintains a SIFT index on `Quantity`, which drives the header's `Total Quantity of Items` FlowField (excluding tips and gift cards). Deleting a header cascades. The `OnDelete` trigger on Order Header explicitly deletes order lines, return headers, refund headers, data capture records, fulfillment order headers, and order fulfillments. +*Updated: 2026-07-29 -- documented the exchange item flag on order lines* + ## Dual-currency design Every amount field on both header and line exists in two versions: shop currency and presentment currency. The header stores `Currency Code` (the shop's settlement currency) and `Presentment Currency Code` (the currency the buyer saw). Lines reference the header for formatting via local helper procedures `OrderCurrencyCode()` and `OrderPresentmentCurrencyCode()`. During processing, the shop's `Currency Handling` setting determines which currency column is used to populate the BC sales document. -## Order Tax Line (polymorphic parent) +## Timestamp semantics + +Order header timestamps with Shopify suffixes are Shopify lifecycle timestamps. `Created At`, `Updated At`, `Closed At`, and `Processed At` are populated from Shopify `createdAt`, `updatedAt`, `closedAt`, and `processedAt`. `Document Date` is different: it is the BC document date and is initialized from Shopify `createdAt` when the order header is first imported. + +The `Orders to Import` queue stores only the Shopify `Created At` and `Updated At` values needed for review and incremental import decisions. + +*Updated: 2026-07-29 -- Shopify timestamp captions and tooltips were disambiguated* + +## Order tax line (polymorphic parent) + +`Shpfy Order Tax Line` (table 30122) uses a `Parent Id` field that can point to an Order Header (`Shopify Order Id`), an Order Line (`Line Id`), or an Order Shipping Charges record (`Shopify Shipping Line Id`). The `OrderCurrencyCode()` and `OrderPresentmentCurrencyCode()` helpers attempt to resolve the parent as an order line first, then as shipping charges, then as the header. This polymorphic key is not enforced by a table relation; the code simply tries those lookups. The `Channel Liable` flag indicates marketplace-collected taxes, and the header has a FlowField `Channel Liable Taxes` that checks for their existence. + +Shipping charge tax lines are imported by `ShpfyShippingCharges.UpdateShippingCostInfos` from each GraphQL `shippingLines.nodes[*].taxLines` array. Deleting a shipping charge deletes tax lines whose `Parent Id` is that shipping line id. -`Shpfy Order Tax Line` (table 30122) uses a `Parent Id` field that can point to either an Order Header (`Shopify Order Id`) or an Order Line (`Line Id`). The `OrderCurrencyCode()` helper attempts to resolve the parent as an Order Line first, then walks up to the header. This polymorphic key is not enforced by a table relation; the code simply tries both lookups. The `Channel Liable` flag indicates marketplace-collected taxes, and the header has a FlowField `Channel Liable Taxes` that checks for their existence. +*Updated: 2026-07-29 -- shipping charge tax lines now share the order tax line table* -## Orders to Import (queue table) +## Orders to import (queue table) `Shpfy Orders to Import` (table 30121) is a transient queue populated by `ShpfyOrdersAPI.GetOrdersToImport` and consumed by `ShpfyImportOrder`. It carries summary fields (amount, quantity, financial status, fulfillment status, tags) so users can review and filter before importing. The `Import Action` enum distinguishes `New` from `Update`. Error tracking uses blob fields for the message and call stack since error text can be long. @@ -60,6 +78,6 @@ When `Sell-to Customer No.` is validated, it automatically re-resolves both `Sel The Order Header has a cluster of B2B fields: `Company Id`, `Company Main Contact Id`, `Company Main Contact Email`, `Company Main Contact Phone No.`, `Company Main Contact Cust. Id`, `Company Location Id`, `B2B` (boolean), and `PO Number`. When `B2B` is true, mapping takes a different path through `MapB2BHeaderFields` in `ShpfyOrderMapping`. -## High Risk +## High risk `High Risk` is a FlowField (`CalcFormula = exist`) that checks for any `Shpfy Order Risk` record with `Level = High` for the order. It is not stored; it is calculated on demand. diff --git a/src/Apps/W1/Shopify/App/src/Order handling/docs/extensibility.md b/src/Apps/W1/Shopify/App/src/Order handling/docs/extensibility.md index 85f5341e6c8..9433413ecbc 100644 --- a/src/Apps/W1/Shopify/App/src/Order handling/docs/extensibility.md +++ b/src/Apps/W1/Shopify/App/src/Order handling/docs/extensibility.md @@ -7,6 +7,10 @@ All events are published by `ShpfyOrderEvents` (codeunit 30162). Events marked ` - `OnBeforeMapCustomer` / `OnAfterMapCustomer` -- wrap the customer resolution for B2C orders. Setting `Handled = true` on the Before event skips the built-in mapping entirely, letting you assign `Sell-to Customer No.` and `Bill-to Customer No.` yourself. - `OnBeforeMapCompany` / `OnAfterMapCompany` -- same pattern for B2B orders. +If you do not handle `OnBeforeMapCustomer`, the built-in B2C mapper preserves an existing `Sell-to Customer No.` and only fills it when blank. If bill-to mapping fails and no default customer is configured, it copies the existing sell-to customer to bill-to. Subscribers that partially prefill customers should account for that fallback. + +*Updated: 2026-07-29 -- documented the built-in sell-to preservation fallback* + ## Shipping and payment mapping - `OnBeforeMapShipmentMethod` / `OnAfterMapShipmentMethod` -- override how the shipping charge title maps to a BC shipment method code. @@ -23,9 +27,17 @@ All events are published by `ShpfyOrderEvents` (codeunit 30162). Events marked ` - `OnBeforeCreateItemSalesLine` / `OnAfterCreateItemSalesLine` -- wrap the creation of each item/tip/gift card sales line. The Before event supports the Handled pattern. When handled, the built-in line creation is skipped but the After event still fires. Use this to customize pricing, account assignment, or to skip specific lines entirely. - `OnBeforeCreateShippingCostSalesLine` / `OnAfterCreateShippingCostSalesLine` -- wrap the creation of each shipping charge line. Also supports the Handled pattern. +Exchange item order lines are filtered out before `OnBeforeCreateItemSalesLine` runs. If an extension needs to inspect those lines, use import-level events after order lines are created rather than sales-line events. + +*Updated: 2026-07-29 -- exchange items are not exposed through item sales-line creation events* + ## Import and status conversion - `OnAfterImportShopifyOrderHeader` -- fires after the order header JSON is parsed and the header record is updated. The `IsNew` parameter indicates first import vs. update. - `OnAfterCreateShopifyOrderAndLines` -- fires after both header and lines are fully imported and related records (tax lines, attributes, fulfillments) are created. - `OnAfterConsiderRefundsInQuantityAndAmounts` -- fires after each order line's quantity and the header's amounts are adjusted for refunds. - `OnBeforeConvertToFinancialStatus`, `OnBeforeConvertToFulfillmentStatus`, `OnBeforeConvertToOrderReturnStatus` -- internal events that let you override enum conversion from Shopify's status strings, useful when Shopify adds new status values before the enum is updated. + +`OnAfterCreateShopifyOrderAndLines` runs after `MarkExchangeItemOrderLines` and refund adjustments. At that point `Is Exchange Item` has already been set on any order lines matched from Shopify return exchange line items. + +*Updated: 2026-07-29 -- clarified import event timing for exchange item flags* diff --git a/src/Apps/W1/Shopify/App/src/Products/docs/CLAUDE.md b/src/Apps/W1/Shopify/App/src/Products/docs/CLAUDE.md index e904d632056..efaa7afac9b 100644 --- a/src/Apps/W1/Shopify/App/src/Products/docs/CLAUDE.md +++ b/src/Apps/W1/Shopify/App/src/Products/docs/CLAUDE.md @@ -5,8 +5,11 @@ Bi-directional product and variant sync between BC Items and Shopify Products. ## What it does Export pushes BC Item data (title, description, vendor, price, variants, images, -tags, metafields, translations) to Shopify. Import pulls Shopify Products back -and either maps them to existing Items or auto-creates new ones via templates. +tags, metafields, translations, tariff numbers, and country of origin) to +Shopify. Import pulls Shopify Products back and either maps them to existing +Items or auto-creates new ones via templates. + +*Updated: 2026-07-29 -- HS code and country of origin now participate in product sync* ## How it works @@ -16,14 +19,17 @@ field-by-field with a snapshot, and only calls the API when something changed. Variants are matched by Item Variant SystemId and optionally by UoM option slot. New BC Item Variants that have no Shopify counterpart are created as new Shopify variants; existing ones are updated. Price-only sync is a separate fast path that -uses bulk GraphQL mutations. +uses bulk GraphQL mutations when enough changed prices have been collected. Import (`ShpfyProductImport`) uses `ShpfyProductMapping` to find a BC Item for each variant. Mapping is SKU-driven -- the Shop's SKU Mapping setting determines whether SKU is matched as Item No., Variant Code, Item No.+Variant Code, Barcode, -or Vendor Item No. Unmatched products can auto-create Items via +or Vendor Item No. If that primary strategy fails, the shop can optionally fall +back to the variant barcode. Unmatched products can auto-create Items via `ShpfyCreateItem`, which applies an Item Template and creates references. +*Updated: 2026-07-29 -- price-only sync uses changed-price count and mapping fallback is configurable* + ## Things to know - Product-to-Item linking uses `Item SystemId` (a Guid), not Item No. The @@ -38,9 +44,37 @@ or Vendor Item No. Unmatched products can auto-create Items via - Hash fields (`Image Hash`, `Tags Hash`, `Description Html Hash`) enable cheap change detection without comparing blob content. - `ICreateProductStatusValue` determines whether newly created products start - as Active or Draft. + as Active, Draft, or Unlisted. Active means the connector creates the product + as immediately active, Draft keeps it in draft, and Unlisted sends Shopify's + Unlisted status without additional local branching. Archived is separate -- + it is a product status and removal outcome, not a "Status for Created + Products" choice. - Max 2048 variants per product -- enforced in `ShpfyCreateProduct`. - Price sync silently skips items whose unit of measure is invalid (not in `Unit of Measure` table or not in `Item Unit of Measure` for that item). A `Shpfy Skipped Record` entry is logged instead of raising an error. - Item attributes marked "As Option" can drive Shopify product options instead of the default Variant/UoM scheme, with validation for uniqueness and completeness. +- When `Find Mapping by Barcode` is enabled, barcode lookup is a fallback after + SKU mapping fails. Disable it when barcodes are shared, recycled, or not a + trustworthy item identity in BC. +- HS code and country of origin flow through `Shpfy Variant` during import and + export. Import only writes existing BC Tariff Number and Country/Region values + to Items, so blank or unknown Shopify values do not wipe existing item data. +- Product import skips products whose Shopify `Updated At` is not newer than + both the local product timestamp and `Last Updated by BC`. When a stale product + is skipped, mapped variants are left intact rather than being deleted. +- `Shpfy Product Price Calc.` is SingleInstance and caches a temporary Sales + Quote. Its shop cache is refreshed when WorkDate changes, so price sync does + not keep using yesterday's document date in long-running sessions. +- The Item Card treats variant-mapped items as already mapped. "Add to Shopify" + is only enabled for shops where neither a Product nor a Variant maps to the + current Item. +- In product export, an existing Shopify variant can map to a different BC Item + than the parent product. `UpdateProductData` re-fetches the parent Item after + updating existing variants so child-item variants do not corrupt the remaining + create/update decisions for the parent. +- Bulk variant price updates send `compareAtPrice: null` when the compare-at + price is not greater than the current price. The Shopify Variants page exposes + Compare-at Price for diagnostics and personalization. + +*Updated: 2026-07-29 -- product status, mapping, origin sync, import staleness, and price-sync gotchas added* diff --git a/src/Apps/W1/Shopify/App/src/Products/docs/business-logic.md b/src/Apps/W1/Shopify/App/src/Products/docs/business-logic.md index 9656f71a4ec..e6da6ca8a40 100644 --- a/src/Apps/W1/Shopify/App/src/Products/docs/business-logic.md +++ b/src/Apps/W1/Shopify/App/src/Products/docs/business-logic.md @@ -37,9 +37,37 @@ option slots to find the matching UoM value before deciding create-vs-update. Th logic in `UpdateProductData` is intentionally exhaustive -- it checks Option 1, then Option 2, then Option 3 for the UoM match. +Existing Shopify variants may map to child Items, not only to the parent Item +behind the Shopify Product. After updating those existing variants, +`UpdateProductData` re-fetches the parent Item by the product's `Item SystemId` +before it scans BC Item Variants to create missing Shopify variants. Without that +reset, the second half of export can accidentally continue with the last child +Item loaded from the variant loop. + +*Updated: 2026-07-29 -- child-item variant handling now resets to the parent item before creating missing variants* + Price-only mode (`OnlyUpdatePrice`) skips all non-price fields and attempts a bulk -GraphQL mutation through `ShpfyBulkOperationMgt`. If bulk fails, it falls back to -per-variant API calls and reverts any partially applied changes. +GraphQL mutation through `ShpfyBulkOperationMgt`. The bulk decision uses the +number of changed variant prices queued in `GraphQueryList`, not the total number +of variants considered. If bulk cannot be sent, the connector falls back to +per-variant API calls and reverts any failed variants from the request snapshot. +When bulk is sent, request data is stored for rollback and, when all logging is +enabled, the sent JSONL can be downloaded from Shopify Bulk Operations. + +```mermaid +flowchart TD + A[Calculate variant prices] --> B{Any price, compare-at, or cost changed?} + B -- no --> C[Do not queue variant] + B -- yes --> D[Queue GraphQL call and JSONL row] + D --> E{Changed count >= bulk threshold?} + E -- yes --> F[Try SendBulkMutation] + F -- accepted --> G[Webhook completes and failed lines are reverted or logged] + F -- not accepted --> H[Send queued variants individually] + E -- no --> H + H --> I[Revert individual failures] +``` + +*Updated: 2026-07-29 -- bulk price threshold is based on changed prices and JSONL diagnostics are retained* ### Product creation @@ -51,6 +79,25 @@ sales-blocked item variants are skipped with a logged reason. The 2048-variant Shopify limit is enforced upfront -- if the expected variant count exceeds it, the entire item is skipped. +```mermaid +flowchart TD + A[Read Shop Status for Created Products] --> B{Selected value} + B -- Active --> C[Set Product.Status = Active] + B -- Draft --> D[Set Product.Status = Draft] + B -- Unlisted --> E[Set Product.Status = Unlisted] + C --> F[Create Shopify product] + D --> F + E --> F +``` + +Unlisted is a creation status only when the shop setting chooses it. The +connector sets the product enum value and sends it through the normal create +payload; it does not add special local branching for Unlisted. Existing product +removal still uses the separate "Action for Removed Products" interface, where +blocked items can be archived, drafted, or left unchanged. + +*Updated: 2026-07-29 -- product creation can now choose Unlisted status* + ### Price calculation `ShpfyProductPriceCalc` creates a temporary Sales Quote header using the Shop's @@ -60,6 +107,14 @@ calculated `Unit Price`, `Line Amount`, and `Unit Cost`. If ComparePrice is less than or equal to Price after calculation, ComparePrice is zeroed out. Events fire before and after to allow overrides. +Because the codeunit is SingleInstance, the temporary Sales Quote is cached across +calls. `SetShop` now also compares the cached quote's document date with +`WorkDate()` before reusing it. This matters for unattended Sync Prices sessions: +without that check, a long-lived session could keep calculating against an old +document date after WorkDate changed. + +*Updated: 2026-07-29 -- price calculation cache is refreshed when WorkDate changes* + Before any price calculation, `CalcPrice` validates the unit of measure via `IsValidUoM`. This checks that the UoM code exists in the `Unit of Measure` table and is a valid `Item Unit of Measure` for the given item. If validation @@ -80,7 +135,7 @@ images are handled separately -- they update Item Variant pictures when the variant maps to an Item Variant, or the Item picture when it maps to the Item itself. -### Body HTML generation +### Body html generation `CreateProductBody` in `ShpfyProductExport` assembles the Shopify product description from three optional sources: extended text lines, marketing text @@ -106,12 +161,48 @@ flowchart TD Mapping in `ShpfyProductMapping.DoFindMapping` is SKU-strategy-driven. Based on the Shop's SKU Mapping setting, it tries to match the variant's SKU to an Item No., Vendor Item No., Variant Code, Item No.+Variant Code (split by separator), -or Barcode. If SKU matching fails, it falls back to barcode matching as a last -resort. The `OnBeforeFindProductMapping` event fires before any of this, allowing -complete override. +or Barcode. If the primary strategy fails, barcode matching runs only when the +Shop's `Find Mapping by Barcode` setting is enabled. The `OnBeforeFindProductMapping` +event fires before any of this, allowing complete override. + +```mermaid +flowchart TD + A[Start mapping] --> B[Run OnBeforeFindProductMapping] + B --> C{Handled?} + C -- yes --> D[Use subscriber result] + C -- no --> E[Try selected SKU mapping] + E --> F{Found?} + F -- yes --> G[Write Item and Item Variant links] + F -- no --> H{Find Mapping by Barcode?} + H -- yes --> I[Try variant barcode lookup] + H -- no --> J[Remain unmapped] + I --> K{Found by barcode?} + K -- yes --> G + K -- no --> J +``` + +Disable the fallback when barcodes are not unique enough to be an identity key. +The SKU strategy still runs first, so the setting only controls the last-chance +barcode lookup. + +*Updated: 2026-07-29 -- barcode fallback is now controlled by a shop setting* Item creation in `ShpfyCreateItem` applies an Item Template (from Shop config or event override), sets description, prices (converted from shop currency if needed), vendor, and item category. For `Item No. + Variant Code` and `Variant Code` SKU mappings, it also creates Item Variants and cross-references (barcodes, vendor item references). + +When `"Sync HS Code and Country"` (captioned *Sync HS Code and Country of Origin*) is enabled, new Items copy the variant's +HS code and country from Shopify. Existing Items are updated only when the +Shopify value resolves to an existing BC Tariff Number or Country/Region, so an +unknown or blank Shopify value does not clear a valid BC value. + +Product import first compares Shopify product IDs and timestamps. A product is +retrieved only when Shopify's `Updated At` is newer than both the local product +`Updated At` and `Last Updated by BC`, or when the product is new locally. If a +product is skipped as stale, the importer never enters the variant retrieval path +that deletes records for variants missing in Shopify. This preserves mapped +variants when only the timestamp comparison is stale. + +*Updated: 2026-07-29 -- HS code/country import and stale-product variant preservation documented* diff --git a/src/Apps/W1/Shopify/App/src/Products/docs/data-model.md b/src/Apps/W1/Shopify/App/src/Products/docs/data-model.md index 3171d70c765..a324dba8bdc 100644 --- a/src/Apps/W1/Shopify/App/src/Products/docs/data-model.md +++ b/src/Apps/W1/Shopify/App/src/Products/docs/data-model.md @@ -7,9 +7,34 @@ erDiagram Product ||--o{ Variant : "has" Variant ||--o| InventoryItem : "tracked by" Product }o--|| Shop : "belongs to" + Product }o--o| Item : "maps by SystemId" + Variant }o--o| Item : "can map directly" Variant }o--o| ItemVariant : "maps to" + + Product { + bigint Id + enum Status + guid Item_SystemId + datetime Last_Updated_by_BC + } + Variant { + bigint Id + guid Item_SystemId + guid Item_Variant_SystemId + code Tariff_No + code Country_Region_of_Origin_Code + decimal Compare_at_Price + } + InventoryItem { + bigint Id + bigint Variant_Id + text Country_Region_of_Origin + datetime Updated_At + } ``` +*Updated: 2026-07-29 -- diagram now shows item mappings and origin/compare-at fields carried by variants* + ## Product (table 30127) The central record linking a Shopify product to a BC Item. The link is through @@ -28,6 +53,13 @@ The OnDelete trigger is where product removal policy lives. It reads the Shop's implementation, and calls `RemoveProductAction` before cascading deletes to Variants and Metafields. +`Status` is Shopify-facing and now includes Active, Archived, Draft, and +Unlisted. The creation setting on the Shop can choose Active, Draft, or Unlisted +through `ICreateProductStatusValue`; Archived is reserved for existing products +and removal flows. + +*Updated: 2026-07-29 -- Unlisted product status added and creation status choices clarified* + ## Variant (table 30129) Each Shopify variant belongs to a Product via `Product Id`. It maps to a BC Item @@ -44,12 +76,31 @@ the option slots hold attribute name/value pairs instead. The `Image Hash` field tracks the variant-level image separately from the product image, enabling per-variant image sync. +`Tariff No.` and `Country/Region of Origin Code` are stored on the Variant +because Shopify exposes HS code and origin on the variant inventory item payload. +Export fills them from the BC Item when `"Sync HS Code and Country"` (captioned *Sync HS Code and Country of Origin*) is +enabled. Import reads them from Shopify and uses them when creating or updating +BC Items, but update logic only applies values that resolve to existing BC +Tariff Number and Country/Region records. + +`Compare at Price` is part of the variant's pricing state. Price sync only sends +it when it is greater than the actual price; otherwise the GraphQL payload clears +Shopify's compare-at price with `null`. + +*Updated: 2026-07-29 -- variant origin fields and compare-at price behavior added* + ## InventoryItem (table 30126) A Shopify inventory item, linked to a Variant by `Variant Id`. This is Shopify's -physical-goods record -- it holds country of origin, shipping requirements, and -whether inventory is tracked. There is no direct BC table counterpart; it exists -purely to mirror the Shopify data model. +physical-goods record -- it holds country of origin, shipping requirements, unit +cost, and whether inventory is tracked. There is no direct BC table counterpart; +it exists purely to mirror the Shopify data model. + +The import path updates `InventoryItem` only when Shopify's inventory item +`updatedAt` is newer than the local `Updated At`. The Variant still carries the +HS code and country code used by BC mapping and export. + +*Updated: 2026-07-29 -- inventory item origin data and timestamp behavior clarified* ## The "Has Variants" gotcha @@ -59,3 +110,9 @@ Variant needed. When `Has Variants` is true, the connector expects each non-defa Variant to carry an `Item Variant SystemId`. This flag drives branching throughout export, import, and mapping -- if it gets out of sync with reality, mapping will silently fail or skip variants. + +Variant-mapped items count as mapped for the Item Card and Item List actions. +The UI checks both `Shpfy Product.Item SystemId` and `Shpfy Variant.Item SystemId` +before allowing "Add to Shopify" for a shop. + +*Updated: 2026-07-29 -- variant-level item mappings now affect add-to-Shopify availability* diff --git a/src/Apps/W1/Shopify/App/src/Products/docs/extensibility.md b/src/Apps/W1/Shopify/App/src/Products/docs/extensibility.md index 015dc7e887f..a0cf115e6cb 100644 --- a/src/Apps/W1/Shopify/App/src/Products/docs/extensibility.md +++ b/src/Apps/W1/Shopify/App/src/Products/docs/extensibility.md @@ -15,12 +15,26 @@ These fire inside `ShpfyProductPriceCalc.CalcPrice` and are the right place to implement custom pricing tiers, catalog-specific markups, or currency overrides. +`Shpfy Update Price Source` (codeunit 30272) is public and is manually bound by +`CalcPrice` while the temporary Sales Line is validated. It supplies Shopify +shop pricing context to the price source pipeline when the temporary customer +does not exist as a real BC Customer. + +*Updated: 2026-07-29 -- public price-source subscriber documented* + ## Customizing product mapping (import) - `OnBeforeFindProductMapping` -- intercept the mapping lookup before the SKU-based strategy runs. Set Handled to bypass default logic. You receive the Product, Variant, and output Item/ItemVariant records to populate. +If you do not handle the event, the built-in mapping tries the configured SKU +strategy first and then only falls back to barcode when the Shop's +`Find Mapping by Barcode` setting is enabled. Subscribers that depend on barcode +identity should be explicit rather than assuming the fallback is always active. + +*Updated: 2026-07-29 -- barcode fallback is now a shop-controlled mapping behavior* + ## Customizing item creation (import) - `OnBeforeCreateItem` / `OnAfterCreateItem` -- fire around the creation of @@ -34,7 +48,7 @@ overrides. - `OnBeforeFindItemTemplate` / `OnAfterFindItemTemplate` -- control which Item Template is used when auto-creating items. -## Customizing product body HTML (export) +## Customizing product body html (export) - `OnBeforeCreateProductBodyHtml` -- completely replace the HTML generation. Set Handled and write your own ProductBodyHtml. @@ -78,7 +92,26 @@ record is deleted in BC. Implementations: `ShpfyRemoveProductDoNothing`, `ShpfyToArchivedProduct`, `ShpfyToDraftProduct`. Chosen via the "Action for Removed Products" enum on the Shop. Not extensible (enum is non-extensible). -**ICreateProductStatusValue** -- returns the initial Shopify status (Active or -Draft) when creating a new product from a BC Item. Implementations: -`ShpfyCreateProdStatusActive`, `ShpfyCreateProdStatusDraft`. Chosen via the -"Status for Created Products" enum on the Shop. +**ICreateProductStatusValue** -- returns the initial Shopify status when creating +a new product from a BC Item. Implementations are Active, Draft, and Unlisted: +`ShpfyCreateProdStatusActive`, `ShpfyCreateProdStatusDraft`, and +`ShpfyCreateProdStatusUnlisted`. Chosen via the "Status for Created Products" +enum on the Shop. The enum is non-extensible, so partners can select the built-in +strategies but cannot add another creation-status value through enum extension. + +*Updated: 2026-07-29 -- Unlisted creation status implementation added* + +## Public product facade + +`Shpfy Product` (codeunit 30234) is a public facade over the internal +add-to-Shopify flow. It exposes the same high-level operations used by the Item +Card: confirming the target shop, adding an Item to Shopify, validating +attribute-as-option compatibility, resolving product URLs, and opening the +product overview. + +Use this facade when an extension needs "Add to Shopify" parity without reaching +into internal codeunits such as `Shpfy Sync Products` or `Shpfy Product Export`. +The facade preserves the same checks as the UI, including shop selection and +attribute option validation. + +*Updated: 2026-07-29 -- public facade procedures exposed for add-to-Shopify parity* diff --git a/src/Apps/W1/Shopify/App/src/Shipping/docs/CLAUDE.md b/src/Apps/W1/Shopify/App/src/Shipping/docs/CLAUDE.md index 56e97193355..76a7f623d52 100644 --- a/src/Apps/W1/Shopify/App/src/Shipping/docs/CLAUDE.md +++ b/src/Apps/W1/Shopify/App/src/Shipping/docs/CLAUDE.md @@ -8,7 +8,13 @@ The `Shpfy Sync Shipm. to Shopify` report drives the process. It finds unsynced Tracking information is assembled from the BC Shipping Agent record. The `Shpfy Tracking Company` enum on the Shipping Agent table extension maps BC agents to Shopify-recognized carriers. If the agent has an Internet Address, the tracking URL is resolved from it; otherwise, subscribers can override via `OnBeforeRetrieveTrackingUrl`. The `ShpfyShipmentMethodMapping` table maps Shopify delivery method names to BC Shipment Method Codes, Shipping Agents, and shipping charge G/L accounts or items. -Shipping charges are imported on the order side by `ShpfyShippingCharges`, which pulls `shippingLines` from the order and populates `Shpfy Order Shipping Charges`. Any new shipping method title seen during import is auto-created as an unmapped entry in the mapping table. +Shipping charges are imported on the order side by `ShpfyShippingCharges`, which pulls `shippingLines` from the order and populates `Shpfy Order Shipping Charges`. Each shipping line's `taxLines` are stored in `Shpfy Order Tax Line` with the shipping line id as `Parent Id`, separate from order-level and item-line tax lines. This keeps freight tax attached to the shipping charge that becomes the BC freight sales line, so VAT on freight can be inspected with the charge instead of being flattened into order or item taxes. Any new shipping method title seen during import is auto-created as an unmapped entry in the mapping table. + +*Updated: 2026-07-29 -- Shipping charge tax lines are now imported and linked to the Shopify shipping line* + +Shipping method discovery now branches by Shopify shop features. Shops with `marketDrivenShipping` still create `Shpfy Shipment Method Mapping` records, but `ShpfyShippingMethods` reads active option definitions from Markets instead of legacy delivery profiles. The market query reads 25 markets at a time, stores the last market cursor in the `After` parameter, then switches from `Shipping_GetMarketShippingMethods` to `Shipping_GetNextMarketShippingMethods` until `data.markets.pageInfo.hasNextPage` is false. + +*Updated: 2026-07-29 -- Market-driven shipping methods are fetched from Markets with cursor pagination* ## Things to know @@ -18,3 +24,5 @@ Shipping charges are imported on the order side by `ShpfyShippingCharges`, which - The `Shpfy Tracking Company` enum on Shipping Agent controls whether Shopify gets a recognized carrier name or the free-text agent name. - A Fulfillment Id of -1 on the Sales Shipment Header means the export failed; -2 means no applicable lines. - `ShpfyShippingEvents` exposes two integration events: `OnBeforeRetrieveTrackingUrl` and `OnGetNotifyCustomer` for extensibility. +- Shipping charge tax lines reuse `Shpfy Order Tax Line`. The parent can be an order id, an order line id, or a shipping line id; the tax line table resolves the order currency through whichever parent record exists. +- Market-driven shipping skips disabled markets, inherited shipping configurations, and carrier-calculated options without a static name. Only active named options become shipment method mapping rows. diff --git a/src/Apps/W1/Shopify/App/src/Transactions/docs/CLAUDE.md b/src/Apps/W1/Shopify/App/src/Transactions/docs/CLAUDE.md index 438ef923a06..c539cb43b67 100644 --- a/src/Apps/W1/Shopify/App/src/Transactions/docs/CLAUDE.md +++ b/src/Apps/W1/Shopify/App/src/Transactions/docs/CLAUDE.md @@ -4,7 +4,9 @@ Tracks individual payment transactions on Shopify orders -- captures, authorizat ## How it works -`ShpfyTransactions.UpdateTransactionInfos` fetches all transactions for a given order via the `GetOrderTransactions` GraphQL query and upserts them into the `Shpfy Order Transaction` table. Each transaction record captures the type (Sale, Authorization, Capture, Void, Refund), status, gateway name, credit card details, dual-currency amounts (shop money and presentment money), and rounding amounts. The gateway and credit card company are auto-registered in their respective lookup tables (`Shpfy Transaction Gateway`, `Shpfy Credit Card Company`), and a `Shpfy Payment Method Mapping` entry is auto-created keyed on shop + gateway + credit card company. +`ShpfyTransactions.UpdateTransactionInfos` fetches all transactions for a given order via the `GetOrderTransactions` GraphQL query and upserts them into the `Shpfy Order Transaction` table. Each transaction record captures the type (Sale, Authorization, Capture, Void, Refund), status, gateway name, credit card details, dual-currency amounts (shop money and presentment money), their currency codes, and rounding amounts. The gateway and credit card company are auto-registered in their respective lookup tables (`Shpfy Transaction Gateway`, `Shpfy Credit Card Company`), and a `Shpfy Payment Method Mapping` entry is auto-created keyed on shop + gateway + credit card company. + +*Updated: 2026-07-29 -- Transaction shop currency is now imported from `amountSet.shopMoney.currencyCode`* The `Shpfy Suggest Payments` codeunit hooks into BC's general journal posting to stamp Shopify Transaction IDs onto Customer Ledger Entries. This creates a traceable link from Shopify payment captures to BC accounting entries. The `Used` FlowField on the transaction table checks whether a CLE with that transaction ID exists. On journal reversal, the transaction ID is cleared from both the original and reversal entries. @@ -17,7 +19,7 @@ The `Shpfy Payment Method Mapping` table is the central configuration point -- u - The `Parent Id` field on `Shpfy Order Transaction` links refund transactions back to their original charge, enabling refund-to-charge tracing. - Payment method mapping is keyed on shop + gateway + credit card company -- the same gateway can map to different BC payment methods depending on the card brand. - The `Gift Card Id` is extracted from the transaction's receipt JSON, not from a direct GraphQL field. -- Transaction amounts include both shop money and presentment money, plus separate rounding amounts for each currency. +- Transaction amounts include both shop money and presentment money, plus separate rounding amounts for each currency. `Currency` comes from `amountSet.shopMoney.currencyCode`; `Presentment Currency` comes from `amountSet.presentmentMoney.currencyCode`. `Shpfy Suggest Payments` then uses the order's processed currency handling to apply either shop amount + rounding or presentment amount + rounding. - The `Manual Payment Gateway` boolean distinguishes manual payment methods (like COD or bank transfer) from automated gateways. - FlowFields on the transaction table provide quick lookups to Sales Document No., Posted Invoice No., and Shopify Order No. for the related order. diff --git a/src/Apps/W1/Shopify/Test/.docs-updated b/src/Apps/W1/Shopify/Test/.docs-updated index 51bbc28924c..4e998c37321 100644 --- a/src/Apps/W1/Shopify/Test/.docs-updated +++ b/src/Apps/W1/Shopify/Test/.docs-updated @@ -1,4 +1,4 @@ # Documentation last updated -commit: 343aa21d1e5737d250aabf05cd9233d15c411cdc -date: 2026-04-08 +commit: d86ecf324ac3682e1d2394fb3ba0425a4b95de24 +date: 2026-07-29 scope: full diff --git a/src/Apps/W1/Shopify/Test/Bulk Operations/docs/CLAUDE.md b/src/Apps/W1/Shopify/Test/Bulk Operations/docs/CLAUDE.md index 99d07325365..1bdc1e7fcd5 100644 --- a/src/Apps/W1/Shopify/Test/Bulk Operations/docs/CLAUDE.md +++ b/src/Apps/W1/Shopify/Test/Bulk Operations/docs/CLAUDE.md @@ -6,7 +6,11 @@ Tests for Shopify's bulk mutation API -- the asynchronous batch processing path The bulk operations system works in stages: upload a JSONL file to a staged URL, submit a bulk mutation referencing that upload, then poll for completion and process results. ShpfyBulkOperationsTest covers this entire lifecycle. It tests sending a bulk mutation (which creates a "Shpfy Bulk Operation" record), sending a second after the first completes, blocking a second while the first is still running, and handling upload failures silently. The HTTP handler dispatches on a `GraphQLResponses` queue that tracks expected request types (StagedUpload, BulkMutation, CurrentOperation). -The revert tests are particularly interesting. When a bulk operation completes, the system downloads a JSONL result file to identify which individual operations succeeded. `TestBulkOperationRevertFailed` creates 4 variants with original prices, submits a bulk price update, and simulates a partial failure -- only variants 1 and 4 appear in the success result (loaded from `BulkOperationResult.txt` with ID substitution). Variants 2 and 3, which are missing from the result, get their prices reverted to the values stored in the `RequestData` JsonArray on the bulk operation record. `TestBulkOperationRevertAll` tests the failure path where the entire operation fails and all variants are reverted. +The revert tests are particularly interesting. When a bulk operation completes, the system downloads a JSONL result file to identify which individual operations succeeded. `TestBulkOperationRevertFailed` creates 4 variants with original prices, submits a bulk price update, and simulates a partial failure -- only variants 1 and 4 appear in the success result (loaded from `BulkOperationResult.txt` with ID substitution). Variants 2 and 3, which are missing from the result, get their prices reverted to the values stored in the `RequestData` JsonArray on the bulk operation record. Current coverage also verifies that legacy request data without `unitCost` still reverts price fields without touching existing unit cost, and that failed variants produce `Shpfy Skipped Record` entries pointing to the linked Item or Item Variant with the Shopify error text. `TestBulkOperationRevertAll` tests the failure path where the entire operation fails and all variants are reverted. + +The send-path tests cover the diagnostic payload as well as the lifecycle. When the shop's Logging Mode is All, the JSONL sent to Shopify is stored on the bulk operation record for troubleshooting; other logging modes leave it empty. The price-update JSONL tests also lock down `compareAtPrice` semantics: clearing it sends JSON `null`, unchanged values are omitted, and positive sale compare-at prices are sent as quoted decimals. + +*Updated: 2026-07-29 -- Added skipped-record surfacing, sent JSONL logging, legacy unitCost tolerance, and compareAtPrice JSONL coverage* ShpfyMockBulkProductCreate implements the `Shpfy IBulk Operation` interface for testing. It provides a productCreate mutation template with placeholder substitution for title, productType, and vendor. Its `RevertFailedRequests` and `RevertAllRequests` methods are no-ops since product creation does not need revert logic. @@ -15,8 +19,12 @@ ShpfyBulkOperationType.EnumExt extends the production `Shpfy Bulk Operation Type ## Things to know - The enum extension approach for injecting test bulk operation types is the same pattern used elsewhere in the connector for interface-based extensibility -- the test app adds its own enum value that maps to a mock implementation of the `Shpfy IBulk Operation` interface. -- Revert logic relies on the `RequestData` JsonArray stored on the bulk operation record, which contains the original field values (price, compareAtPrice, unitCost) for each variant. If a variant ID is not found in the Shopify result file, its values are restored from this array. +- Revert logic relies on the `RequestData` JsonArray stored on the bulk operation record, which contains the original field values (price, compareAtPrice, unitCost) for each variant. If a variant ID is not found in the Shopify result file, its values are restored from this array. Legacy blobs that predate `unitCost` are tolerated by leaving Unit Cost unchanged. - The HTTP handler distinguishes three URL patterns: POST to the staged upload URL (UploadUrlLbl), GET for the bulk operation result file (BulkOperationUrl), and POST to the Shopify GraphQL endpoint. This three-way routing in a single handler is unique to this test area. - `BulkOperationRunning` is a module-level boolean that controls whether the CurrentOperation response returns "RUNNING" or "COMPLETED" -- the test for blocked concurrent operations sets this to true before the second send attempt. -- ShpfyBulkOpSubscriber uses `SingleInstance = true` and `EventSubscriberInstance = Manual` but is not explicitly bound in the test codeunit -- it likely gets auto-bound because the test app's event subscribers for `OnInvalidUser` need to fire during bulk operation sends. +- ShpfyBulkOpSubscriber uses `SingleInstance = true` and `EventSubscriberInstance = Manual`, and it is bound in `ShpfyWebhooksTest`, not in `ShpfyBulkOperationsTest`. The webhook tests are the ones that drive a bulk operation callback without a real Shopify user context, so they need `OnInvalidUser` bypassed. Tests in this folder call the bulk operation code directly and do not need the binding. - The `BulkMessageHandler` confirms the user-facing message text that appears when a bulk operation is submitted, ensuring UX consistency. +- Sent JSONL is intentionally conditional on `Shop."Logging Mode" = All`; tests assert both the stored and non-stored paths so troubleshooting support does not become the default storage behavior. +- Failed bulk price-sync rows surface through `Shpfy Skipped Record`, not just rollback. The tests link one failed Shopify variant to an Item and one to an Item Variant to ensure the user-facing skipped record points at the right BC record. + +*Updated: 2026-07-29 -- Added bulk rollback, logging, and skipped-record gotchas* diff --git a/src/Apps/W1/Shopify/Test/CLAUDE.md b/src/Apps/W1/Shopify/Test/CLAUDE.md index 9632b1bbd05..5499ba8fe96 100644 --- a/src/Apps/W1/Shopify/Test/CLAUDE.md +++ b/src/Apps/W1/Shopify/Test/CLAUDE.md @@ -11,25 +11,30 @@ This app is the test suite for the Shopify Connector. It verifies the connector' The central piece of infrastructure is `ShpfyInitializeTest` (codeunit 139561). It creates a fully configured Shpfy Shop record with randomized codes -- posting groups, VAT setup, customer and item templates, GL accounts, number series, and a dummy customer and item. The shop is cached in a temporary record so subsequent calls in the same session return the same shop instead of creating duplicates. Every domain-specific test codeunit calls `Initialize()` which runs this codeunit exactly once via the `isInitialized` boolean flag pattern. -HTTP mocking uses `[HttpClientHandler]` exclusively -- test codeunits declare a handler procedure that intercepts outbound HTTP requests at the platform level. The handler inspects the request URL (via `InitializeTest.VerifyRequestUrl`), decides what mock response to return, and writes it into `TestHttpResponseMessage`. Mock response payloads come from two sources: JSON built programmatically in helper codeunits (like `ShpfyOrderHandlingHelper.CreateShopifyOrderAsJson`) or loaded from `.resources/` files via `NavApp.GetResource()`. The `.resources/` folder mirrors the test folder structure (e.g., `.resources/Bulk Operations/`, `.resources/Products/`). The earlier `IsTestInProgress` pattern for mocking has been fully replaced by `[HttpClientHandler]` (PR #7204); the only remaining `IsTestInProgress` usage is in the webhooks test, where it serves a different purpose (suppressing task scheduling). For testing interface contracts, the app uses enum extensions -- `ShpfyStockCalculationExt` adds a "Return Const" value that maps to `ShpfyConstToReturn`, a trivial implementation that returns a configurable decimal. Similarly, `ShpfyBulkOperationType` is extended with `AddProduct` backed by `ShpfyMockBulkProductCreate`. +HTTP mocking uses `[HttpClientHandler]` exclusively -- test codeunits declare a handler procedure that intercepts outbound HTTP requests at the platform level. The handler inspects the request URL (via `InitializeTest.VerifyRequestUrl`), decides what mock response to return, and writes it into `TestHttpResponseMessage`. Mock response payloads come from two sources: JSON built programmatically in helper codeunits (like `ShpfyOrderHandlingHelper.CreateShopifyOrderAsJson`) or loaded from `.resources/` files via `NavApp.GetResource()`. The `.resources/` folder mirrors the test folder structure (e.g., `.resources/Bulk Operations/`, `.resources/Products/`, `.resources/Shipping/`). The newer product and shipping mocks cover variant retrieval, product/variant update mutations, and market-driven shipping method responses. The earlier `IsTestInProgress` pattern for mocking has been fully replaced by `[HttpClientHandler]` (PR #7204); the only remaining `IsTestInProgress` usage is in the webhooks test, where it serves a different purpose (suppressing task scheduling). For testing interface contracts, the app uses enum extensions -- `ShpfyStockCalculationExt` adds a "Return Const" value that maps to `ShpfyConstToReturn`, a trivial implementation that returns a configurable decimal. Similarly, `ShpfyBulkOperationType` is extended with `AddProduct` backed by `ShpfyMockBulkProductCreate`. *Updated: 2026-04-08 -- IsTestInProgress mocking fully replaced by HttpClientHandler (PR #7204)* +*Updated: 2026-07-29 -- Resource mocks now include product variant, product update, and shipping-market responses* + The app does not test the Shopify admin UI, webhooks delivery, or actual GraphQL network round-trips. It tests the AL-side logic: data transformation, record creation, field mapping, GraphQL query generation, retry/idempotency handling, and error flows. The boundary is always at the HTTP layer. ## Structure - **Base/** -- Shop initialization (`ShpfyInitializeTest`), core smoke tests (`ShpfyTestShopify` -- currently commented out), filter management tests, checklist and connector guide tests -- **Products/** -- Largest area (14 files). Item creation, product mapping, price calculation, variant handling, collections, sales channels, image sync. Has its own init codeunit `ShpfyProductInitTest`. See [Products/docs/CLAUDE.md](Products/docs/CLAUDE.md) +- **Products/** -- Largest area (15 files). Item creation, product mapping, price calculation, variant API retrieval, collections, sales channels, image sync. Has its own init codeunit `ShpfyProductInitTest`. See [Products/docs/CLAUDE.md](Products/docs/CLAUDE.md) - **Companies/** -- B2B company sync: import, export, mapping, locations, tax ID. Init in `ShpfyCompanyInitialize`. See [Companies/docs/CLAUDE.md](Companies/docs/CLAUDE.md) - **Customers/** -- Customer mapping, export, API query generation, name resolution. Init in `ShpfyCustomerInitTest` - **Inventory/** -- Stock calculation, sync, export with retry/idempotency, location mapping. Uses the `ShpfyInventoryRetryScenario` enum for state-machine testing. See [Inventory/docs/CLAUDE.md](Inventory/docs/CLAUDE.md) - **Bulk Operations/** -- Tests for Shopify's bulk mutation API. Uses `ShpfyBulkOpSubscriber` for event isolation and `ShpfyMockBulkProductCreate` as a mock interface implementation. See [Bulk Operations/docs/CLAUDE.md](Bulk%20Operations/docs/CLAUDE.md) +- **Shipping/** -- Shipping charge, fulfillment, and market-driven shipping method tests. Uses resource-backed responses under `.resources/Shipping/` for market feature detection and method sync +- **Transactions/** -- Currency and amount mapping tests for Shopify order transactions. This is a small area with one codeunit, so it is documented here rather than in its own docs folder - **Order Handling/** -- Order import and processing. `ShpfyOrderHandlingHelper` builds complex JSON order structures programmatically - **Integration/** -- The `ShpfyConstToReturn` codeunit and its enum extension. This is the mock stock calculation implementation - **Helpers/** -- JSON helper tests, Base64/hash tests, and the `ShpfyTestFields` temporary table (used to test the filter codeunit against every AL field type) -- **.resources/** -- Mock JSON response files organized by domain subfolder -- **DisabledTests/** -- JSON files listing test methods disabled due to known bugs (currently bug 621557 disabling 3 price calculation tests) +- **.resources/** -- Mock JSON response files organized by domain subfolder, including Products, Bulk Operations, and Shipping + +*Updated: 2026-07-29 -- Added variant API, Transactions, Shipping resources, and removed the stale app-local DisabledTests folder reference* ## Documentation @@ -46,18 +51,22 @@ The app does not test the Shopify admin UI, webhooks delivery, or actual GraphQL - The `[HttpClientHandler]` procedure must return `false` to indicate it handled the request. Returning `true` means "I didn't handle this, let it through" -- which will hit the block policy and fail. -- Event subscriber codeunits like `ShpfyBulkOpSubscriber` and `ShpfySkippedRecordLogSub` use `EventSubscriberInstance = Manual` and `SingleInstance = true`. They must be explicitly bound (`BindSubscription`) in test setup and unbound after. The bulk operations subscriber bypasses user validation (`OnInvalidUser`) since tests run without a real Shopify user context. +- Manual event subscriber codeunits like `ShpfySkippedRecordLogSub` use `EventSubscriberInstance = Manual` and `SingleInstance = true`; bind and unbind them around the test that needs them. `ShpfyBulkOpSubscriber` is the same kind of subscriber for `OnInvalidUser`, but it is bound in `ShpfyWebhooksTest` rather than in `ShpfyBulkOperationsTest` -- the webhook tests are the ones that drive a bulk operation callback without a real Shopify user context. Look there for the binding pattern. - `ShpfyCustomerInitTest.ModifyFields()` uses RecordRef to prepend "!" to every text field on any record. This is used to create "changed" versions of records for update-query testing without hardcoding field names. -- Disabled tests are tracked in `DisabledTests/*.json` files with the structure `{ bug, codeunitId, CodeunitName, Method }`. The test runner reads these to skip known-failing methods. Currently 3 methods are disabled across 2 files, all linked to bug 621557 (price calculation). +- The app-local `DisabledTests/` folder no longer exists. Disabled-test filtering still exists in the build tooling: `RunTestsInBcContainer.ps1` looks for central `DisabledTests//*.json` folders under the repository base folder and passes those entries to the test runner. Do not add new app-local `Test/DisabledTests/*.json` files. + +*Updated: 2026-07-29 -- DisabledTests guidance now points to the central build-tool mechanism* - The `.resources/` folder is declared in `app.json` under `resourceFolders` and accessed via `NavApp.GetResource()`. If you add a new mock response file, it must go under `.resources/` to be included in the compiled app. -- B2B features (companies, catalogs) are now unconditionally available on all Shopify plans -- the old `"B2B Enabled"` field has been obsoleted. Tests no longer set `Shop."B2B Enabled" := true`. Staff member visibility testing now uses `Shop."Advanced Shopify Plan"` to gate the Staff Members action on the Shop Card page. +- B2B company features are now unconditionally available on all Shopify plans -- the old `"B2B Enabled"` field has been obsoleted. Tests no longer set `Shop."B2B Enabled" := true`. Staff member visibility and Auto Create Catalog validation still use `Shop."Advanced Shopify Plan"`: staff actions are hidden for unsupported plans, enabling Auto Create Catalog errors on non-advanced plans, and downgrading the plan clears Auto Create Catalog. *Updated: 2026-04-08 -- B2B Enabled removed from tests; staff tests use Advanced Shopify Plan* +*Updated: 2026-07-29 -- Auto Create Catalog plan-gating tests now share the Advanced Shopify Plan setup* + - `ShpfyOrderHandlingHelper` is the most complex helper -- it builds complete order JSON structures including nested customer, address, tax, line item, and B2B company data. It also creates real BC records (items, shop locations, Shopify products/variants) as side effects of building the JSON. - The `ShpfyInventoryRetryScenario` enum is a state machine for testing retry logic: `Success`, `FailOnceThenSucceed`, `AlwaysFail`. The test codeunit tracks `CallCount` to verify that retries actually happened (e.g., expecting exactly 4 calls for "always fail" = 1 initial + 3 retries). diff --git a/src/Apps/W1/Shopify/Test/Companies/docs/CLAUDE.md b/src/Apps/W1/Shopify/Test/Companies/docs/CLAUDE.md index 25f55798cae..8a8a87918b0 100644 --- a/src/Apps/W1/Shopify/Test/Companies/docs/CLAUDE.md +++ b/src/Apps/W1/Shopify/Test/Companies/docs/CLAUDE.md @@ -8,7 +8,9 @@ ShpfyCompanyInitialize is the shared data factory. It creates company and locati ShpfyCompanyAPITest validates GraphQL query generation for create and update operations, including payment terms and tax registration IDs. It also tests field extraction from Shopify JSON responses (UpdateShopifyCustomerFields, UpdateShopifyCompanyFields). The HTTP-level tests for update operations use a simple handler that returns empty JSON and tracks which queries were executed. -ShpfyCompanyExportTest covers converting a BC Customer into Shopify company and location records via FillInShopifyCompany/FillInShopifyCompanyLocation, verifying address field mapping, payment terms propagation, and county/province handling (provinces are only sent for countries that have them -- tested by switching between US and DE). +ShpfyCompanyExportTest covers converting a BC Customer into Shopify company and location records via FillInShopifyCompany/FillInShopifyCompanyLocation, verifying address field mapping, payment terms propagation, and county/province handling. Province data is only sent for countries that have provinces, and country export uses the ISO code when BC's local Country/Region Code differs from Shopify's expected code. + +*Updated: 2026-07-29 -- Company location export now covers ISO country-code mapping for Tax Area lookup and Shopify payloads* ShpfyCompanyImportTest tests the inbound flow: importing a company with locations from Shopify (including payment terms and tax registration IDs), creating customers from companies, and updating customers from companies. It verifies all location fields are correctly imported from a dictionary-driven mock response. @@ -21,8 +23,13 @@ ShpfyCompanyLocationsTest tests creating company locations from customers, inclu - B2B features are now unconditionally available on all Shopify plans. The old `Shop."B2B Enabled" := true` assignments have been removed from `ShpfyCompanyExportTest`, `ShpfyCompanyImportTest`, and `ShpfyCompanyLocationsTest`. Tests no longer need to set any plan-gating flag for B2B functionality. *Updated: 2026-04-08 -- B2B Enabled removed from company tests; B2B features now unconditional* + - ShpfyCompanyInitialize.ModifyFields uses RecordRef/FieldRef reflection to generically modify all text fields, so tests can detect whether update queries include changed fields without manually setting each one. - ShpfyTaxIdMappingTest uses manual event subscriber binding (BindSubscription) on `OnBeforeValidateVATRegistrationNo` to bypass localization-specific VAT validation that would otherwise reject test data. - Company mapping tests exercise two distinct code paths: FindMapping (inbound -- does this Shopify company match a BC customer?) and DoMapping (outbound -- which customer should this company sync to?). - The CompanyImport HTTP handler builds location responses dynamically from a Dictionary of [Text, Text] via `CompanyInitialize.CreateLocationResponse`, which constructs a nested JSON structure with billing address, payment terms, and tax registration ID. +- Company export tests now include the `EL` to `GR` style mismatch: the BC customer keeps its local Country/Region Code, but the Shopify company location and `Shpfy Tax Area` lookup use the ISO code. + +*Updated: 2026-07-29 -- Added ISO country-code export gotcha* + - ShpfyCompanyLocationsTest verifies skip behavior by checking `Shpfy Skipped Record` entries with expected reason text, rather than asserting on error messages. diff --git a/src/Apps/W1/Shopify/Test/Products/docs/CLAUDE.md b/src/Apps/W1/Shopify/Test/Products/docs/CLAUDE.md index 2c071bb5d3e..60cef782040 100644 --- a/src/Apps/W1/Shopify/Test/Products/docs/CLAUDE.md +++ b/src/Apps/W1/Shopify/Test/Products/docs/CLAUDE.md @@ -6,7 +6,11 @@ Tests for the bidirectional product sync between Business Central items and Shop The tests split into two layers. The lower layer is pure unit tests that exercise business logic without HTTP: ShpfyCreateItemTest covers all SKU mapping modes (Item No., Variant Code, Item No. + Variant Code, Vendor Item No., Bar Code) for both single-variant and multi-variant products, including FCY-to-LCY currency conversion. ShpfyCreateProductTest verifies the reverse direction -- building temporary Shopify product/variant records from BC items -- and checks that shop settings like "Sync Item Extended Text" and "Sync Item Attributes" control which HTML sections appear in the product description. ShpfyProductMappingTest exercises FindMapping across all SKU mapping modes. ShpfyProductPriceCalcTest validates price calculation with and without extended pricing (V16+), including line discount application when a Customer Discount Group is configured. ShpfyItemReferenceMgtTest covers CRUD and lookup operations on the Item Reference table for barcodes and vendor item numbers. -The upper layer uses HttpClientHandler mocking to simulate Shopify API calls. ShpfyCreateItemAPITest tests the full CreateItemFromShopifyProduct flow including error handling -- it uses manual event subscriber binding (BindSubscription/UnbindSubscription) on OnBeforeCreateItem to force failures, and verifies that errors are logged on the product record and cleared on success. ShpfyCreateItemVariantTest tests adding items as variants to existing products, covering option retrieval, non-default option names, multi-option rejection, and same-item deduplication. ShpfyProductCollectionTest and ShpfySalesChannelTest verify import/sync of collections and sales channels from Shopify, plus product publishing to default channels on create. ShpfySyncVariantImagesTest covers bidirectional variant image sync (import from Shopify, upload to Shopify, update existing). ShpfyItemAttrAsOptionTest validates the "Item Attributes as Shopify Product Options" feature including mutual exclusivity with UoM-as-variant, validation of missing/duplicate attribute combinations, and export with up to 3 option attributes. +The upper layer uses HttpClientHandler mocking to simulate Shopify API calls. ShpfyCreateItemAPITest tests the full CreateItemFromShopifyProduct flow including error handling -- it uses manual event subscriber binding (BindSubscription/UnbindSubscription) on OnBeforeCreateItem to force failures, and verifies that errors are logged on the product record and cleared on success. It also covers HS code and country-of-origin import when the shop enables `Sync HS Code and Country`, including matching Shopify's separator-less harmonized system code to a BC Tariff Number. ShpfyCreateItemVariantTest tests adding items as variants to existing products, covering option retrieval, non-default option names, multi-option rejection, and same-item deduplication. ShpfyVariantAPITest focuses on `RetrieveShopifyVariant`: it distinguishes a real missing `productVariant` node from a stale Shopify `updatedAt` timestamp so locally newer variants are not deleted. ShpfyProductCollectionTest and ShpfySalesChannelTest verify import/sync of collections and sales channels from Shopify, plus product publishing to default channels on create. ShpfySyncVariantImagesTest covers bidirectional variant image sync (import from Shopify, upload to Shopify, update existing). ShpfyItemAttrAsOptionTest validates the "Item Attributes as Shopify Product Options" feature including mutual exclusivity with UoM-as-variant, validation of missing/duplicate attribute combinations, and export with up to 3 option attributes. + +Product export coverage now includes HS code and country-of-origin propagation from items to temporary Shopify variants, child-item variant mapping safety, and the price-sync threshold that keeps small price changes on individual synchronous mutations instead of creating a bulk operation. + +*Updated: 2026-07-29 -- Added variant API, HS/country, child-item variant, and price-sync threshold coverage* Two helper codeunits -- ShpfyProductInitTest and ShpfyProdCollectionHelper (plus ShpfySalesChannelHelper) -- provide shared test data factories. ShpfyProductInitTest is particularly important: it creates items with templates, extended text, item attributes, variants, and item references, and builds Shopify product/variant records with SKU values driven by the shop's SKU mapping mode. @@ -17,5 +21,9 @@ Two helper codeunits -- ShpfyProductInitTest and ShpfyProdCollectionHelper (plus - ShpfyCreateItemAPITest uses `EventSubscriberInstance = Manual` with BindSubscription/UnbindSubscription to inject OnBeforeCreateItem errors only in specific tests -- the subscriber raises `Error(CreateItemErr)` to simulate item creation failure. - Price calculation tests must toggle extended pricing via `LibraryPriceCalculation.DisableExtendedPriceCalculation()` / `EnableExtendedPriceCalculation()` because the two pricing engines use different table structures (Sales Price vs Price List Line). - ShpfyItemAttrAsOptionTest is organized into `#region` blocks by variant/attribute combination (no variants + no attributes, no variants + 2 attributes, 2 variants + 3 attributes, etc.), which makes navigation easier. -- Mock API responses are loaded from resource files under `Products/` (e.g., `ProductDetailsResponse.txt`, `CreatedVariantResponse.txt`) with placeholder substitution for dynamic IDs using `StrSubstNo`. +- Mock API responses are loaded from resource files under `Products/` (e.g., `ProductDetailsResponse.txt`, `CreatedVariantResponse.txt`, `ProductVariantDetailsResponse.txt`, `ProductUpdateResponse.txt`) with placeholder substitution for dynamic IDs using `StrSubstNo`. The variant details response includes inventory-item HS code and country-of-origin fields for import coverage. - The `TestHttpRequestPolicy = BlockOutboundRequests` attribute on API test codeunits ensures no real HTTP calls escape -- the handler must explicitly return `false` to suppress the actual call. + +- `ShpfyVariantAPITest` is intentionally narrow: a local variant with `Updated At` ahead of Shopify should still be reported as existing, while a null `productVariant` node means the caller may delete the local record. + +*Updated: 2026-07-29 -- Added variant API and product resource gotchas* diff --git a/src/Apps/W1/Shopify/Test/docs/business-logic.md b/src/Apps/W1/Shopify/Test/docs/business-logic.md index c68f2bcff20..472ca0e2731 100644 --- a/src/Apps/W1/Shopify/Test/docs/business-logic.md +++ b/src/Apps/W1/Shopify/Test/docs/business-logic.md @@ -62,7 +62,9 @@ flowchart TD There are two distinct patterns for building mock responses: -**Resource-based responses** are used when the response structure is stable and complex. The bulk operations tests load responses from `.resources/Bulk Operations/StagedUploadResult.txt`, `.resources/Bulk Operations/BulkMutationResponse.txt`, etc. These files contain JSON templates with `%1`, `%2` placeholders that are filled via `StrSubstNo`. This keeps large JSON payloads out of AL code. +**Resource-based responses** are used when the response structure is stable and complex. The bulk operations tests load responses from `.resources/Bulk Operations/StagedUploadResult.txt`, `.resources/Bulk Operations/BulkMutationResponse.txt`, etc. Product export and variant retrieval tests load product update and variant detail responses from `.resources/Products/`, and shipping tests load market feature and market-method responses from `.resources/Shipping/`. These files contain JSON templates with placeholders that are filled via `StrSubstNo` or token replacement. This keeps large JSON payloads out of AL code. + +*Updated: 2026-07-29 -- Resource-based mocks now include product variant, product update, and shipping market responses* **Programmatic responses** are used when the response content depends on test parameters. `ShpfyOrderHandlingHelper.CreateShopifyOrderAsJson()` builds an entire order JSON structure in AL, computing prices, tax, and discount amounts based on the order amount from `OrdersToImport`. Similarly, `ShpfyCustomerInitTest.DummyJsonCustomerObjectFromShopify()` builds customer JSON with parameterized IDs and today's date. @@ -99,6 +101,25 @@ flowchart TD This two-phase approach (extract-then-import) mirrors how the real connector works: it first gets a lightweight list of orders to import, then fetches full details for each one. The test helper preserves this separation so the connector's actual import codeunits are exercised. +## Transaction import test flow + +`ShpfyTransactionsTest` is a small area, so it is documented at the app level. It verifies how the transaction importer maps Shopify `shopMoney` and `presentmentMoney` into BC transaction fields, and how currency codes are resolved. LCY is represented by an empty BC currency code, ISO Code matches take precedence, Code matches are the fallback, and a resolved LCY currency also returns blank. + +```mermaid +flowchart TD + A[Test creates order header with unique Shopify order id] --> B[Enqueue shop and presentment money values] + B --> C[Transactions.UpdateTransactionInfos] + C --> D[HttpClientHandler returns one transaction JSON] + D --> E[Importer translates currency codes] + E --> F{Resolved currency equals LCY?} + F -->|Yes| G[Store blank BC currency code] + F -->|No| H[Store resolved Currency.Code] + G --> I[Assert amount and presentment fields] + H --> I +``` + +*Updated: 2026-07-29 -- Added Transactions test flow for currency and amount mapping* + ## Access token registration Tests that make HTTP calls (even mocked ones) need a registered access token because the connector checks for it before making API calls. `ShpfyInitializeTest.RegisterAccessTokenForShop()` creates a `Shpfy Registered Store New` record with a comprehensive scope string and a random token value. This is called in the `Initialize()` of test codeunits that use `[HttpClientHandler]`, like `ShpfyInventoryExportTest` and `ShpfyBulkOperationsTest`. diff --git a/src/Apps/W1/Shopify/Test/docs/patterns.md b/src/Apps/W1/Shopify/Test/docs/patterns.md index 268d39072d4..239b3ca86d9 100644 --- a/src/Apps/W1/Shopify/Test/docs/patterns.md +++ b/src/Apps/W1/Shopify/Test/docs/patterns.md @@ -20,7 +20,9 @@ Nearly every test codeunit uses a module-level `isInitialized: Boolean` that gat Complex or stable JSON response payloads are stored as `.txt` files under `.resources/` and loaded via `NavApp.GetResource()`. This avoids embedding large JSON strings in AL code and makes it easy to update response formats. -**Example**: In `ShpfyBulkOperationsTest`, the `BulkOperationHttpHandler` loads different resources based on which GraphQL operation was requested. It uses `Library - Variable Storage` as a queue (`GraphQLResponses.Enqueue('StagedUpload')`) to control which response to return for each sequential API call. The handler dequeues the next expected response type and loads the corresponding resource file. +**Example**: In `ShpfyBulkOperationsTest`, the `BulkOperationHttpHandler` loads different resources based on which GraphQL operation was requested. It uses `Library - Variable Storage` as a queue (`GraphQLResponses.Enqueue('StagedUpload')`) to control which response to return for each sequential API call. The handler dequeues the next expected response type and loads the corresponding resource file. Product export tests use the same pattern for `ProductUpdateResponse.txt` and `ProductVariantsBulkUpdateResponse.txt`, while shipping tests use it for market-driven feature detection before loading market shipping methods. + +*Updated: 2026-07-29 -- Product export and shipping tests now use additional resource-backed mock responses* **Gotcha**: Resource files use `%1`, `%2` placeholders filled by `StrSubstNo`. If your response needs a literal `%` character, you must escape it. Also, resource files must be under the `.resources/` path declared in `app.json`'s `resourceFolders` -- putting them elsewhere means they won't be compiled into the app. @@ -58,11 +60,43 @@ All Shopify GraphQL requests hit the same endpoint (`/admin/api/graphql.json`), ## Disabled tests tracking -Known-failing test methods are tracked in JSON files under `DisabledTests/` rather than being deleted or commented out. Each entry records the bug ID, codeunit ID, codeunit name, and method name. +Known-failing test methods are still tracked as JSON, but not inside the Shopify test app folder. The local `Test/DisabledTests/` directory was removed; `RunTestsInBcContainer.ps1` now searches for central `DisabledTests//*.json` folders and passes those entries to the test runner. For this app, the app-name folder is `Shopify_Connector_Test`. + +**Example**: A central entry contains `codeunitId`, `codeunitName`, and `method`; the build script derives the folder name by replacing spaces in the app name with underscores. + +**Gotcha**: Do not add new `Test/DisabledTests/*.json` files. If you rename a test method or codeunit, update the central app-name disabled-test file, otherwise the filter will not match the intended method. + +*Updated: 2026-07-29 -- App-local DisabledTests JSON files were removed and the build uses the central app-name folder mechanism* + +## Price calculation state isolation + +Price calculation tests are sensitive to cached state because `Shpfy Product Price Calc.` is SingleInstance and keeps temporary sales context. Tests that depend on Work Date must change `WorkDate()` deliberately, call `SetShop()` for each calculation, and restore the original Work Date before asserting so failures do not leak state into later tests. + +**Example**: `ShpfyProductPriceCalcTest.UnitTestCalcPriceUsesCurrentWorkDate` creates two dated All Customers discounts around a boundary date, calculates once after the boundary and once before it, and proves the second call uses the current Work Date instead of a stale cached document date. + +**Gotcha**: Legacy price tests that exercise Sales Price / Sales Line Discount data should explicitly choose the V15 handler with `LibraryPriceCalculation.SetupDefaultHandler("Price Calculation Handler"::"Business Central (Version 15.0)")`. Disabling extended pricing is not enough when the tenant defaults have already moved to the newer price engine. + +*Updated: 2026-07-29 -- Price calculation tests now guard against stale Work Date and explicit legacy handler selection* + +## Suppressing duplicate-contact confirmations + +Customer creation and update tests can cascade into contact creation. In demo or localized data, Marketing Setup may have duplicate-contact auto-search enabled, which raises an interactive confirm while tests run with UI handlers enabled. + +**Example**: `ShpfyCreateCustomerTest.Initialize()` turns off `Marketing Setup."Autosearch for Duplicates"` and `"Maintain Dupl. Search Strings"` before committing setup. The tests then verify no-default-address customer creation and update paths without needing a confirm handler. + +**Gotcha**: Prefer disabling the data-dependent confirmation at setup time for these integration tests. A confirm handler would hide the source of the prompt and could accidentally approve unrelated confirmations. + +*Updated: 2026-07-29 -- Customer creation tests now avoid duplicate-contact confirms through setup data* + +## GraphQL rate-limit elapsed-time checks + +The rate limiter tests measure elapsed time, so they use broad tolerances and explicit sleeps only where the production behavior is time-based. + +**Example**: `ShpfyGraphQLRateLimitTest.UnitTestWaitForRequestAvailableAfterElapsedTime` sets availability below the next query cost, sleeps for part of the restore interval, then asserts that `WaitForRequestAvailable` waits only the remaining time. -**Example**: `DisabledTests/ShpfyProductPriceCalcTest.json` disables `UnitTestCalcPriceTest` from codeunit 139605 with bug reference 621557. +**Gotcha**: These tests should verify a range around the expected wait rather than exact milliseconds. The purpose is to catch under-waiting or restarting the full wait after elapsed time, not scheduler jitter. -**Gotcha**: The disabled tests mechanism is read by the test runner infrastructure. If you rename a test method or codeunit, update the JSON file too, otherwise the filter won't match and the test will run (and presumably fail). +*Updated: 2026-07-29 -- Rate-limit tests now cover elapsed restore time* ## Legacy patterns From 1b8465eff867e0da8910d924f7643e9db51717bc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Magnus=20Hartvig=20Gr=C3=B8nbech?= Date: Wed, 29 Jul 2026 10:21:49 +0200 Subject: [PATCH 2/3] [E-Document] Refresh connector documentation and update Privacy Notice + API Reports docs Brings the in-repo AL documentation up to date with the code. Documentation only. No AL source changes. E-Document Core (main part of this change): The E-Document docs were created in March (a090c078, #7255) and only three files have been touched since. 55 commits and 149 AL files changed in that window, so the 18 module docs had drifted badly. - Import V2 is now the default for new services, with the draft format selected on the service. Documented the pipeline stages, the reader interfaces, and how a partner plugs in a new inbound format - Data Exchange definitions moved from labels to app resource files, and the v2 bridge lets those definitions feed the V2 pipeline. Access = Internal was removed from the Data Exchange Impl. codeunit - PEPPOL now lives in a standalone app under src/Apps/W1/PEPPOL. Added the EDI flows: purchase order send path, sales order receive path, and order response message handling - New structured format readers, including XRechnung and OIOUBL - Corrected documentation that still described the E-Document Purchase Order Matching Copilot as current. It is obsoleted; AI-assisted line matching now happens at import time via "E-Doc. AI Tool Processor" - Document date, due date and configurable default posting date behaviour, plus VAT rate resolution and VAT amount difference - E-Document Purchase Header/Line exposed for country-app consumers, and Sales Header/Line data sensitivity classification - Confirmed the E-Document Header/Line Mapping tables are retained for compatibility only and are not part of the V2 draft flow Privacy Notice: - Registered the Microsoft Copilot privacy notice, whose default approval is keyed to the EU Data Boundary, so EEA environments start unapproved - Noted the inherent permissions on the facade that let service-invoked callers reach it API Reports - Finance: - The six API queries now carry AboutText consumed by MCP tooling for tool selection Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/Apps/W1/APIReportsFinance/App/CLAUDE.md | 8 + src/Apps/W1/EDocument/App/CLAUDE.md | 21 +- .../W1/EDocument/App/docs/business-logic.md | 44 ++-- src/Apps/W1/EDocument/App/docs/data-model.md | 33 ++- .../W1/EDocument/App/docs/extensibility.md | 44 +++- src/Apps/W1/EDocument/App/docs/patterns.md | 36 +++- .../App/src/DataExchange/docs/CLAUDE.md | 12 +- .../EDocument/App/src/Document/docs/CLAUDE.md | 26 ++- .../App/src/Document/docs/data-model.md | 38 +++- .../App/src/Extensions/docs/CLAUDE.md | 28 ++- .../EDocument/App/src/Format/docs/CLAUDE.md | 17 +- .../EDocument/App/src/Helpers/docs/CLAUDE.md | 14 +- .../App/src/Processing/Import/docs/CLAUDE.md | 46 +++- .../Processing/Import/docs/business-logic.md | 197 +++++++++++++---- .../src/Processing/Import/docs/data-model.md | 77 +++++-- .../Processing/Import/docs/extensibility.md | 201 +++++++++++------- .../App/src/Processing/docs/CLAUDE.md | 34 ++- .../App/src/Processing/docs/business-logic.md | 132 ++++++++---- .../App/src/Service/docs/data-model.md | 26 +++ .../W1/EDocument/App/src/Setup/docs/CLAUDE.md | 8 +- .../EDocument/App/src/Workflow/docs/CLAUDE.md | 8 +- .../App/Privacy Notice/CLAUDE.md | 31 ++- 22 files changed, 804 insertions(+), 277 deletions(-) diff --git a/src/Apps/W1/APIReportsFinance/App/CLAUDE.md b/src/Apps/W1/APIReportsFinance/App/CLAUDE.md index 27f2d9eb065..395d0bbf73d 100644 --- a/src/Apps/W1/APIReportsFinance/App/CLAUDE.md +++ b/src/Apps/W1/APIReportsFinance/App/CLAUDE.md @@ -28,6 +28,14 @@ Every object follows the same template: `PageType = API` or `QueryType = API`, = SystemId`. The app defines no tables of its own -- it reads directly from base app tables (G/L Entry, Cust. Ledger Entry, etc.). +The six queries additionally carry an `AboutText` describing what the dataset +contains and what it is useful for. This is consumed by MCP tooling, which lists +API queries as callable tools and uses the text to decide when a query is +relevant. Treat `AboutText` as a functional description aimed at an agent, not a +caption -- vague wording degrades tool selection. The API pages do not carry it. + +*Updated: 2026-07-29 -- AboutText added to the queries for MCP Query API tool support* + The only procedural logic lives in two page triggers: - **Accounting Periods** (`APIFinanceAccPeriods.Page.al`): computes diff --git a/src/Apps/W1/EDocument/App/CLAUDE.md b/src/Apps/W1/EDocument/App/CLAUDE.md index 5104ddfb242..e8419e7469a 100644 --- a/src/Apps/W1/EDocument/App/CLAUDE.md +++ b/src/Apps/W1/EDocument/App/CLAUDE.md @@ -1,6 +1,6 @@ # E-Document Core -E-Document Core is the framework for electronic document exchange in Business Central. It converts outbound sales/service/purchase documents into structured formats (PEPPOL BIS 3.0, Data Exchange, or custom), transmits them through configurable service integrations, and processes inbound electronic documents into purchase invoices, credit memos, or journal lines. The framework is interface-driven: it defines the contracts, and connector apps provide the actual service communication. +E-Document Core is the framework for electronic document exchange in Business Central. It converts outbound sales/service/purchase documents into structured formats (Data Exchange, PEPPOL-backed formats, or custom), transmits them through configurable service integrations, and processes inbound electronic documents into purchase invoices, credit memos, sales orders, or journal lines. The framework is interface-driven: it defines the contracts, and connector or format apps provide the actual service communication and specialized format logic. See [README.md](README.md) for detailed interface implementation examples with code samples. @@ -14,11 +14,13 @@ Everything starts with the **Document Sending Profile**. When a BC document is p An **E-Document Service** (`EDocumentService.Table.al`) combines a **Document Format** (how to serialize, e.g. PEPPOL BIS 3.0) with a **Service Integration** (where to send/receive, e.g. a connector app). Both are enum-based with interface implementations, so you extend them by adding enum values that bind to your interface implementation -- classic strategy pattern via AL enums. -Outbound documents flow through: check on release -> create E-Document record on post -> export to blob via the "E-Document" interface -> send via IDocumentSender -> optionally poll for async response via IDocumentResponseHandler. Inbound documents have a V2.0 multi-stage pipeline: receive document list -> download each document -> structure (convert PDF to structured data via ADI or passthrough for XML) -> read into draft staging tables -> prepare draft (resolve vendors, items, accounts) -> finish draft (create actual BC purchase document). Each stage is independently reversible. +Outbound documents flow through: check on release -> create E-Document record on post or purchase order release -> export to blob via the "E-Document" interface -> send via IDocumentSender -> optionally poll for async response via IDocumentResponseHandler. Inbound documents use the V2.0 multi-stage pipeline by default for new services: receive document list -> download each document -> structure (convert PDF to structured data via ADI/MLLM or passthrough for XML) -> read into draft staging tables -> prepare draft (resolve vendors/customers, items, accounts) -> finish draft (create the BC purchase document or sales order). Each stage is independently reversible. -The framework does not provide any service connector itself. The `"Service Integration"` enum ships with only `"No Integration"`. Connector apps (like E-Document Connector for Pagero or Avalara) extend this enum and implement IDocumentSender, IDocumentReceiver, and other integration interfaces. Similarly, the `"E-Document Format"` enum ships with two built-in values: `"Data Exchange"` (using BC Data Exchange Definitions) and `"PEPPOL BIS 3.0"`, but can be extended. +The framework does not provide any service connector itself. The `"Service Integration"` enum ships with only `"No Integration"`. Connector apps (like E-Document Connector for Pagero or Avalara) extend this enum and implement IDocumentSender, IDocumentReceiver, and other integration interfaces. The `"E-Document Format"` enum still exposes `"Data Exchange"` and `"PEPPOL BIS 3.0"`, but Core now depends on the standalone PEPPOL app for PEPPOL XML builders, validation interfaces, purchase order export, and order response payload construction. -Three status dimensions track document lifecycle: **E-Document Status** (In Progress / Processed / Error -- the overall state), **E-Document Service Status** (per-service granular status like Exported, Sent, Pending Response, Approved, etc.), and **Import Processing Status** (the V2.0 import pipeline stage: Unprocessed -> Readable -> Ready for draft -> Draft Ready -> Processed). +Three status dimensions track document lifecycle: **E-Document Status** (In Progress / Processed / Error -- the overall state), **E-Document Service Status** (per-service granular status like Exported, Sent, Pending Response, Approved, etc.), and **Import Processing Status** (the V2.0 import pipeline stage: Unprocessed -> Readable -> Ready for draft -> Draft Ready -> Processed). Lifecycle messages such as PEPPOL Order Responses are stored separately from BC documents and linked back to the parent E-Document. + +*Updated: 2026-07-29 -- clarified V2.0 defaults, PEPPOL app dependency, sales order import, and message handling.* ## Structure @@ -26,15 +28,15 @@ Three status dimensions track document lifecycle: **E-Document Status** (In Prog src/ ClearanceModel/ -- Tax authority QR code clearance on posted invoices/credit memos ControlAddIn/ -- PDF Viewer browser control add-in - DataExchange/ -- Data Exchange Definition format impl + PEPPOL pre-mapping + DataExchange/ -- Data Exchange Definition format impl, resource import, and v2 bridge support Document/ -- Core E-Document table, status model, direction/type enums, notification Extensions/ -- Table/page extensions hooking into BC sales, purchase, service documents - Format/ -- PEPPOL BIS 3.0 export/import implementation + Format/ -- Core format adapters, including PEPPOL BIS 3.0 bindings over the PEPPOL app Helpers/ -- Utilities: error handling, JSON helpers, logging, blob processing Integration/ -- Service integration framework: send/receive/action interfaces, runners, context Logging/ -- E-Document Log, Integration Log, Data Storage tables Mapping/ -- Field mapping engine for import/export transformations - Processing/ -- Core orchestration: export, import pipeline, subscribers, order matching, AI, providers + Processing/ -- Core orchestration: export, import pipeline, messages, order matching, AI, providers SampleInvoice/ -- Demo sample invoice generation Service/ -- Service configuration, participants, supported document types Setup/ -- Installation, upgrade, consent management @@ -55,8 +57,11 @@ src/ - The import pipeline implementation fields live directly on the E-Document table: `"Structure Data Impl."`, `"Read into Draft Impl."`, and `"Process Draft Impl."`. These enums determine which interface implementations run at each stage. - `"E-Document Service Status"` enum implements `IEDocumentStatus` interface -- each status value knows whether it means "in progress", "processed", or "error", via `EDocInProgressStatus`, `EDocProcessedStatus`, and `EDocErrorStatus` codeunits. - Batch processing and single-document processing are distinct code paths. Batch mode uses recurrent background jobs configured on the service (fields 21-26 on `"E-Document Service"`). -- The V1.0 import process (`"Import Process" = "Version 1.0"`) collapses all pipeline stages into a single "Finish draft" step. V2.0 is the current architecture, and new services default to it. The import version is selectable on the service card. For V2.0 services the service-level `"Read into Draft Impl."` field ("Draft Format" on the card) selects how already-structured documents (e.g. PEPPOL XML) are read into a draft; it is the fallback used when neither the structuring step nor the receiving integration sets a reader, so leaving it `Unspecified` for such formats makes import fail. +- The V1.0 import process (`"Import Process" = "Version 1.0"`) collapses all pipeline stages into a single "Finish draft" step. V2.0 is the current architecture, and new services default to it. The import version is selectable on the service card. For V2.0 services the service-level `"Read into Draft Impl."` field ("Draft Format" on the card) selects how already-structured documents (for example PEPPOL, Data Exchange Purchase, OIOUBL, or XRechnung XML from dependent format apps) are read into a draft; it is the fallback used when neither the structuring step nor the receiving integration sets a reader, so leaving it `Unspecified` for such formats makes import fail. - `#if not CLEAN26` and `#if not CLEAN27` blocks mark deprecated code scheduled for removal. The old `"E-Document Integration"` enum and its `"Service Integration"` field on the service table are fully replaced by `"Service Integration V2"`. - The framework uses the "if codeunit.run" pattern extensively -- interface calls are wrapped in codeunits that run with error trapping, so a connector failure produces a logged error rather than a crash. - `"E-Document Background Jobs"` manages Job Queue Entries for recurrent import polling and batch send processing. - The `IExportEligibilityEvaluator` interface (field `"Export Eligibility Evaluator"` on the service) lets connector apps control which documents should be exported via their service, beyond the basic document type check. +- Purchase draft staging tables are intentionally consumable by dependent country apps; sales order staging is internal to Core and participates in sensitivity classification with the other E-Document tables. + +*Updated: 2026-07-29 -- added current import format selection, PEPPOL split, and staging visibility notes.* diff --git a/src/Apps/W1/EDocument/App/docs/business-logic.md b/src/Apps/W1/EDocument/App/docs/business-logic.md index 7000297800e..9d9fcfcbc74 100644 --- a/src/Apps/W1/EDocument/App/docs/business-logic.md +++ b/src/Apps/W1/EDocument/App/docs/business-logic.md @@ -6,7 +6,7 @@ This document covers the main processing flows in E-Document Core, the decision ### Trigger: posting a BC document -`EDocumentSubscribers.Codeunit.al` subscribes to posting events across Sales-Post, Purch.-Post, Service-Post, and Gen. Jnl.-Post Line. When a sales invoice is posted, for example, `OnAfterPostSalesDoc` fires. The subscriber also hooks into release events (`OnBeforeReleaseSalesDoc`, etc.) to run pre-flight checks before the user commits to posting. +`EDocumentSubscribers.Codeunit.al` subscribes to posting events across Sales-Post, Purch.-Post, Service-Post, and Gen. Jnl.-Post Line. When a sales invoice is posted, for example, `OnAfterPostSalesDoc` fires. The subscriber also hooks into release events (`OnBeforeReleaseSalesDoc`, etc.) to run pre-flight checks before the user commits to posting. Purchase orders are special: when a released purchase order uses an electronic document sending profile, Core can create the outbound E-Document at release time so the PEPPOL ordering send path does not wait for posting. The subscriber resolves the Document Sending Profile for the document and, if it specifies `"Extended E-Document Service Flow"`, delegates to `EDocExport.Codeunit.al`. @@ -16,7 +16,7 @@ The outbound process has four distinct phases, orchestrated by `EDocExport` and ```mermaid flowchart TD - A[BC document posted] --> B[EDocumentSubscribers catches event] + A[BC document posted or purchase order released] --> B[EDocumentSubscribers catches event] B --> C{Document Sending Profile?} C -->|Not E-Doc| D[Skip] C -->|Extended E-Document Service Flow| E["EDocExport.CheckEDocument()"] @@ -59,7 +59,7 @@ Errors during export or send do not crash the posting transaction. Every interfa `EDocImport.ReceiveAndProcessAutomatically()` is the entry point, typically triggered by a recurrent import Job Queue Entry (configured via `"Auto Import"` on the service). It calls `EDocIntegrationManagement.ReceiveDocuments()`, which invokes `IDocumentReceiver.ReceiveDocuments()` to get a list of document metadata blobs, then for each document calls `DownloadDocument()` to fetch the actual content. If the receiver also implements `IReceivedDocumentMarker`, the framework calls `MarkFetched()` after download so the service knows the document was consumed. -Each successfully downloaded document gets an E-Document record (Direction = Incoming, Status = Imported), with the raw content stored in `"E-Doc. Data Storage"` and linked via `"Unstructured Data Entry No."`. +Each successfully downloaded document gets an E-Document record (Direction = Incoming, Status = Imported), with the raw content stored in `"E-Doc. Data Storage"` and linked via `"Unstructured Data Entry No."`. New services default to `"Import Process" = "Version 2.0"`; users can still select Version 1.0 on the service card when they need the legacy path. ### Import pipeline stages @@ -68,26 +68,37 @@ After receiving, each document goes through the V2.0 pipeline managed by `Import ```mermaid flowchart TD A[Unprocessed] -->|"Structure received data"| B[Readable] - B -->|"Read into Draft"| C["Ready for draft"] - C -->|"Prepare draft"| D["Draft Ready"] - D -->|"Finish draft"| E[Processed] - E -.->|Undo| D - D -.->|Undo| C - C -.->|Undo| B + B -->|"Read into Draft"| C{Draft kind} + C -->|Purchase Invoice or Credit Memo| D[Purchase staging tables] + C -->|Sales Order| E[Sales staging tables] + C -->|Order Response| F[Store lifecycle message and stop import] + D -->|"Prepare draft"| G["Draft Ready"] + E -->|"Prepare draft"| G + G -->|"Finish draft"| H[Processed] + H -.->|Undo| G + G -.->|Undo| B B -.->|Undo| A ``` +*Updated: 2026-07-29 -- reflected the V2.0 draft split for purchase documents, sales orders, and lifecycle messages.* + **Structure received data** (`ImportEDocumentProcess.StructureReceivedData()`): Converts the raw blob into a structured format. The implementation is determined by `"Structure Data Impl."` on the E-Document, which defaults to the preferred implementation for the file format (e.g., PDFs default to Azure Document Intelligence processing). The `IStructureReceivedEDocument` interface returns an `IStructuredDataType` that contains the structured content and specifies how to read it. For already-structured documents (XML), this is a passthrough -- the structured entry number just points to the same storage as the unstructured one. -**Read into Draft** (`ImportEDocumentProcess.ReadIntoDraft()`): The `IStructuredFormatReader.ReadIntoDraft()` method parses the structured data and populates the E-Document Purchase Header/Line staging tables. It returns the `"E-Doc. Process Draft"` enum value that determines which `IProcessStructuredData` implementation runs next. +**Read into Draft** (`ImportEDocumentProcess.ReadIntoDraft()`): The `IStructuredFormatReader.ReadIntoDraft()` method parses the structured data and populates draft staging tables. Purchase invoices and credit memos use E-Document Purchase Header/Line; inbound orders use E-Document Sales Header/Line. The reader returns the `"E-Doc. Process Draft"` enum value that determines which `IProcessStructuredData` implementation runs next. After this step, the service's document format can also provide a lifecycle response message, such as a PEPPOL Order Response acknowledgement for an inbound sales order. + +**Prepare draft** (`ImportEDocumentProcess.PrepareDraft()`): `IProcessStructuredData.PrepareDraft()` resolves BC entities from the parsed data. Purchase drafts find the vendor, match items, resolve units of measure, assign G/L accounts, resolve VAT product posting groups from extracted VAT rates, and compute any allowed VAT amount difference. Vendor resolution records whether the vendor came from an existing assignment, provider lookup, historical assignment, or no match in `E-Doc. Imp. Session Telemetry`. Sales drafts resolve the customer, units of measure, and sales line entities through the sales provider interfaces. The result is a draft with BC-specific field values filled in. + +**Finish draft** (`ImportEDocumentProcess.FinishDraft()`): `IEDocumentFinishDraft.ApplyDraftToBC()` creates the actual BC document from the staging tables. Current Core implementations create Purchase Invoices, Purchase Credit Memos, and Sales Orders. The method returns the RecordId of the created document, which is stored in `"Document Record ID"` on the E-Document. -**Prepare draft** (`ImportEDocumentProcess.PrepareDraft()`): `IProcessStructuredData.PrepareDraft()` resolves BC entities from the parsed data -- finding the vendor, matching items, resolving units of measure, assigning GL accounts. This uses a chain of provider interfaces (`IVendorProvider`, `IItemProvider`, `IUnitOfMeasureProvider`, `IPurchaseLineProvider`, `IPurchaseOrderProvider`). The result is a draft with BC-specific field values filled in. +### Message handling -**Finish draft** (`ImportEDocumentProcess.FinishDraft()`): `IEDocumentFinishDraft.ApplyDraftToBC()` creates the actual BC document (Purchase Invoice, Purchase Credit Memo, etc.) from the staging tables. It returns the RecordId of the created document, which is stored in `"Document Record ID"` on the E-Document. +The V2.0 import pipeline treats lifecycle messages as related records, not as BC source documents. `ReadIntoDraft()` asks the selected document format for a response message type after the draft reader has populated `"Process Draft Impl."`. For PEPPOL sales orders, Core builds an outgoing `"PEPPOL Order Response"` message and stores it in `"E-Document Message"`. If an inbound PEPPOL `OrderResponse` arrives, the PEPPOL reader finds the matching outgoing E-Document by order reference, stores the incoming message on that E-Document, deletes the temporary inbound carrier, and stops the import pipeline. + +*Updated: 2026-07-29 -- added message handling for PEPPOL order responses.* ### Reversibility -Each step can be undone via `UndoProcessingStep()`. Undoing "Finish draft" calls `IEDocumentFinishDraft.RevertDraftActions()`, which de-links the BC document from the E-Document (clears the `"E-Document Link"` field), transfers PO matches and attachments back to the E-Document, but does **not** delete the BC document itself -- it must be handled separately. Undoing "Prepare draft" clears the BC-resolved fields and resets the document type. Undoing "Structure received data" clears the structured data reference. This lets users go back to an earlier stage, correct data, and re-process. +Each step can be undone via `UndoProcessingStep()`. Undoing "Finish draft" calls `IEDocumentFinishDraft.RevertDraftActions()`, which de-links the BC document from the E-Document (clears the `"E-Document Link"` field), transfers PO matches and attachments back to the E-Document, but does **not** delete the BC document itself -- it must be handled separately. Undoing "Prepare draft" clears the BC-resolved fields and resets the document type. Sales order drafts use the same rewind model through their sales staging tables. Undoing "Structure received data" clears the structured data reference. This lets users go back to an earlier stage, correct data, and re-process. ### Automatic vs. manual processing @@ -108,7 +119,11 @@ Beyond send and receive, the framework supports arbitrary actions on documents v - **E-Document Status is derived**: The overall `E-Document Status` is derived from the service statuses. `EDocumentProcessing.ModifyEDocumentStatus()` computes it after every service status change using the `IEDocumentStatus` interface on the enum values. - **Import Processing Status is a FlowField**: On the E-Document table, `"Import Processing Status"` is a FlowField that reads from `"E-Document Service Status"`. You must call `CalcFields` before reading it. - **V1.0 and V2.0 coexistence**: The `"Import Process"` field on the service determines which path runs; new services default to Version 2.0, and the version is selectable on the service card. V1.0 collapses everything into a single "Finish draft" step that calls the old `V1_ProcessEDocument` logic. The pipeline state machine still runs, but only the last step does anything for V1.0 documents. -- **Selecting the V2.0 draft format**: For Version 2.0 services the `"Read into Draft Impl."` field (shown as "Draft Format" on the service card) selects how the inbound structured e-document is read into a draft. For already-structured formats such as PEPPOL XML the structuring step returns `Unspecified`, so the read step falls back to the service's `"Read into Draft Impl."`; if that is also `Unspecified`, the `E-Doc. Unspecified Impl.` runs and import errors with "No method to read the e-document has been provided." PDF services (e.g. Azure Document Intelligence) resolve the reader automatically during structuring, so their format field can stay `Unspecified`. +- **Selecting the V2.0 draft format**: For Version 2.0 services the `"Read into Draft Impl."` field (shown as "Draft Format" on the service card) selects how the inbound structured e-document is read into a draft. For already-structured formats such as PEPPOL, Data Exchange Purchase, OIOUBL, or XRechnung XML the structuring step can return `Unspecified`, so the read step falls back to the service's `"Read into Draft Impl."`; if that is also `Unspecified`, the `E-Doc. Unspecified Impl.` runs and import errors with "No method to read the e-document has been provided." PDF services (for example Azure Document Intelligence or MLLM) resolve the reader automatically during structuring, so their format field can stay `Unspecified`. +- **Duplicate detection uses document date**: `E-Document.IsDuplicate()` checks for matching `"Incoming E-Document No."`, `"Bill-to/Pay-to No."`, and `"Document Date"`. V2.0 Prepare draft copies Document Date and Due Date from the purchase staging header back to the E-Document before this matters, and purchase document creation validates those dates onto the created invoice or credit memo. Duplicates can be deleted without confirmation; unique documents require explicit user confirmation. +- **Posting date default for purchase drafts**: Purchase invoice and credit memo creation can use Purchases & Payables Setup `"E-Doc. Def. Posting Date" = "Document Date"` to validate the created purchase document's Posting Date from the draft Document Date; otherwise the normal BC default remains in control. + +*Updated: 2026-07-29 -- added date propagation, VAT handling, sales drafts, and message gotchas.* ## Inbound flow (V1.0 pipeline) @@ -144,4 +159,3 @@ flowchart TD ### Why V1.0 is deprecated V1.0 has no staging tables, no user review step, and no reversibility. Format implementations must know how to produce complete purchase documents, violating separation of concerns. The V2.0 pipeline splits this into discrete, undoable stages with provider interfaces for entity resolution. -- **Duplicate detection**: `E-Document.IsDuplicate()` checks for matching `"Incoming E-Document No."`, `"Bill-to/Pay-to No."`, and `"Document Date"`. Duplicates can be deleted without confirmation; unique documents require explicit user confirmation. diff --git a/src/Apps/W1/EDocument/App/docs/data-model.md b/src/Apps/W1/EDocument/App/docs/data-model.md index 4733b23fe87..24be610e3d9 100644 --- a/src/Apps/W1/EDocument/App/docs/data-model.md +++ b/src/Apps/W1/EDocument/App/docs/data-model.md @@ -8,7 +8,7 @@ The central entity is the **E-Document** table (`table 6121 "E-Document"`). Ever An E-Document is always associated with at least one **E-Document Service Status** record (`table 6138`), which tracks the per-service state. The composite key is `(E-Document Entry No, E-Document Service Code)`. In practice, most documents have exactly one service status record, but the model supports multiple services per document (e.g., send via PEPPOL and also via email through different workflow steps). -Every state transition produces an **E-Document Log** entry (`table 6124`), which captures the service status at that moment and optionally links to a **E-Doc. Data Storage** record (`table 6125`) containing the document blob. The separation between log and storage is intentional: multiple log entries can reference the same storage entry (for batch imports where many documents share one downloaded blob). +Every state transition produces an **E-Document Log** entry (`table 6124`), which captures the service status at that moment and optionally links to a **E-Doc. Data Storage** record (`table 6125`) containing the document blob. The separation between log and storage is intentional: multiple log entries can reference the same storage entry (for batch imports where many documents share one downloaded blob). Lifecycle payloads that are not BC documents, such as PEPPOL Order Responses, are stored in **E-Document Message** records and linked back to the parent E-Document. The **E-Document Integration Log** (`table 6126`, in the Logging folder) stores HTTP request/response pairs from service communication. It links to both the E-Document and the service, providing a full communication audit trail. @@ -18,13 +18,16 @@ erDiagram E-DOCUMENT ||--o{ E-DOCUMENT-LOG : "has log entries" E-DOCUMENT-LOG }o--o| E-DOC-DATA-STORAGE : "references blob" E-DOCUMENT ||--o{ E-DOCUMENT-INTEGRATION-LOG : "has comms log" + E-DOCUMENT ||--o{ E-DOCUMENT-MESSAGE : "has messages" ``` +*Updated: 2026-07-29 -- added lifecycle messages to the core model.* + ## Service configuration An **E-Document Service** (`table 6103`) defines a processing endpoint. It combines a Document Format (the `"E-Document Format"` enum, which implements the `"E-Document"` interface for serialization) with a Service Integration (the `"Service Integration"` enum, which implements `IDocumentSender` and `IDocumentReceiver`). Both are extensible enums that connector apps extend. -The service also holds import pipeline configuration: `"Import Process"` (Version 1.0 or 2.0), `"Read into Draft Impl."` (how to parse structured data), and a set of boolean flags that control import behavior (Validate Receiving Company, Resolve Unit Of Measure, Lookup Item Reference, Lookup Item GTIN, Lookup Account Mapping, etc.). +The service also holds import pipeline configuration: `"Import Process"` (Version 1.0 or 2.0, with Version 2.0 as the table default), `"Read into Draft Impl."` (the Draft Format fallback for already-structured data), and a set of boolean flags that control import behavior (Validate Receiving Company, Resolve Unit Of Measure, Lookup Item Reference, Lookup Item GTIN, Lookup Account Mapping, etc.). Data Exchange definitions are linked through **E-Doc. Service Data Exch. Def.** and are installed from app resource XML files during install/upgrade rather than being embedded as AL labels. **E-Doc. Service Supported Type** links services to the document types they handle. This is a simple junction table keyed by `(Service Code, E-Document Type)`. A service that handles Sales Invoices and Sales Credit Memos has two records here. @@ -37,25 +40,41 @@ erDiagram E-DOCUMENT-SERVICE ||--o{ E-DOC-SERVICE-SUPPORTED-TYPE : "supports types" E-DOCUMENT-SERVICE ||--o{ SERVICE-PARTICIPANT : "has participants" E-DOCUMENT-SERVICE ||--o{ E-DOC-MAPPING : "defines mappings" + E-DOCUMENT-SERVICE ||--o{ E-DOC-SERVICE-DATA-EXCH-DEF : "selects data exchange defs" ``` +*Updated: 2026-07-29 -- noted V2.0 default import settings and resource-backed Data Exchange definitions.* + ## Inbound staging tables (V2.0 import pipeline) -V2.0 inbound processing uses dedicated staging tables to hold parsed document data before creating actual BC purchase documents. This three-layer approach (raw blob -> staging tables -> BC documents) isolates parsing from business logic and allows users to review and correct data before committing. +V2.0 inbound processing uses dedicated staging tables to hold parsed document data before creating actual BC documents. This three-layer approach (raw blob -> staging tables -> BC documents) isolates parsing from business logic and allows users to review and correct data before committing. -**E-Document Purchase Header** (`table 6100`) and **E-Document Purchase Line** (`table 6101`, in `Processing/Import/Purchase/`) hold the parsed invoice data in a vendor-neutral format. The purchase header has two kinds of fields: external data (fields 2-100, prefixed with vendor/customer names, addresses, amounts as raw text) and BC-resolved fields (fields 101+, prefixed with `[BC]`, containing resolved vendor numbers, posting groups, etc.). The purchase line follows the same pattern. Both are keyed by E-Document Entry No. +**E-Document Purchase Header** (`table 6100`) and **E-Document Purchase Line** (`table 6101`, in `Processing/Import/Purchase/`) hold parsed purchase invoice and credit memo data in a vendor-neutral format. The purchase header has two kinds of fields: external data (fields 2-100, prefixed with vendor/customer names, addresses, amounts as raw text) and BC-resolved fields (fields 101+, prefixed with `[BC]`, containing resolved vendor numbers, posting groups, etc.). The purchase line follows the same pattern. Both are keyed by E-Document Entry No. These purchase staging tables are deliberately consumable by dependent country apps. + +**E-Document Sales Header** (`table 6153`) and **E-Document Sales Line** (`table 6154`, in `Processing/Import/Sales/`) mirror the staging pattern for inbound sales orders. They store buyer/seller order data, requested delivery dates, totals, and BC-resolved customer and sales line fields before a Sales Order is created. These tables are internal and are included in the app's sensitivity classification subscriber. **E-Document Header Mapping** and **E-Document Line Mapping** (`Processing/Import/`) track how staging table fields were mapped to BC entities during the "Prepare draft" step. -For order matching scenarios (linking inbound invoices to existing purchase orders), the **E-Doc. Imported Line** table (`Processing/OrderMatching/`) stores lines from the imported document alongside references to matched purchase order lines. +For order matching scenarios (linking inbound invoices to existing purchase orders), the **E-Doc. Imported Line** table (`Processing/OrderMatching/`) stores lines from the imported document alongside references to matched purchase order lines. **E-Doc. Record Link** records the temporary source-to-target links between draft headers/lines and created purchase or sales lines so later posting and history logic can find the origin. ```mermaid erDiagram - E-DOCUMENT ||--o| E-DOCUMENT-PURCHASE-HEADER : "parsed into" + E-DOCUMENT ||--o| E-DOCUMENT-PURCHASE-HEADER : "purchase draft" E-DOCUMENT-PURCHASE-HEADER ||--o{ E-DOCUMENT-PURCHASE-LINE : "has lines" + E-DOCUMENT ||--o| E-DOCUMENT-SALES-HEADER : "sales draft" + E-DOCUMENT-SALES-HEADER ||--o{ E-DOCUMENT-SALES-LINE : "has lines" E-DOCUMENT ||--o{ E-DOC-IMPORTED-LINE : "for order matching" + E-DOCUMENT ||--o{ E-DOC-RECORD-LINK : "tracks created records" ``` +*Updated: 2026-07-29 -- added sales order staging, record links, and purchase staging visibility.* + +## Purchase draft VAT and dates + +Purchase draft data now carries more posting intent before the BC document exists. `Document Date` and `Due Date` are copied from the purchase staging header back to the E-Document during Prepare draft, so duplicate detection and document creation use the parsed date rather than only the received file metadata. Purchases & Payables Setup controls whether purchase drafts resolve VAT Product Posting Groups from extracted VAT rates and whether an allowed VAT amount difference is applied to created purchase lines. The VAT difference is derived from the difference between header Total VAT and line-calculated VAT, and only applies when both purchase and general ledger setup permit it. + +*Updated: 2026-07-29 -- added purchase draft date and VAT handling.* + ## Dual data storage Every inbound E-Document can have two blob references: `"Unstructured Data Entry No."` and `"Structured Data Entry No."`, both pointing to `"E-Doc. Data Storage"` records. The unstructured slot holds the original file as received (e.g., a PDF). The structured slot holds the machine-readable version (e.g., XML extracted by Azure Document Intelligence, or the original XML if the document was already structured). @@ -68,7 +87,7 @@ Each `"E-Doc. Data Storage"` record has a `"File Format"` enum field that implem Outbound E-Documents link to their source BC document via `"Document Record ID"` (a RecordId). This is a BC runtime reference, not a traditional foreign key, which means it works across any source table (Sales Invoice Header, Service Cr.Memo Header, etc.) without explicit table relations. -Inbound E-Documents, once the import pipeline completes, store the created BC document's RecordId in the same `"Document Record ID"` field. The E-Document table's `OnDelete` trigger prevents deletion of linked or processed documents. +Inbound E-Documents, once the import pipeline completes, store the created BC document's RecordId in the same `"Document Record ID"` field. Purchase documents still keep the `"E-Document Link"` Guid while the draft is linked to an open purchase document; sales orders use the same link pattern for inbound order drafts. The E-Document table's `OnDelete` trigger prevents deletion of linked or processed documents. Extensions on BC tables add E-Document awareness: `"E-Document Link"` (a Guid) on `Purchase Header` links back to the E-Document's SystemId for V1.0 processing (marked for removal in CLEAN27). Page extensions on Posted Sales Invoice, Posted Purchase Invoice, etc. add factboxes showing E-Document status. diff --git a/src/Apps/W1/EDocument/App/docs/extensibility.md b/src/Apps/W1/EDocument/App/docs/extensibility.md index 903e5eb495c..07700155ed9 100644 --- a/src/Apps/W1/EDocument/App/docs/extensibility.md +++ b/src/Apps/W1/EDocument/App/docs/extensibility.md @@ -22,7 +22,9 @@ Example: to add a custom document format, extend `enum 6101 "E-Document Format"` - `GetBasicInfoFromReceivedDocument(EDocument, TempBlob)` -- extract header info from received blob (V1.0 import) - `GetCompleteInfoFromReceivedDocument(EDocument, CreatedDocumentHeader, CreatedDocumentLines, TempBlob)` -- parse into BC records (V1.0 import) -**Binding enum**: `"E-Document Format"` (`enum 6101`). Built-in values: `"Data Exchange"` and `"PEPPOL BIS 3.0"`. +**Binding enum**: `"E-Document Format"` (`enum 6101`). Built-in values: `"Data Exchange"` and `"PEPPOL BIS 3.0"`. PEPPOL XML construction and validation primitives live in the standalone PEPPOL app that Core depends on; Core keeps the enum binding and adapter/reader logic. + +*Updated: 2026-07-29 -- clarified the PEPPOL split between Core and the standalone PEPPOL app.* **Where configured**: `"Document Format"` field on `"E-Document Service"` table. @@ -103,13 +105,13 @@ For the two built-in actions, the framework also checks if the service integrati procedure StructureReceivedEDocument(EDocumentDataStorage): Interface IStructuredDataType ``` -Returns an `IStructuredDataType` that wraps the structured content, its file format, and which `"E-Doc. Read into Draft"` implementation should read it. +Returns an `IStructuredDataType` that wraps the structured content, its file format, and which `"E-Doc. Read into Draft"` implementation should read it. If it returns `Unspecified`, the pipeline falls back to the E-Document's value and then to the service's Draft Format field. -**Binding enum**: `"Structure Received E-Doc."` (`enum 6120`). +**Binding enum**: `"Structure Received E-Doc."` (`enum 6103`). ### Reading structured data into staging tables -**Goal**: Parse structured data (XML, JSON, ADI output) into the E-Document Purchase Header/Line staging tables. +**Goal**: Parse structured data (XML, JSON, ADI output) into the E-Document purchase or sales staging tables. **Interface**: `IStructuredFormatReader` (in `Processing/Interfaces/IStructuredFormatReader.Interface.al`) @@ -118,9 +120,11 @@ procedure ReadIntoDraft(EDocument, TempBlob): Enum "E-Doc. Process Draft" procedure View(EDocument, TempBlob) ``` -`ReadIntoDraft` populates the staging tables and returns the `"E-Doc. Process Draft"` enum value that determines which `IProcessStructuredData` runs next. +`ReadIntoDraft` populates the staging tables and returns the `"E-Doc. Process Draft"` enum value that determines which `IProcessStructuredData` runs next. Built-in readers include PEPPOL, ADI, MLLM, Blank Draft, and Data Exchange Purchase; dependent format apps can add readers such as OIOUBL or XRechnung by extending the enum. + +**Binding enum**: `"E-Doc. Read into Draft"` (`enum 6109`). -**Binding enum**: `"E-Doc. Read into Draft"` (`enum 6113`). +The Data Exchange Purchase reader uses `E-Doc. Data Exchange Impl.` as a reusable XML-to-intermediate-data bridge and then maps the result into `E-Document Purchase Header` and `E-Document Purchase Line`. Those draft tables are exposed for dependent country apps to consume, while the bridge events below remain the safer customization points for adjusting mapped data. `E-Doc. Data Exchange Impl.` is no longer restricted with `Access = Internal`, but extensions should still prefer enum bindings and published events before depending on implementation details. ### Processing the draft @@ -135,9 +139,9 @@ procedure OpenDraftPage(var EDocument) procedure CleanUpDraft(EDocument) ``` -`PrepareDraft` returns the resolved document type. `GetVendor` is called separately to populate the E-Document's vendor fields. `CleanUpDraft` is called when an E-Document is deleted. +`PrepareDraft` returns the resolved document type. `GetVendor` is called separately for purchase drafts to populate the E-Document's vendor fields. Sales implementations can extend the contract through `IProcessStructuredDataSales` to resolve customers. `CleanUpDraft` is called when an E-Document is deleted. -**Binding enum**: `"E-Doc. Process Draft"` (`enum 6112`). +**Binding enum**: `"E-Doc. Process Draft"` (`enum 6107`). ### Finishing the draft (creating BC documents) @@ -150,9 +154,11 @@ procedure ApplyDraftToBC(EDocument, EDocImportParameters): RecordId procedure RevertDraftActions(EDocument) ``` -`ApplyDraftToBC` creates the Purchase Invoice/Credit Memo and returns its RecordId. `RevertDraftActions` undoes the creation (deletes the BC document). +`ApplyDraftToBC` creates the BC document for the resolved E-Document Type and returns its RecordId. Current Core implementations handle Purchase Invoice, Purchase Credit Memo, and Sales Order. `RevertDraftActions` undoes the Core link and moves attachments back to the E-Document; it does not post or delete the user's business document for them. + +**Binding enum**: `"E-Document Type"` (`enum 6121`). Each document type value implements this interface to handle its specific creation logic. -**Binding enum**: `"E-Document Type"` (`enum 6105`). Each document type value implements this interface to handle its specific creation logic. +*Updated: 2026-07-29 -- added Draft Format fallback, Data Exchange Purchase, sales order import, and dependent format readers.* ## Provider interfaces @@ -163,6 +169,9 @@ These interfaces allow customization of specific resolution steps during the "Pr - **IUnitOfMeasureProvider** -- resolve unit of measure from external code - **IPurchaseLineProvider** -- determine purchase line type and account (replaces deprecated `IPurchaseLineAccountProvider`) - **IPurchaseOrderProvider** -- match E-Document to an existing purchase order +- **ICustomerProvider** -- resolve a customer for inbound sales order drafts +- **ISalesLineProvider** -- resolve sales line type, number, item reference, dimensions, and unit of measure +- **IEDocumentCreateSalesOrder** -- customize how a staged inbound order becomes a BC Sales Order ## Export eligibility @@ -238,6 +247,11 @@ Beyond interfaces, the framework publishes integration events for finer-grained - `OnADIProcessingCompleted` -- react to Azure Document Intelligence processing completion - `OnFoundVendorNo` -- react to vendor resolution during draft preparation +**Data Exchange V2 bridge** (in `EDocDataExchPurchHandler.Codeunit.al`): + +- `OnAfterBridgeToStagingTables` -- react after Data Exchange intermediate data has been mapped to the purchase staging header +- `OnAfterMapLineToStaging` -- react after each intermediate purchase line record becomes an E-Document Purchase Line + **Service configuration** (in `EDocumentService.Table.al`): - `OnAfterGetDefaultFileExtension` -- override the default file extension for the service @@ -246,10 +260,18 @@ Beyond interfaces, the framework publishes integration events for finer-grained - Events on draft page field validations for reacting to user edits +**Attachments** (in `EDocAttachmentProcessor.Codeunit.al`): + +- Document Attachment Management subscribers make E-Document act like an attachment-bearing source record, including filters by `"E-Document Entry No."`. This is the safe extension surface for flows that need to attach or move files around E-Documents, including Digital Voucher attachment flows. + **Import processing** (in `EDocImport.Codeunit.al`): - `OnAfterProcessIncomingEDocument` -- react after the import pipeline completes or advances a step +*Updated: 2026-07-29 -- added Data Exchange bridge and attachment extension points.* + ## Processing customizations -The `"E-Doc. Proc. Customizations"` enum on the service (field 61) provides a secondary customization axis. It is passed to `IProcessStructuredData.GetVendor()` as a parameter, allowing different vendor resolution strategies per service. +The `"E-Doc. Proc. Customizations"` enum on the service (field 61) provides a secondary customization axis. It is passed into purchase and sales draft processing, allowing different vendor, customer, line, unit of measure, purchase document creation, and sales order creation strategies per service. + +*Updated: 2026-07-29 -- expanded processing customization coverage to sales order import.* diff --git a/src/Apps/W1/EDocument/App/docs/patterns.md b/src/Apps/W1/EDocument/App/docs/patterns.md index df5383f7816..18306a11478 100644 --- a/src/Apps/W1/EDocument/App/docs/patterns.md +++ b/src/Apps/W1/EDocument/App/docs/patterns.md @@ -12,12 +12,14 @@ This pattern appears in: - `"Service Integration"` (`enum 6151`) implements `IDocumentSender`, `IDocumentReceiver`, `IConsentManager` -- service communication - `"Integration Action Type"` (`enum 6170`) implements `IDocumentAction` -- extensible actions - `"E-Document Service Status"` (`enum 6106`) implements `IEDocumentStatus` -- status-dependent behavior -- `"Structure Received E-Doc."` (`enum 6120`) implements `IStructureReceivedEDocument` -- structuring raw data -- `"E-Doc. Read into Draft"` (`enum 6113`) implements `IStructuredFormatReader` -- parsing structured data -- `"E-Doc. Process Draft"` (`enum 6112`) implements `IProcessStructuredData` -- draft preparation -- `"E-Document Type"` (`enum 6105`) implements `IEDocumentFinishDraft` -- document creation per type +- `"Structure Received E-Doc."` (`enum 6103`) implements `IStructureReceivedEDocument` -- structuring raw data +- `"E-Doc. Read into Draft"` (`enum 6109`) implements `IStructuredFormatReader` -- parsing structured data +- `"E-Doc. Process Draft"` (`enum 6107`) implements `IProcessStructuredData` -- draft preparation, including purchase and sales draft types +- `"E-Document Type"` (`enum 6121`) implements `IEDocumentFinishDraft` -- document creation per type - `"Export Eligibility Evaluator"` implements `IExportEligibilityEvaluator` -- export gating - `"E-Doc. File Format"` implements `IEDocFileFormat`, `IBlobType` -- file type behavior +- `"E-Document Message Type"` implements `IEDocMessageBuilder` -- lifecycle message payload construction +- `"E-Document Format"` also implements `IEDocResponseProvider` -- lets a format request a related response message The enum value is stored on the E-Document or Service table and read at the dispatch site. For example, in `ImportEDocumentProcess.ReadIntoDraft()`: @@ -28,6 +30,8 @@ EDocument."Process Draft Impl." := IStructuredFormatReader.ReadIntoDraft(EDocume This is clean but has a consequence: the dispatch decision is locked to a single field value. You cannot chain implementations or compose behaviors without building that into the interface contract. +*Updated: 2026-07-29 -- added sales draft and message dispatch enum patterns.* + ## Error-trapping codeunit wrapper ("if codeunit.run") Every call to an external interface implementation is wrapped in a dedicated codeunit that runs inside a `if not Codeunit.Run()` pattern. The framework commits before the call, runs the wrapper codeunit, and catches runtime errors. This isolates interface failures from the calling transaction. @@ -94,7 +98,17 @@ codeunit 6177 "E-Doc. Historical Matching" implements "AOAI Function", IEDocAISy } ``` -This pattern appears in `EDocHistoricalMatching`, `EDocGLAccountMatching`, `EDocDeferralMatching`, and `EDocSimilarDescriptions`. The key advantage is that state flows through the instance rather than through event parameters, keeping the AI tool interface clean while preserving row-level context across asynchronous function calls. +This pattern appears in `EDocHistoricalMatching`, `EDocGLAccountMatching`, `EDocDeferralMatching`, and `EDocSimilarDescriptions`. The key advantage is that state flows through the instance rather than through event parameters, keeping the AI tool interface clean while preserving row-level context across asynchronous function calls. The older Purchase Order Matching Copilot action and buffer/page objects are pending obsolete; current AI-assisted matching runs during import-time draft preparation through `E-Doc. AI Tool Processor` and the matching tool codeunits. + +*Updated: 2026-07-29 -- noted the obsolete PO matching Copilot path and current import-time AI matching.* + +## Data Exchange bridge into staging tables + +The Data Exchange V2 purchase reader uses a bridge pattern. It lets the platform Data Exchange pipeline parse XML into `Intermediate Data Import`, then maps that intermediate data into `E-Document Purchase Header` and `E-Document Purchase Line` staging records. This keeps Data Exchange definitions declarative while still fitting the V2 import pipeline's draft, prepare, and finish stages. + +The bridge also performs work that Data Exchange mappings cannot express cleanly: choosing the configured definition by root namespace, applying the blank-LCY currency convention, decoding embedded attachments, and adding PEPPOL allowance charge lines to intermediate data before staging line creation. + +*Updated: 2026-07-29 -- documented the Data Exchange V2 bridge pattern.* ## Bidirectional state machine (import pipeline) @@ -105,7 +119,7 @@ The V2.0 import pipeline in `ImportEDocumentProcess.Codeunit.al` is a bidirectio The `StatusStepIndex()` function maps each `"Import E-Doc. Proc. Status"` to a numeric index (0-4), and `GetNextStep()` / `GetPreviousStep()` map statuses to the `"Import E-Document Steps"` enum values that transition between them. -This design makes it possible to revert a processed document back to any earlier stage. For example, if a user notices the vendor was resolved incorrectly, they can revert from "Draft Ready" to "Ready for draft", change vendor data, and re-run "Prepare draft". +This design makes it possible to revert a processed document back to any earlier stage. For example, if a user notices the vendor was resolved incorrectly, they can revert from "Draft Ready" to "Ready for draft", change vendor data, and re-run "Prepare draft". The same state machine now routes to different draft types: purchase invoices and credit memos use purchase staging, while inbound orders use sales staging and Sales Order creation. ## Workflow as orchestration layer @@ -113,6 +127,14 @@ The framework delegates flow control to BC's Workflow engine rather than hardcod This means the sequence of operations (export -> send -> email -> approval) is configured in the workflow, not in code. `EDocumentCreatedFlow.Codeunit.al` triggers the workflow after document creation, and each workflow step response delegates to the appropriate framework method. +## Lifecycle messages as related payloads + +Some PEPPOL interactions are not new BC documents. Core models them as `E-Document Message` records linked to the parent E-Document, with payloads stored in `E-Doc. Data Storage`. The message type enum dispatches to a builder, while the document format can opt in by implementing `IEDocResponseProvider`. + +This keeps order responses out of the document creation pipeline while preserving an auditable payload and response status on the E-Document. Inbound `OrderResponse` XML is stored against the matching outbound E-Document and the temporary inbound carrier is deleted. + +*Updated: 2026-07-29 -- added lifecycle message pattern for order responses.* + ## Blob management (TempBlob and Data Storage) Blobs flow through the system in two forms: in-memory `TempBlob` codeunits during processing, and persisted `"E-Doc. Data Storage"` records for permanent storage. The conversion happens in `EDocumentLog.InsertLog()`, which writes the TempBlob to a new Data Storage record and links it via `"E-Doc. Data Storage Entry No."` on the log entry. @@ -149,7 +171,7 @@ These patterns exist in the codebase but are deprecated. Understanding them help **Why deprecated**: No staging tables, no user review, no reversibility. Format implementations had to know how to create Purchase Invoices, violating separation of concerns. -**What to do instead**: Use V2.0 pipeline with `"Import Process" = "Version 2.0"`. Implement `IStructuredFormatReader` to populate staging tables and let the framework handle BC document creation. +**What to do instead**: Use V2.0 pipeline with `"Import Process" = "Version 2.0"`, which is the default for new services. Implement `IStructuredFormatReader` to populate staging tables and let the framework handle BC document creation. ### Old "E-Document Integration" enum and interface diff --git a/src/Apps/W1/EDocument/App/src/DataExchange/docs/CLAUDE.md b/src/Apps/W1/EDocument/App/src/DataExchange/docs/CLAUDE.md index 10fb4c46e1a..f9a61b77e51 100644 --- a/src/Apps/W1/EDocument/App/src/DataExchange/docs/CLAUDE.md +++ b/src/Apps/W1/EDocument/App/src/DataExchange/docs/CLAUDE.md @@ -4,7 +4,13 @@ An alternative to the interface-based Format module -- this implements the `"E-D ## How it works -`EDocDataExchangeImpl` implements the same `"E-Document"` interface as the Format module's PEPPOL codeunit, but its `Create` method works differently. It looks up the `E-Doc. Service Data Exch. Def.` table to find the export Data Exchange Definition for the document type, creates a `Data Exch.` record with line filters, and calls `DataExch.ExportFromDataExch` to run the configured mapping. The resulting XML blob is extracted from the Data Exch record's field 3. For import, `GetCompleteInfoFromReceivedDocument` uses the import Data Exchange Definition to parse incoming XML into `Intermediate Data Import` records, which are then processed by `EDocDEDPEPPOLPreMapping`. +*Updated: 2026-07-29 -- Added resource-backed definitions and the Data Exchange to import v2 bridge.* + +`EDocDataExchangeImpl` implements the same `"E-Document"` interface as the Format module's PEPPOL codeunit, but its `Create` method works differently. It looks up the `E-Doc. Service Data Exch. Def.` table to find the export Data Exchange Definition for the document type, creates a `Data Exch.` record with line filters, and calls `DataExch.ExportFromDataExch` to run the configured mapping. The resulting XML blob is extracted from the Data Exch record's field 3. + +For import, Data Exchange is now a bridge into the v2 import pipeline. `GetBasicInfoFromReceivedDocument` tests import Data Exchange Definitions that map only through `Intermediate Data Import`, picks the definition that yields the most intermediate rows, stores that definition code and document type on the E-Document, and then extracts header fields by reusing the Data Exchange paths. `GetCompleteInfoFromReceivedDocument` runs the selected definition, processes the Data Exchange, and converts the intermediate rows into temporary purchase header, purchase line, and attachment records for the caller. + +The definitions are packaged as app resource files under `.resources/DataExchange/*.xml`. `E-Document Install` loads them with `NavApp.GetResource` and XMLport `Imp / Exp Data Exch Def & Map`; upgrade code also imports the v2 invoice and credit memo definitions so existing tenants receive the bridge definitions. The AL labels now name resource paths rather than carrying the XML payload. The `E-Doc. Service Data Exch. Def.` table links an E-Document Service code and document type to both an import and export Data Exchange Definition code, displayed via the `E-Doc. Service Data Exch. Sub` subpage on the service card. @@ -14,7 +20,11 @@ The `E-Doc. Service Data Exch. Def.` table links an E-Document Service code and ## Things to know +*Updated: 2026-07-29 -- Noted the public Data Exchange implementation surface and v2 definitions.* + - `EDocDEDPEPPOLExternal` is a dummy codeunit with an empty `OnRun` -- it exists solely to be referenced as the "External Data Handling Codeunit" in Data Exchange Definitions, satisfying a BC framework requirement. +- `E-Doc. Data Exchange Impl.` is no longer marked `Access = Internal`, so the Data Exchange implementation is part of the app surface instead of being confined to this module. +- The v2 PEPPOL import resources create purchase-draft definitions (`EDOCPEPINVPURCHDRAFT` and `EDOCPEPCMPURCHDRAFT`) that target `Purchase Header`, `Purchase Line`, and `Document Attachment` through `Intermediate Data Import`. - The Data Exchange approach is more configurable but less flexible than the XMLport-based Format approach. Localizations that need complex XML structures often use the Format interface directly. - The pre-mapping import path validates that referenced purchase invoices are posted before allowing credit memo creation (`YouMustFirstPostTheRelatedInvoiceErr`). - `EDocDEDPEPPOLSubscribers` uses `SingleInstance` because the Data Exchange framework processes records one at a time through event subscribers, and state (loop counters, VAT amounts) must persist across those calls. diff --git a/src/Apps/W1/EDocument/App/src/Document/docs/CLAUDE.md b/src/Apps/W1/EDocument/App/src/Document/docs/CLAUDE.md index 1aaa6db1d00..91eb522fc31 100644 --- a/src/Apps/W1/EDocument/App/src/Document/docs/CLAUDE.md +++ b/src/Apps/W1/EDocument/App/src/Document/docs/CLAUDE.md @@ -1,27 +1,37 @@ # Document -The E-Document table (table 6121 in `EDocument.Table.al`) is the aggregate root of the entire framework. Every electronic document -- whether an outbound sales invoice or an inbound purchase credit memo -- lives as a single row here. This module also owns the status model, direction semantics, document type taxonomy, and user notification infrastructure. +The E-Document table (table 6121 in `EDocument.Table.al`) is the aggregate root of the entire framework. Every electronic document -- whether an outbound sales invoice, an inbound purchase credit memo, or a response message for an existing order -- lives or is tracked from here. This module also owns the status model, direction semantics, document type taxonomy, message factboxes, and user notification infrastructure. ## How it works -When a BC document is posted or an external document arrives from a service endpoint, the framework creates an E-Document record via the `Create` procedure, stamping it with a direction (Incoming/Outgoing from `EDocumentDirection.Enum.al`), a document type from the extensive `E-Document Type` enum (22 values covering sales, purchase, service, finance charge, reminders, journals, and shipments), and the originating service code. +*Updated: 2026-07-29 -- Added message handling, sales order import, and refreshed incoming draft semantics.* -The E-Document carries three independent status dimensions that evolve separately. The top-level `Status` field (enum 6108: In Progress, Processed, Error) is derived automatically from the per-service `E-Document Service Status` via the strategy pattern -- each service status value implements `IEDocumentStatus` (defined in `Interfaces/IEDocumentStatus.Interface.al`), and the three codeunits in `Status/` (`EDocErrorStatus`, `EDocInProgressStatus`, `EDocProcessedStatus`) return the corresponding top-level status. Most service statuses default to "In Progress" unless explicitly mapped to Error or Processed in the enum implementation declarations. The third dimension, `Import Processing Status`, is a FlowField that reads from the `E-Document Service Status` table, tracking inbound documents through a five-step pipeline: Unprocessed, Readable, Ready for draft, Draft Ready, Processed. +When a BC document is posted or an external document arrives from a service endpoint, the framework creates an E-Document record via the `Create` procedure, stamping it with a direction (Incoming/Outgoing from `EDocumentDirection.Enum.al`), a document type from the extensive `E-Document Type` enum (covering sales, purchase, service, finance charge, reminders, journals, shipments, and order responses), and the originating service code. -The `E-Document` interface (`Interfaces/EDocument.Interface.al`) defines the format contract that document format implementations must satisfy -- `Check`, `Create`, `CreateBatch` for outbound, and `GetBasicInfoFromReceivedDocument` / `GetCompleteInfoFromReceivedDocument` for inbound. +The E-Document carries three independent status dimensions that evolve separately. The top-level `Status` field (enum 6108: In Progress, Processed, Error, plus Canceled for explicit cancellation checks) is normally derived from the per-service `E-Document Service Status` via the strategy pattern -- each service status value implements `IEDocumentStatus` (defined in `Interfaces/IEDocumentStatus.Interface.al`), and the three codeunits in `Status/` (`EDocErrorStatus`, `EDocInProgressStatus`, `EDocProcessedStatus`) return the corresponding top-level status. Most service statuses default to "In Progress" unless explicitly mapped to Error or Processed in the enum implementation declarations. The third dimension, `Import Processing Status`, is a FlowField that reads from the `E-Document Service Status` table, tracking inbound documents through a five-step pipeline: Unprocessed, Readable, Ready for draft, Draft Ready, Processed. + +The `E-Document` interface (`Interfaces/EDocument.Interface.al`) defines the format contract that document format implementations must satisfy -- `Check`, `Create`, `CreateBatch` for outbound, and `GetBasicInfoFromReceivedDocument` / `GetCompleteInfoFromReceivedDocument` for inbound. The `E-Document Format` enum now also implements `IEDocResponseProvider`; the PEPPOL implementation can classify an inbound order response as a message for an existing outbound E-Document instead of continuing through draft creation. + +Order response messages are stored in `E-Document Message` records and shown on the E-Document card, the E-Documents list, and inbound pages through the messages factbox. A message can carry its own data storage entry and response type, but it does not create a BC purchase or sales document. The inbound carrier record is cleaned up after the message is attached to the outbound E-Document. ## Things to know -- Duplicate detection uses `IsDuplicate()` which checks the composite `(Incoming E-Document No., Bill-to/Pay-to No., Document Date)` with `ReadIsolation::ReadUncommitted` -- this means it can see uncommitted records from other sessions, avoiding race conditions during batch imports. +*Updated: 2026-07-29 -- Refreshed duplicate, deletion, cleanup, and draft metadata notes.* + +- Duplicate detection uses `IsDuplicate()` which checks the composite `(Incoming E-Document No., Bill-to/Pay-to No., Document Date)` with `ReadIsolation::ReadUncommitted` -- this means it can see uncommitted records from other sessions, avoiding race conditions during batch imports. For V2 purchase imports, `PrepareDraft` copies `Document Date` and `Due Date` from `E-Document Purchase Header` back to the E-Document, so the duplicate check depends on the extracted document date rather than only on the received file identity. -- Deletion is heavily guarded: you cannot delete a Processed document or one linked to a source document (`Document Record ID`). Non-duplicate documents require explicit user confirmation, and non-GUI contexts block outright. +- Deletion is still guarded by the table trigger: you cannot delete a Processed document or one linked to a source document (`Document Record ID`). List pages now allow users to invoke delete, but unique non-GUI deletes still block and unique GUI deletes still require confirmation. Orphaned inbound carrier records for order response messages use `DeleteOrphanedImport()` after cleanup. -- The `CleanupDocument` procedure cascades deletes to logs, integration logs, service statuses, mapping logs, imported lines, and document attachments. It also invokes `IProcessStructuredData.CleanUpDraft` for version 2 processing cleanup. +- The `CleanupDocument` procedure cascades deletes to logs, integration logs, service statuses, mapping logs, imported lines, document attachments, E-Document messages, and error messages. It also invokes `IProcessStructuredData.CleanUpDraft` for version 2 processing cleanup. - The `E-Documents Setup` table (table 6107) is marked `ObsoleteState = Pending` with tag '28.0'. It controls the "new E-Document experience" feature gate, which is activated per-tenant via AAD tenant ID allowlist, environment setting, or country code list. - The `E-Document` interface's `CreateBatch` method receives a record set of E-Documents rather than a single record -- format implementations must handle multi-document serialization into a single blob. -- Fields 42-44 (`Structure Data Impl.`, `Read into Draft Impl.`, `Process Draft Impl.`) are enum-based strategy selectors for the import processing pipeline, allowing different implementations per document. +- Fields 42-44 (`Structure Data Impl.`, `Read into Draft Impl.`, `Process Draft Impl.`) are enum-based strategy selectors for the import processing pipeline, allowing different implementations per document. `E-Document Type` now routes Purchase Credit Memo and Sales Order to finish-draft implementations in addition to Purchase Invoice. + +- PEPPOL basic-info parsing sets `Order No.` from `OrderReference` for both purchase invoices and purchase credit memos before full draft processing. Credit memo finish draft also keeps apply-to references: `EDocCreatePurchCrMemo` copies the purchase order reference to `Vendor Order No.` and can resolve the apply-to invoice either from `Applies-to Doc. No.` or from the external invoice number. + +- Inbound page actions that continue processing (`Analyze PDF`, `Prepare Draft`, and `Open draft document`) call `ThrowIfHasErrors` after running import steps, so processing errors are surfaced to the user instead of only changing status. - Notification infrastructure (`Notification/`) currently handles a single scenario -- alerting users when an inbound vendor was matched by name but not address. Notifications are per-user, dismissable, and backed by the `My Notifications` framework. diff --git a/src/Apps/W1/EDocument/App/src/Document/docs/data-model.md b/src/Apps/W1/EDocument/App/src/Document/docs/data-model.md index c765affc7f5..9bdbc3e7a51 100644 --- a/src/Apps/W1/EDocument/App/src/Document/docs/data-model.md +++ b/src/Apps/W1/EDocument/App/src/Document/docs/data-model.md @@ -4,6 +4,8 @@ This describes the data model for the E-Document aggregate root and its immediat ## Core entity and status tracking +*Updated: 2026-07-29 -- Added draft staging, response messages, and current date semantics.* + The `E-Document` table (6121) is the central record. Each E-Document points to its originating BC document via `Document Record ID` (a RecordId field) and to its content via `Structured Data Entry No.` and `Unstructured Data Entry No.`, both foreign keys to `E-Doc. Data Storage`. The `E-Document Service Status` table (6138) tracks per-service processing state for each document, creating a one-to-many relationship between documents and services. ```mermaid @@ -11,34 +13,64 @@ erDiagram E-Document ||--o{ E-Document-Service-Status : "has per-service status" E-Document ||--o| E-Doc-Data-Storage-Structured : "structured content" E-Document ||--o| E-Doc-Data-Storage-Unstructured : "unstructured content" + E-Document ||--o| E-Document-Purchase-Header : "has purchase draft" + E-Document-Purchase-Header ||--o{ E-Document-Purchase-Line : "contains lines" + E-Document ||--o| E-Document-Sales-Header : "has sales order draft" + E-Document-Sales-Header ||--o{ E-Document-Sales-Line : "contains lines" + E-Document ||--o{ E-Document-Message : "has responses" + E-Document-Message }o--o| E-Doc-Data-Storage : "stores payload" E-Document-Service-Status }o--|| E-Document-Service : "belongs to service" ``` The `E-Document Service Status` table uses a composite primary key of `(E-Document Entry No, E-Document Service Code)`. Its `Import Processing Status` field has a validate trigger that automatically synchronizes the `Status` field -- when import processing reaches Processed, the service status flips to "Imported Document Created"; otherwise it stays at "Imported". This coupling means you cannot set import processing status without side-effecting the service status. +`Document Date` and `Due Date` on E-Document are now part of the inbound V2 draft contract. During `PrepareDraft`, the purchase draft header's dates are copied back to the E-Document when present. `Document Date` is also part of Key3 and the `IsDuplicate()` filter, so duplicate purchase invoice detection is date-sensitive. + +## Inbound draft staging + +*Updated: 2026-07-29 -- Documented purchase table exposure and sales draft classification.* + +Inbound purchase drafts are staged in `E-Document Purchase Header` and `E-Document Purchase Line`, keyed by the E-Document entry number. These tables are no longer `Access = Internal`, while their inherent permissions stay RIMDX. That is the intended consumption surface for country apps that need to inspect or enrich incoming purchase drafts without depending on the final BC purchase document. + +The purchase header stores external document identity and accounting context, including `Document Date`, `Due Date`, order references, and apply-to references for credit memos. The purchase line stores extracted line data and the validated BC match. VAT product posting group is now a validated field on the draft line, and the header can calculate a bounded VAT amount difference from draft lines when the Purchases & Payables setup allows it. + +Sales order import uses its own staging pair, `E-Document Sales Header` and `E-Document Sales Line`. The sales staging tables do not replicate data and classify external buyer, seller, address, amount, item, and date fields as customer content; only technical linkage fields such as E-Document entry and line number are system metadata. This keeps imported customer order payloads treated as business data while still linking them back to the aggregate root. + ## Three status dimensions +*Updated: 2026-07-29 -- Added the Canceled top-level status.* + The status model is the most important design decision in this module. Rather than a single linear state machine, the framework uses three orthogonal dimensions. -**E-Document Status** (enum 6108) is the top-level rollup with only three values: In Progress, Processed, Error. It is never set directly -- it is derived from the service status via the `IEDocumentStatus` interface. Each `E-Document Service Status` enum value declares which `IEDocumentStatus` implementation it uses. For example, "Exported", "Sent", "Canceled", "Approved", "Rejected", "Cleared", and "Imported Document Created" all map to Processed; "Sending Error", "Cancel Error", "Export Error", "Imported Document Processing Error", and "Approval Error" map to Error; everything else defaults to In Progress. +**E-Document Status** (enum 6108) is the top-level rollup with In Progress, Processed, Error, and an explicit Canceled value used by linked-record checks. The usual rollup is derived from the service status via the `IEDocumentStatus` interface. Each `E-Document Service Status` enum value declares which `IEDocumentStatus` implementation it uses. For example, service statuses "Exported", "Sent", "Canceled", "Approved", "Rejected", "Cleared", and "Imported Document Created" all map to Processed; "Sending Error", "Cancel Error", "Export Error", "Imported Document Processing Error", and "Approval Error" map to Error; everything else defaults to In Progress. **E-Document Service Status** (enum 6106) is the fine-grained operational status with 20+ values spanning the full lifecycle: Created, Exported, Sent, Imported, Canceled, Pending Batch, Pending Response, Order Linked, Cleared, and various error states. The clearance model values (30-31: Not Cleared, Cleared) are reserved in a separate range for tax authority clearance workflows. **Import Processing Status** (enum 6100) is a five-step inbound pipeline: Unprocessed, Readable, Ready for draft, Draft Ready, Processed. Each step corresponds to a processing action (structure received data, read into intermediate representation, prepare draft, finish draft). This enum is not extensible. -## Notification model +## Notification and message model + +*Updated: 2026-07-29 -- Added E-Document messages alongside user notifications.* ```mermaid erDiagram E-Document ||--o{ E-Document-Notification : "has notifications" + E-Document ||--o{ E-Document-Message : "has messages" + E-Document-Message }o--o| E-Doc-Data-Storage : "stores message payload" ``` The `E-Document Notification` table (6126) uses a composite key of `(E-Document Entry No., ID, User Id)` where ID is a well-known Guid identifying the notification type. This design allows multiple notification types per document per user. Currently only one type exists ("Vendor Matched By Name Not Address"), but the structure supports adding more notification scenarios without schema changes. Notifications integrate with BC's `My Notifications` framework for user-level opt-out. +The `E-Document Message` table stores response-style traffic, for example a PEPPOL Order Response, against the E-Document it belongs to. It records message type, direction, status, response type, service, and an optional data storage payload. Messages are not notifications and do not create draft documents; they update the lifecycle context of an existing E-Document. + ## Design decisions and gotchas +*Updated: 2026-07-29 -- Refreshed cleanup and reference gotchas.* + - The `Document Record ID` field stores a RecordId, which is a BC-specific composite reference encoding table number and primary key. This means the E-Document can point to any source document table without a fixed foreign key, but it also means the reference breaks if the source record is renumbered or the table ID changes. -- Key3 on the E-Document table `(Incoming E-Document No., Bill-to/Pay-to No., Document Date, Entry No)` exists specifically for the `IsDuplicate()` check. The inclusion of `Entry No` in the key allows efficient exclusion of the current record during the duplicate scan. +- Key3 on the E-Document table `(Incoming E-Document No., Bill-to/Pay-to No., Document Date, Entry No)` exists specifically for the `IsDuplicate()` check. The inclusion of `Entry No` in the key allows efficient exclusion of the current record during the duplicate scan. Because `Document Date` is part of this key, incomplete date extraction can change duplicate behavior. + +- Cleanup now includes error messages and E-Document messages, not only logs, service statuses, mapping logs, imported lines, attachments, and draft records. If a new child table is added to the aggregate, `CleanupDocument()` must be reviewed. - The `E-Documents Setup` table is obsolete (pending removal in v28). Its feature gating logic checks three sources in priority order: explicit table flag, AAD tenant ID allowlist, environment setting, then country code list. The country list is hardcoded to 14 specific localizations plus W1. diff --git a/src/Apps/W1/EDocument/App/src/Extensions/docs/CLAUDE.md b/src/Apps/W1/EDocument/App/src/Extensions/docs/CLAUDE.md index cb6fee5cc17..6ce7a59da27 100644 --- a/src/Apps/W1/EDocument/App/src/Extensions/docs/CLAUDE.md +++ b/src/Apps/W1/EDocument/App/src/Extensions/docs/CLAUDE.md @@ -1,25 +1,43 @@ # Extensions -Base app integration surface -- table extensions, page extensions, and enum extensions that embed E-Document capabilities into existing BC entities. This module does not contain business logic; it adds fields, actions, and factboxes so that E-Documents are visible and actionable from the pages users already work in. +Base app integration surface -- table extensions, page extensions, and enum extensions that embed E-Document capabilities into existing BC entities. This module contains little orchestration logic; it mainly adds fields, guards, actions, previews, and factboxes so that E-Documents are visible and actionable from the pages users already work in. ## How it works +*Updated: 2026-07-29 -- Refreshed purchase setup, preview, attachment, and page-action behavior.* + The module operates at three levels: **Document Sending Profile integration** (`Sending/` subfolder): The `EDocSendProfileElecDoc` enum extension adds the `"Extended E-Document Service Flow"` option to the Electronic Document field. The `EDocumentSendingProfile` table extension adds `"Electronic Service Flow"` (a workflow code reference). The `EDocSendingProfAttType` enum extension adds `"E-Document"` and `"PDF & E-Document"` as email attachment types. Together, these connect BC's existing document sending infrastructure to the E-Document workflow engine. -**Purchase-side fields**: `EDocPurchaseHeader` adds `"E-Document Link"` (a Guid matching the E-Document's SystemId for linking incoming documents to purchase orders) and `"Amount Incl. VAT To Inv."` (a FlowField summing line amounts for partial invoicing). `EDocPurchaseLine` adds matching `"Amount Incl. VAT To Inv."` with rounding logic and an `HasEDocMatch` helper for order matching. The `EDocPurchPayablesSetup` table extension adds `"E-Document Matching Difference"` (tolerance percentage) and a Copilot learning flag. +**Purchase-side fields**: `EDocPurchaseHeader` adds `"E-Document Link"` (a Guid matching the E-Document's SystemId for linking incoming documents to purchase orders and invoices) and `"Amount Incl. VAT To Inv."` (a FlowField summing line amounts for partial invoicing). It also prevents deleting a purchase order that is linked to an active, non-canceled E-Document. `EDocPurchaseLine` adds matching `"Amount Incl. VAT To Inv."` with rounding logic and an `HasEDocMatch` helper for order matching. The `EDocPurchPayablesSetup` table extension adds matching tolerance, Copilot learning, default posting date behavior for purchase invoices created from e-documents, VAT difference handling, and VAT product posting group resolution. + +**Sales-side fields**: `EDocSalesHeader` adds `"E-Document Link"` to Sales Header with a secondary key and the same `IsLinkedToEDoc` helper pattern used by Purchase Header. This supports incoming sales order draft processing that needs to link a BC sales document back to the E-Document SystemId. **Vendor/Location/Attachment fields**: Vendor and Vendor Template get `"Receive E-Document To"` (controls whether incoming e-docs create Purchase Orders or Purchase Invoices, defaulting to Purchase Order). Location gets `"Transfer Doc. Sending Profile"` for transfer shipment routing. Document Attachment gets `"E-Document Attachment"` and `"E-Document Entry No."` to link attachments back to their source E-Document. -**Page extensions** add E-Document action groups (Open/Create) and factboxes to posted sales invoices, credit memos, service documents, purchase documents, and shipments. Role center extensions (`RoleCenter/` subfolder) add E-Document activities/cues to Accountant, Business Manager, Inventory Manager, and other standard role centers. +**Page extensions** add E-Document action groups (Open/Create), PDF preview factboxes, and document-source actions to posted sales invoices, credit memos, service documents, purchase documents, purchase invoice lists, and shipments. Role center extensions (`RoleCenter/` subfolder) add E-Document activities/cues to Accountant, Business Manager, Inventory Manager, and other standard role centers. ## Things to know -- The `"E-Document Link"` Guid on Purchase Header is indexed (secondary key) for fast lookup during incoming document matching. It stores the E-Document's `SystemId`, not its `"Entry No"`. +*Updated: 2026-07-29 -- Added factbox preview, attachment upload, and failure-message gotchas.* + +- The `"E-Document Link"` Guid on Purchase Header is indexed (secondary key) for fast lookup during incoming document matching. It stores the E-Document's `SystemId`, not its `"Entry No"`. Sales Header now follows the same link pattern for incoming sales order processing. + - The `"Receive E-Document To"` field on Vendor only allows `"Purchase Order"` or `"Purchase Invoice"` -- it uses `ValuesAllowed` to restrict the enum. This determines the default document type created when an incoming e-document is received from that vendor. -- Page extensions follow a consistent pattern: an "E-Document" action group with "Open" (enabled when an E-Document exists for the record) and "Create" (enabled when none exists). The `EDocumentExists` boolean is computed on page load. + +- Purchase invoice pages and posted purchase invoice pages host the `Inbound E-Doc. Picture` part before the standard Incoming Document attachment factbox. The host page drives the preview by calling `GetInboundPdfPreviewEntryNo` with the current record id and, for open purchase invoices, the E-Document link Guid. The preview only appears when the linked unstructured data storage entry is a PDF. + +- E-Document attachment uploads from the standard document attachment factbox rely on event subscribers, not custom upload UI. `OnAfterGetRecRefFail` resolves the E-Document from the factbox's `E-Document Entry No.` filter, and `OnBeforeInsertAttachment` stamps `E-Document Attachment` plus `E-Document Entry No.` before insert. This is why attachment behavior depends on the factbox filters being preserved. + +- Page extensions follow a consistent pattern: an "E-Document" action group with "Open" (enabled when an E-Document exists for the record) and "Create" (enabled when none exists). Posted document Create actions now show an explicit failure message when `CreateEDocumentFromPostedDocumentPage` returns false, so page actions should not silently no-op. + +- Purchase order page actions still expose manual line mapping when the linked E-Document service status is `Order Linked`. The Copilot prompt actions are hidden and obsolete in CLEAN29; AI-assisted matching moved to the purchase draft import experience. + +- The purchase draft subform exposes a hidden, read-only `Name` column for personalization. It displays the matched item, G/L account, resource, fixed asset, allocation account, or item charge name via `GetMatchedEntityName`, not the extracted invoice description. + - The `"Electronic Service Flow"` on Document Sending Profile has a table relation filtered to `Category = 'EDOC'` and `Template = false`, ensuring only enabled E-Document workflows can be selected. + - `EDocOrderMapActivities` is a standalone page (not an extension) providing the order mapping activities cue for role centers. See the [app-level CLAUDE.md](../../docs/CLAUDE.md) for broader architecture context. diff --git a/src/Apps/W1/EDocument/App/src/Format/docs/CLAUDE.md b/src/Apps/W1/EDocument/App/src/Format/docs/CLAUDE.md index 0703d773310..9bd196c0889 100644 --- a/src/Apps/W1/EDocument/App/src/Format/docs/CLAUDE.md +++ b/src/Apps/W1/EDocument/App/src/Format/docs/CLAUDE.md @@ -1,24 +1,29 @@ # Format -PEPPOL BIS 3.0 export and import implementation -- the built-in document format that ships with E-Document Core. This module owns the serialization/deserialization of UBL 2.1 XML for invoices, credit memos, reminders, finance charge memos, and shipments. Localizations add their own formats via the extensible `E-Document` enum; this module provides the W1 baseline. +PEPPOL BIS 3.0 adapter for E-Document Core. The reusable PEPPOL 3.0 validation and XMLport implementations now live in the standalone `Microsoft.Peppol` app; this module keeps the E-Document interface wrapper, the incoming PEPPOL reader, and the E-Document-specific financial result and shipment exporters. Localizations add their own formats via the extensible `E-Document` enum; this module provides the W1 baseline. ## How it works +*Updated: 2026-07-29 -- Reflected the standalone PEPPOL app split, purchase-order export, scoped file extension, and prefixed-root import support.* + The main entry point is `EDocPEPPOLBIS30.Codeunit.al`, which implements the `"E-Document"` interface with five methods: `Check`, `Create`, `CreateBatch`, `GetBasicInfoFromReceivedDocument`, and `GetCompleteInfoFromReceivedDocument`. -**Export path:** `Create` dispatches by document type to dedicated XML generators. Invoices and credit memos use base app XMLports (`Sales Invoice - PEPPOL BIS 3.0`, `Sales Cr.Memo - PEPPOL BIS 3.0`) which produce UBL 2.1 XML with proper `cac`/`cbc` namespaces. Shipments and transfer shipments use custom codeunits (`EDocShipmentExportToXml`, `EDocTransferShptToXML`) that build XML via `XML DOM Management`. Reminders and finance charge memos share the `FinResultsPEPPOLBIS30` XMLport, which wraps them as UBL Invoice documents with special type codes. After generation, `OnAfterCreatePEPPOLXMLDocument` fires as an integration event, letting subscribers modify the XML blob before it leaves the format layer. +**Export path:** `Create` dispatches by document type to dedicated XML generators. Sales and service invoices and credit memos use the standalone PEPPOL 3.0 XMLports (`Sales Invoice - PEPPOL30`, `Sales Cr.Memo - PEPPOL30`) and pass the sales or service format selected in `PEPPOL 3.0 Setup`. Purchase orders use `Export Purchase Order PEPPOL30` with the purchase format from the same setup table. Shipments and transfer shipments still use E-Document Core codeunits (`EDocShipmentExportToXml`, `EDocTransferShptToXML`) that build XML via `XML DOM Management`. Reminders and finance charge memos share the `FinResultsPEPPOLBIS30` XMLport, which wraps them as UBL Invoice documents with special type codes. After generation, `OnAfterCreatePEPPOLXMLDocument` fires as an integration event, letting subscribers modify the XML blob before it leaves the format layer. -**Import path:** `EDocImportPEPPOLBIS30` parses incoming UBL XML into temporary `Purchase Header` / `Purchase Line` records. It uses `XML Buffer` (not DOM) for XPath-style traversal. Vendor resolution cascades through three strategies: GLN/VAT number lookup, then service participant matching, then name+address fuzzy matching. +**Import path:** `EDocImportPEPPOLBIS30` parses incoming UBL XML into temporary `Purchase Header` / `Purchase Line` records. It uses `XML Buffer` (not DOM) for XPath-style traversal. The reader detects the root element path once and builds all invoice or credit memo lookups from that path, so PEPPOL files whose root element has a namespace prefix are handled the same as unprefixed roots. Vendor resolution cascades through GLN/VAT number lookup, service participant matching, then name+address fuzzy matching. -**Validation:** `Check` delegates to three separate validators depending on source document type: base app `PEPPOL Validation` for sales documents, `PEPPOL Service Validation` for service documents, and `EDocPEPPOLValidation` (in this module) for reminders and finance charge memos. The in-module validator checks company info completeness, country/region codes, currency codes, and customer identification. +**Validation:** `Check` uses the `PEPPOL30 Validation` interface implementation selected by `PEPPOL 3.0 Setup` for sales and service documents, and keeps `EDocPEPPOLValidation` in this module for reminders and finance charge memos. The in-module validator checks company info completeness, country/region codes, currency codes, and customer identification. ## Things to know +*Updated: 2026-07-29 -- Added current PEPPOL setup and file-extension behavior.* + - `CreateBatch` is intentionally empty -- PEPPOL BIS 3.0 does not support batch export. The interface method exists only to satisfy the contract. -- The `"Embed PDF in export"` flag on E-Document Service controls whether a base64-encoded PDF is embedded inside the XML. This applies to invoices, credit memos, and shipments. +- The `"Embed PDF in export"` flag on E-Document Service controls whether a base64-encoded PDF is embedded inside the XML. This applies to invoices, credit memos, shipments, and purchase orders. +- `SetFileExt` appends `.xml` only when the E-Document log's `Document Format` is `PEPPOL BIS 3.0`, so other formats that export through the log are not forced to use the PEPPOL extension. - Currency on import uses a subtle convention: if the document currency matches `GLSetup."LCY Code"`, it is left blank on the E-Document (BC convention for local currency). Only foreign currencies are stored explicitly. - The `EDocumentStructuredFormat` enum is marked `ObsoleteState = Pending` for removal in v26 -- it bridges an older structured-format reader pattern that is being replaced by newer processing interfaces. -- When PEPPOL BIS 3.0 is selected as document format on a service, the `OnAfterValidateDocumentFormat` subscriber auto-populates supported document types (Sales Invoice, Sales Credit Memo, Service Invoice, Service Credit Memo). +- When PEPPOL BIS 3.0 is selected as document format on a service, the `OnAfterValidateDocumentFormat` subscriber auto-populates only Sales Invoice, Sales Credit Memo, Service Invoice, and Service Credit Memo. Purchase-order export exists, but this subscriber does not add that type automatically. - Shipment exports use a custom XML schema (not standard UBL Despatch Advice) -- they are simpler, flat structures with supplier/customer/delivery sections rather than full PEPPOL Despatch. See the [app-level CLAUDE.md](../../docs/CLAUDE.md) for broader architecture context. diff --git a/src/Apps/W1/EDocument/App/src/Helpers/docs/CLAUDE.md b/src/Apps/W1/EDocument/App/src/Helpers/docs/CLAUDE.md index cd5dc626f50..9db667dae2d 100644 --- a/src/Apps/W1/EDocument/App/src/Helpers/docs/CLAUDE.md +++ b/src/Apps/W1/EDocument/App/src/Helpers/docs/CLAUDE.md @@ -4,19 +4,25 @@ Utility codeunits used across the E-Document framework. These are not standalone ## How it works -**`EDocumentErrorHelper`** implements the collecting-parameter pattern for errors. Instead of throwing on first failure, callers use `LogSimpleErrorMessage` or `LogErrorMessage` to accumulate errors against an E-Document's context. The errors are stored in BC's `"Error Message"` table, scoped by the E-Document's `RecordId`. After processing completes, the framework checks `HasErrors` to decide whether to proceed or mark the document as failed. This pattern is essential because a single E-Document can fail validation for multiple independent reasons, and the user needs to see all of them at once. All errors are also forwarded to Feature Telemetry for monitoring. +*Updated: 2026-07-29 -- Added persisted error throwing, PDF preview lookup, customer matching, import telemetry, and safer ADI JSON handling.* -**`EDocumentImportHelper`** handles the resolution of incoming document data to BC master data. Its main responsibilities are: resolving units of measure (by code, then by International Standard Code, then by description), finding items (by Item Reference, GTIN, vendor item number, or item description), and finding vendors (by GLN, VAT registration number, service participant, or name+address). Each resolution method follows a cascade pattern -- try the most specific match first, fall back to less specific. Errors are logged (not thrown) when no match is found. +**`EDocumentErrorHelper`** implements the collecting-parameter pattern for errors. Instead of throwing on first failure, callers use `LogSimpleErrorMessage` or `LogErrorMessage` to accumulate errors against an E-Document's context. The errors are stored in BC's `"Error Message"` table, scoped by the E-Document's `RecordId`. After processing completes, the framework checks `HasErrors` or calls `ThrowIfHasErrors` to decide whether to proceed or surface the collected errors. `ThrowIfHasErrors` copies the persisted messages to a temporary record, commits them, and throws the collected error so the messages survive the failure. All errors are also forwarded to Feature Telemetry for monitoring. -**`EDocumentHelper`** provides document-level utilities: checking if a RecordRef is an E-Document (`IsElectronicDocument`), retrieving the E-Document Service for a document (checking both live and archived workflow step instances), getting the E-Document blob from logs, and enabling HTTP client requests for the core extension. For outbound documents, service resolution walks the workflow step instance chain; for inbound, it uses the service status table. +**`EDocumentImportHelper`** handles the resolution of incoming document data to BC master data. Its main responsibilities are: resolving units of measure (by code, then by International Standard Code, then by description), finding items (by Item Reference, GTIN, vendor item number, or item description), finding vendors (by number, GLN, VAT registration number, service participant, bank account, or name+address), and fuzzy-matching customers by name+address. Each resolution method follows a cascade pattern -- try the most specific match first, fall back to less specific. Vendor matching also records session telemetry for the match method and for name-only candidates whose address did not match. Errors are logged (not thrown) when no match is found. + +**`EDocumentHelper`** provides document-level utilities: checking if a RecordRef is an E-Document (`IsElectronicDocument`), retrieving the E-Document Service for a document (checking both live and archived workflow step instances), getting the E-Document blob from logs, finding an inbound PDF preview data-storage entry, and enabling HTTP client requests for the core extension. For outbound documents, service resolution walks the workflow step instance chain; for inbound, it uses the service status table. PDF preview lookup first follows an `E-Document Link` SystemId for open purchase documents, then falls back to the source document RecordId for posted documents, and only returns entries whose unstructured data is actually PDF. **`EDocumentLogHelper`** is a thin public facade over the internal `"E-Document Log"` codeunit, exposing `InsertLog` and `InsertIntegrationLog` for connector implementations that need to record HTTP request/response pairs or status transitions. -**`EDocumentJsonHelper`** is internal, used for Azure Document Intelligence integration. It parses a specific JSON structure (`outputs.1.result.fields`/`items`) and extracts typed values (text, date, number, currency) into AL variables. +**`EDocumentJsonHelper`** is internal, used for Azure Document Intelligence integration. It parses a specific JSON structure (`outputs.1.result.fields`/`items`) and extracts typed values (text, date, number, currency) into AL variables. Missing `outputs`, `1`, `result`, or `fields` properties now log warning telemetry and return an empty object instead of causing an untrappable JSON error. ## Things to know +*Updated: 2026-07-29 -- Added gotchas for collected-error throwing and inbound PDF preview lookup.* + - `LogSimpleErrorMessage` vs `LogErrorMessage`: the simple version only takes a message string. The full version also takes a related record and field number, which enables drill-down navigation from the error message to the source record. +- Use `ThrowIfHasErrors` when the caller needs to stop after collecting multiple errors but still keep those messages available to the user after the transaction fails. +- `GetInboundPdfPreviewEntryNo` returns 0 for non-PDF unstructured data, missing links, or documents without an E-Document source. Callers must treat 0 as "no preview" rather than as an error. - The error helper's `ClearErrorMessages` is called before re-processing to avoid stacking duplicate errors from retry attempts. - `EDocumentImportHelper` UOM resolution tries three paths: exact Code match, International Standard Code match, then Description match. If all fail, the error is logged but import continues -- the line will just have no UOM set. - `EDocumentHelper.AllowEDocumentCoreHttpCalls` directly manipulates the `"NAV App Setting"` table using the E-Document Core extension's hardcoded app ID (`e1d97edc-c239-46b4-8d84-6368bdf67c8b`). diff --git a/src/Apps/W1/EDocument/App/src/Processing/Import/docs/CLAUDE.md b/src/Apps/W1/EDocument/App/src/Processing/Import/docs/CLAUDE.md index ef28c2289a7..a398274f2a5 100644 --- a/src/Apps/W1/EDocument/App/src/Processing/Import/docs/CLAUDE.md +++ b/src/Apps/W1/EDocument/App/src/Processing/Import/docs/CLAUDE.md @@ -1,27 +1,53 @@ # Import pipeline -The V2 import pipeline converts a received blob (XML, JSON, PDF) into a posted BC purchase invoice through four discrete stages, each producing an intermediate status that can be undone and re-run. The pipeline is orchestrated by `ImportEDocumentProcess.Codeunit.al`, which dispatches to interface implementations so that every stage is replaceable by extensions. +The V2 import pipeline converts a received blob (XML, JSON, PDF) into a reviewable Business Central draft and then into the target BC document. It is no longer purchase-invoice-only: built-in process draft implementations now create purchase invoices, purchase credit memos, and inbound sales orders. The pipeline is orchestrated by `ImportEDocumentProcess.Codeunit.al`, which dispatches each stage through enum-backed interfaces so formats and providers can be swapped without changing the orchestrator. + +*Updated: 2026-07-29 -- Import V2 now routes to purchase invoice, purchase credit memo, or sales order drafts, and format selection is enum-driven.* ## How it works -An incoming E-Document enters the pipeline at status `Unprocessed` with a raw blob in `E-Doc. Data Storage`. Each step advances the status by one notch: **Structure** converts unstructured data (e.g. a PDF via Azure Document Intelligence) into a structured blob and moves to `Readable`. **Read into draft** parses that structured blob into purchase staging tables (`E-Document Purchase Header` / `E-Document Purchase Line`) and moves to `Ready for draft`. **Prepare draft** resolves vendor, items, UOM, GL accounts, and purchase order matches -- filling in the `[BC]` validated columns on those staging tables -- then moves to `Draft ready`. **Finish draft** creates the actual BC purchase invoice (or links to an existing document), writes `E-Doc. Record Link` entries for traceability, and moves to `Processed`. +An incoming E-Document enters the pipeline at status `Unprocessed` with a raw blob in `E-Doc. Data Storage`. Each step advances the status by one notch: + +- **Structure received data** converts unstructured data into a structured blob and moves to `Readable`. XML and JSON use `Already Structured`. PDF now prefers `MLLM`, which calls Azure OpenAI with the prompt in `App/.resources/Prompts/EDocMLLMExtraction-SystemPrompt.md` and falls back to ADI if the model returns empty or invalid JSON. ADI remains available and falls back to `Blank Draft` when it cannot produce structured data. + +- **Read into draft** parses the structured blob into staging tables and moves to `Ready for draft`. PEPPOL reads invoices, credit notes, orders, and order responses. `Data Exchange Purchase` lets configured Data Exchange definitions feed the V2 purchase staging tables. External apps can add more readers, such as XRechnung and OIOUBL, by extending the read-into-draft enum. + +- **Prepare draft** resolves BC entities and moves to `Draft ready`. Purchase drafts resolve vendor, purchase order, UOM, item or account, VAT product posting group, historical values, GL account suggestions, and deferrals. Sales drafts resolve customer, UOM, and sales line item references. -Each step is undoable. Undoing Finish Draft deletes the purchase invoice and restores PO matches. Undoing Prepare Draft clears header mappings, vendor assignment, and resets Document Type. Undoing Structure clears the structured data pointer. The user can fix data at any stage and re-run forward from there. +- **Finish draft** creates or links the BC document and moves to `Processed`. Purchase invoice, purchase credit memo, and sales order finalizers all write `E-Doc. Record Link` entries or `E-Document Link` values for traceability and attachment movement. -V1 services are still supported: when `GetImportProcessVersion()` returns `Version 1.0`, the pipeline collapses all stages into a single "Finish draft" call that delegates to the legacy `E-Doc. Import` codeunit. +Each step is undoable. Undoing Finish Draft delegates to the current `E-Document Type` implementation and clears `Document Record ID`. Purchase and sales helper code moves attachments back and clears the `E-Document Link`; it does not delete the created BC document. Undoing Prepare Draft clears header mappings, bill-to/pay-to fields, and `Document Type`. Undoing Structure clears the structured data pointer. The user can fix data at any stage and re-run forward from there. + +V1 services are still supported: when `GetImportProcessVersion()` returns `Version 1.0`, the pipeline collapses to the legacy import code path and only the Finish Draft step is meaningful. New services initialize with `Import Process = Version 2.0`, and for V2 services the service card exposes `Read into Draft Impl.` as the user-facing draft format fallback for already-structured documents. + +*Updated: 2026-07-29 -- Added MLLM, Data Exchange bridge, sales order, credit memo, and V2 default behavior.* ## Things to know -- The pipeline status is an ordered enum (`Unprocessed` = 0 through `Processed` = 4). `StatusStepIndex()` maps each status to a numeric index used for comparison and navigation -- this is how `GetNextStep()` / `GetPreviousStep()` work. +- The pipeline status is an ordered enum (`Unprocessed` = 0 through `Processed` = 4). `StatusStepIndex()` maps each status to a numeric index used for comparison and navigation -- this is how `GetNextStep()` and `GetPreviousStep()` work. + +- The `E-Doc. Import Parameters` table is temporary and controls pipeline execution: which step to run, whether to target a step or desired status, processing customizations, V1 compatibility flags, and `Existing Doc. RecordId` for linking to an existing document instead of creating one. -- The `E-Doc. Import Parameters` table is temporary and controls pipeline execution: which step to run, whether to target a step or a desired status, processing customizations, and V1-compatibility flags. +- Interface dispatch is layered: `IEDocFileFormat` determines the preferred `IStructureReceivedEDocument`, which returns an `IStructuredDataType` that can override the `IStructuredFormatReader`, which returns the `IProcessStructuredData` enum. If no reader is set by the document or structuring step, the service-level `Read into Draft Impl.` field is used. -- Interface dispatch is layered: `IEDocFileFormat` determines the preferred `IStructureReceivedEDocument`, which returns an `IStructuredDataType` that specifies the `IStructuredFormatReader`, which returns the `IProcessStructuredData` enum. Each stage's output feeds the next stage's interface selection. +- The `E-Doc. Process Draft` enum now separates `Purchase Invoice`, `Purchase Credit Memo`, and `Sales Order`. The old `Purchase Document` value is pending obsolete for v29 and should not be described as the current route. -- The `E-Doc. Proc. Customizations` enum is a multi-interface enum that bundles `IVendorProvider`, `IPurchaseOrderProvider`, `IPurchaseLineProvider`, `IUnitOfMeasureProvider`, and `IEDocumentCreatePurchaseInvoice` with defaults from `EDocProviders.Codeunit.al`. Extensions add a new enum value to swap all five at once. +- The `E-Doc. Proc. Customizations` enum is a multi-interface enum. It now bundles purchase providers, purchase invoice creation, purchase credit memo creation, customer resolution, sales line resolution, and sales order creation. Extensions add one enum value to swap a coherent set of providers. -- AI-assisted matching runs during Prepare Draft: historical matching first, then Copilot GL account matching for remaining unresolved lines, then deferral matching. Each step commits before invoking the next codeunit to isolate failures. +- AI-assisted purchase line matching runs during Prepare Draft: historical matching first, then GL account matching for remaining unresolved lines, then deferral matching. The old Purchase Order Matching Copilot is obsolete; PO matching itself remains a manual and rules-based purchase draft feature. + +- Purchase draft dates are deliberately carried forward. `Document Date` and `Due Date` from the staging header are copied to the E-Document during Prepare Draft, then validated onto the purchase document during Finish Draft. If purchase setup requests it, posting date defaults from the draft document date. + +- VAT handling is split between Prepare and Finish. Prepare resolves `[BC] VAT Prod. Posting Group` from extracted VAT rates when purchase setup enables it, computes whether a VAT amount difference is allowed, and logs why it was or was not applied. Finish distributes the allowed difference across purchase lines. + +- Errors raised while finalizing fields are wrapped with validation context by `E-Doc. Import Error Context`, so a failed additional field or purchase field validation points to the field being applied instead of surfacing only the lower-level error. + +- Reader failures, including JSON parsing failures from ADI or MLLM structured data, are run through `E-Doc. Import.RunConfiguredImportStep()`. That wrapper logs the error on the E-Document instead of letting malformed structured data break the session. + +- Purchase invoice pages outside this folder now show an inbound PDF preview factbox when `E-Document Helper.GetInboundPdfPreviewEntryNo()` can find a PDF source linked to the E-Document. The purchase draft line subform also has a hidden `Name` column that users can personalize in; it displays the matched item, account, or other entity name rather than the extracted invoice description. - The history system is populated by event subscribers on `Purch.-Post`: `OnAfterPurchInvLineInsert` and `OnAfterPostPurchaseDoc` create entries in `E-Doc. Purchase Line History` and `E-Doc. Vendor Assign. History`, completing the learning loop. -- See `../docs/CLAUDE.md` for the parent Processing module context. The `src/Processing/Interfaces/` folder defines 18 interfaces that underpin this pipeline. +- See `../docs/CLAUDE.md` for the parent Processing module context. The interface contracts live in `src/Processing/Interfaces/`, with import-specific implementations under this folder. + +*Updated: 2026-07-29 -- Added current enum values, setup fallback, VAT/date behavior, and obsolete Copilot note.* diff --git a/src/Apps/W1/EDocument/App/src/Processing/Import/docs/business-logic.md b/src/Apps/W1/EDocument/App/src/Processing/Import/docs/business-logic.md index 896ff52c330..0a8cc25e956 100644 --- a/src/Apps/W1/EDocument/App/src/Processing/Import/docs/business-logic.md +++ b/src/Apps/W1/EDocument/App/src/Processing/Import/docs/business-logic.md @@ -2,105 +2,212 @@ ## Pipeline overview +*Updated: 2026-07-29 -- Flow now shows reader selection, process draft routing, purchase credit memo, and sales order outcomes.* + ```mermaid flowchart TD - A["Unprocessed
(raw blob)"] -->|"Structure received data"| B["Readable
(structured blob)"] - B -->|"Read into draft"| C["Ready for draft
(staging tables populated)"] - C -->|"Prepare draft"| D["Draft ready
(BC entities resolved)"] - D -->|"Finish draft"| E["Processed
(Purchase Invoice created)"] + A["Unprocessed
(raw blob)"] --> S["Structure received data
IEDocFileFormat chooses structuring"] + S --> B["Readable
(structured blob)"] + B --> R["Read into draft
IStructuredFormatReader parses content"] + R --> C["Ready for draft
(staging tables populated)"] + C --> P{ "Process draft impl." } + P -->|"Purchase Invoice"| PI["Prepare purchase invoice draft"] + P -->|"Purchase Credit Memo"| PC["Prepare purchase credit memo draft"] + P -->|"Sales Order"| SO["Prepare sales order draft"] + PI --> D["Draft ready
(BC entities resolved)"] + PC --> D + SO --> D + D --> F{ "E-Document Type" } + F -->|"Purchase Invoice"| FI["Create or link purchase invoice"] + F -->|"Purchase Credit Memo"| FC["Create or link purchase credit memo"] + F -->|"Sales Order"| FS["Create sales order"] + FI --> E["Processed"] + FC --> E + FS --> E E -.->|"Undo finish"| D D -.->|"Undo prepare"| C C -.->|"Undo read"| B B -.->|"Undo structure"| A ``` -The pipeline is driven by `ImportEDocumentProcess.Codeunit.al`. Its `OnRun()` trigger checks the service's import process version. V1 services skip straight to a legacy code path. V2 services dispatch to one of four local procedures based on the configured step. After each step, the processing status is advanced (or rolled back for undo) and the E-Document status is recalculated. +The pipeline is driven by `ImportEDocumentProcess.Codeunit.al`. Its `OnRun()` trigger checks the service's import process version. V1 services skip straight to a legacy code path. V2 services dispatch to one of four procedures based on the configured step. After each step, the processing status is advanced or rolled back and the E-Document status is recalculated. ## Stage 1 -- Structure received data +*Updated: 2026-07-29 -- PDF now prefers MLLM, with ADI as fallback rather than the primary PDF path.* + **Transition:** Unprocessed --> Readable -The E-Document arrives with an `Unstructured Data Entry No.` pointing to a raw blob in `E-Doc. Data Storage`. The Structure step loads that blob, resolves the file format via `IEDocFileFormat` (the enum value stored on the data storage record), and asks it for a `PreferredStructureDataImplementation()`. +The E-Document arrives with an `Unstructured Data Entry No.` pointing to a raw blob in `E-Doc. Data Storage`. The Structure step loads that blob, resolves the file format via `IEDocFileFormat`, and asks it for `PreferredStructureDataImplementation()` when the document has not already chosen one. -For XML, the preferred implementation is `"Already Structured"` -- the blob is already parseable, so the unstructured and structured data entry numbers are set to the same value and nothing is converted. For PDF, the preferred implementation is `"ADI"` -- Azure Document Intelligence converts the binary into a JSON blob. The ADI handler (`EDocumentADIHandler.Codeunit.al`) base64-encodes the blob, calls `AzureDocumentIntelligence.AnalyzeInvoice()`, and stores the JSON result. If ADI fails (returns empty), it falls back to `"Blank Draft"` so the user can populate fields manually. +For XML and JSON, the preferred implementation is `Already Structured`; the raw data entry becomes the structured data entry. For PDF, `E-Doc. PDF File Format` now returns `MLLM`. `E-Document MLLM Handler` sends the PDF to Azure OpenAI with a UBL-shaped schema and the `EDocMLLMExtraction-SystemPrompt.md` resource. That prompt explicitly tells the model to extract only visible values, keep customer and vendor roles separate, use XML decimal format, and avoid combining post-discount prices with discount percentages. If MLLM returns empty text, invalid JSON, or JSON missing required vendor fields, the handler falls back to ADI. -When the data is actually converted (i.e. not "Already Structured"), the original unstructured blob is saved as a document attachment on the E-Document for reference. The structured result is stored as a new `E-Doc. Data Storage` entry via the log, and its entry number goes into `E-Document."Structured Data Entry No."`. +`E-Document ADI Handler` still implements both structuring and reading. It registers the E-Document Analysis capability when needed, calls `AzureDocumentIntelligence.AnalyzeInvoice()`, stores JSON when it succeeds, and switches to `Blank Draft` when ADI returns an empty result. The ADI reader maps Azure field names such as `vendorName`, `invoiceId`, `productCode`, `taxRate`, and `tax` into purchase staging fields. -The `IStructuredDataType` returned by the structure implementation also specifies which `IStructuredFormatReader` should be used in the next stage. If the structure step says ADI, the reader will be ADI. If it says nothing (`Unspecified`), the service's default reader is used. This chaining means a single PDF upload can auto-select the entire downstream pipeline. +When the data is converted, the original unstructured blob is saved as a document attachment on the E-Document. The structured result is stored as a new `E-Doc. Data Storage` entry through the log, and its entry number is saved on the E-Document. A structuring implementation can also force the next reader by returning a non-`Unspecified` `GetReadIntoDraftImpl()` value; this overrides a reader that was already on the E-Document and logs telemetry. ## Stage 2 -- Read into draft +*Updated: 2026-07-29 -- Added PEPPOL credit notes, orders, order responses, Data Exchange bridge, and external XRechnung/OIOUBL readers.* + **Transition:** Readable --> Ready for draft -The structured blob is loaded and passed to the `IStructuredFormatReader` determined in Stage 1. Two readers ship in the core: +The structured blob is loaded and passed to the `IStructuredFormatReader` determined in Stage 1. If the E-Document still has `Read into Draft Impl. = Unspecified`, the read step uses the service's `Read into Draft Impl.` field. This field is shown as **Draft Format** on the service card for V2 services, so already-structured services must choose a reader such as PEPPOL, Data Exchange Purchase, or a country-app reader. + +Built-in readers in the core import folder are: + +- **PEPPOL** (`EDocumentPEPPOLHandler.Codeunit.al`): Parses UBL Invoice, CreditNote, Order, and OrderResponse XML. Invoices route to `Purchase Invoice`, credit notes route to `Purchase Credit Memo`, and orders route to `Sales Order`. Order responses are stored as messages on the matching outbound E-Document and then stop the import carrier. + +- **ADI** (`EDocumentADIHandler.Codeunit.al`): Parses the ADI JSON schema into purchase staging. It treats zero or negative quantity as one, resolves VAT rate from `taxRate` first, and falls back to computing a percentage from a tax amount when the subtotal is available. + +- **MLLM** (`EDocumentMLLMHandler.Codeunit.al`): Parses the model's UBL-shaped JSON through `EDocMLLMSchemaHelper`. Decimal parsing uses XML decimal format with evaluation style 9, matching the prompt instruction that model output must use `.` as decimal separator and no thousands separators. + +- **Data Exchange Purchase** (`EDocDataExchPurchHandler.Codeunit.al`): Detects the configured Data Exchange Definition by matching the XML root namespace against service data exchange setup, runs the standard Data Exchange pipeline, and bridge-maps `Intermediate Data Import` into the V2 purchase staging tables. + +External apps extend the same enum. XRechnung and OIOUBL both add `E-Doc. Read into Draft` values with `IStructuredFormatReader` implementations that populate the same purchase staging tables and return either `Purchase Invoice` or `Purchase Credit Memo`. + +Both purchase and sales staging tables keep the same design: fields 2-100 store extracted source data, while fields 101-200 store validated BC references that users or providers can change before finishing. + +Reader errors are intentionally caught one level above the reader. `E-Doc. Import.RunConfiguredImportStep()` runs `Import E-Document Process` through `Codeunit.Run`, wraps any available validation context, and logs the error against the E-Document. This is especially important for JSON readers such as ADI and MLLM, where malformed structured data should become an import error instead of an untrappable session failure. + +### Data Exchange bridge -- **PEPPOL** (`EDocumentPEPPOLHandler.Codeunit.al`): Parses UBL 2.1 XML. Extracts vendor party info, invoice totals, currency, dates, and iterates `cac:InvoiceLine` nodes to create `E-Document Purchase Line` records. Uses XPath with UBL namespace prefixes. -- **ADI** (`EDocumentADIHandler.Codeunit.al`): Parses the ADI JSON schema. Maps ADI field names like `vendorName`, `invoiceId`, `productCode` to the staging table columns. Sets quantity to 1 when ADI returns zero or negative. +*Updated: 2026-07-29 -- Documented why the bridge exists and how it maps legacy Data Exchange output into V2 staging.* -Both readers insert an `E-Document Purchase Header` and one `E-Document Purchase Line` per invoice line. These staging tables use **dual nomenclature**: fields 2-100 hold the raw external data exactly as extracted (e.g. `"Vendor Company Name"`, `"Product Code"`), while fields 101-200 hold validated BC references (e.g. `"[BC] Vendor No."`, `"[BC] Purchase Type No."`). At this stage, only the external-data columns are populated. +The Data Exchange reader exists so older Data Exchange Definition investments can feed the V2 import pipeline. It does not ask Data Exchange to create the final purchase document. Instead, it runs `ProcessDataExchange()` to populate Data Exchange's intermediate tables, then maps those intermediate rows into `E-Document Purchase Header` and `E-Document Purchase Line`. -The reader returns an `E-Doc. Process Draft` enum value (currently only `"Purchase Document"`) that determines which `IProcessStructuredData` implementation runs in Stage 3. +The bridge handles work that declarative mappings cannot express cleanly: + +- It selects a definition by XML namespace from `E-Doc. Service Data Exch. Def.`. + +- It pre-inserts the purchase staging header so the Data Exchange post-mapping codeunit can write to it. + +- It applies BC's blank-LCY currency convention after mapping. + +- It promotes PEPPOL document-level charges into extra purchase staging lines through `E-Doc. PEPPOL DX Post-Mapping`. + +- It decodes embedded attachments from intermediate data and attaches them to the E-Document. + +After bridging, the temporary Data Exchange and intermediate records are deleted. The V2 pipeline continues with Prepare Draft exactly as if a native reader had populated the staging tables. ## Stage 3 -- Prepare draft +*Updated: 2026-07-29 -- Split purchase and sales preparation, added VAT rate resolution, VAT difference handling, and vendor telemetry.* + **Transition:** Ready for draft --> Draft ready -This is where the system resolves external data into BC entities. `PreparePurchaseEDocDraft.Codeunit.al` implements `IProcessStructuredData` and orchestrates the resolution. +Prepare Draft resolves extracted data into BC entities. `E-Doc. Process Draft` chooses the implementation. + +### Purchase preparation + +`Prepare Purchase E-Doc. Draft` and `EDoc Prepare Cr. Memo Draft` both delegate to `EDoc Prepare Purch. Draft`. The shared purchase helper resolves vendor, purchase order, unit of measure, purchase line type and number, VAT product posting group, historical values, GL account suggestions, and deferral suggestions. -### Vendor resolution +Vendor resolution uses the `IVendorProvider` from processing customizations. The default provider tries VAT ID and GLN, then service participant by external ID scoped to the service and then unscoped, then name and address. If the provider does not assign a vendor, history is searched by GLN, VAT ID, company name, and address, newest first. The import session telemetry records whether vendor information was present, whether a vendor was assigned, and whether the source was already assigned, provider, history, or none. -`GetVendor()` delegates to `IVendorProvider`. The default provider (`EDocProviders.Codeunit.al`) tries a four-step waterfall: +Line enrichment runs only after a vendor is known. The default UOM provider tries code, international standard code, and description. The default purchase line provider tries item references by vendor, product code, UOM, and validity dates, then falls back to Text-to-Account Mapping. Each successful provider path stores activity-log reasoning. -1. **VAT ID + GLN** -- calls `EDocumentImportHelper.FindVendor()` with the extracted VAT ID and GLN -2. **Service Participant** -- looks up the vendor's external ID in the `Service Participant` table, first scoped to the specific service, then across all services -3. **Name + Address** -- calls `FindVendorByNameAndAddress()` as a last resort -4. **Historical** -- if the direct provider returns nothing, `EDocPurchaseHistMapping.FindRelatedPurchaseHeaderInHistory()` searches `E-Doc. Vendor Assign. History` by GLN, then VAT ID, then company name, then address (most specific first, most recent first). If a match is found, the vendor number is copied from the linked posted purchase invoice header. +After direct provider matching, `ResolveVATProductPostingGroups()` can fill `[BC] VAT Prod. Posting Group` from the extracted line VAT rate when purchase setup enables `Resolve VAT Group Purch EDoc`. It uses the vendor's VAT business posting group and looks for a single normal or reverse charge VAT Posting Setup at that rate. If no unique setup exists, the draft line gets an activity-log explanation. -### Line enrichment +Then AI-assisted matching runs in this order: -For each `E-Document Purchase Line`, the pipeline resolves: +1. Historical matching applies posted purchase invoice history to unresolved lines. +2. GL account matching uses the prompt in `GLAccountMatching-SystemPrompt.md` and processes all plausible lines. +3. Deferral matching suggests deferral codes for lines with a type but no deferral. -- **Unit of Measure** via `IUnitOfMeasureProvider` -- tries Code, then International Standard Code, then Description -- **Purchase line type and number** via `IPurchaseLineProvider` -- the default implementation tries Item Reference first (filtering by vendor, product code, UOM, and date validity), then falls back to Text-to-Account Mapping. Each successful match writes an Activity Log entry explaining the reasoning. +Each AI step commits before running. Deferral matching is deliberately non-blocking. -### Purchase order matching +Finally, the helper computes a VAT amount difference from the staged total VAT and line VAT amounts. The difference is only considered usable when purchase setup enables E-Document VAT differences, purchase setup allows VAT differences generally, and the value does not exceed the General Ledger Setup maximum. The decision is logged on the header either way. -`IPurchaseOrderProvider.GetPurchaseOrder()` checks if the `"Purchase Order No."` extracted from the document matches an existing PO. If found, the PO number is stored on the purchase header's `[BC] Purchase Order No.` field for use during Finish Draft. +### Sales order preparation -### Copilot-assisted matching +`Prepare Sales E-Doc. Draft` delegates to `EDoc Prepare Sales Draft`. It resolves `[BC] Customer No.` through `ICustomerProvider` and resolves each sales line through `ISalesLineProvider`. -After the direct resolution pass, three Copilot codeunits run sequentially on lines that still lack a `[BC] Purchase Type No.`: +The default customer provider tries buyer GLN on Customer, service participant by buyer GLN, service participant by buyer external ID, buyer VAT ID on Customer, then name and address. The default sales line provider tries seller item ID directly against Item, standard item ID through Item GTIN or bar-code item reference, then buyer item ID through customer item reference. Successful matches log activity reasoning and line telemetry. -1. **Historical matching** (`E-Doc. Historical Matching`) -- searches `E-Doc. Purchase Line History` for past invoices with the same product code or similar description from the same vendor. Applies the posted line's type, number, deferral code, dimensions, and UOM to the draft. -2. **GL account matching** (`E-Doc. GL Account Matching`) -- uses AI to suggest a G/L account for lines with no item match. -3. **Deferral matching** (`E-Doc. Deferral Matching`) -- for lines that have a type but no deferral code, suggests a deferral template. +### Dates on the E-Document -Each step commits before invoking the next to isolate failures. Deferral matching swallows errors silently to avoid blocking the pipeline. +After Prepare Draft, `ImportEDocumentProcess` copies purchase staging `Document Date` and `Due Date` onto the E-Document when present. This is why incoming document lists and factboxes can show the document dates before a purchase document is finalized. ## Stage 4 -- Finish draft +*Updated: 2026-07-29 -- Added finalization routing, credit memo and sales order paths, posting description, dimensions, VAT difference, and validation context.* + **Transition:** Draft ready --> Processed -`IEDocumentFinishDraft.ApplyDraftToBC()` creates the actual BC document. The default implementation (`EDocCreatePurchaseInvoice.Codeunit.al`) performs: +Finish Draft chooses an `IEDocumentFinishDraft` implementation from `E-Document Type`. -1. **Validation**: Checks that all draft lines have a type and number. Verifies PO match validity -- matched lines must be receivable and have UOM info. -2. **Receipt suggestion**: For lines matched to PO lines, `SuggestReceiptsForMatchedOrderLines()` proposes receipt lines. -3. **Invoice creation**: Creates a `Purchase Header` (type Invoice), sets vendor, dates, currency, and vendor invoice number. Checks for duplicate external document numbers. Inserts lines without PO matches first, then lines grouped by receipt number with comment-line separators. -4. **PO match transfer**: `TransferPOMatchesFromEDocumentToInvoice()` moves match records from the E-Document to the purchase invoice. -5. **Traceability**: `EDocRecordLink.InsertEDocumentHeaderLink()` and `InsertEDocumentLineLink()` create `E-Doc. Record Link` entries linking draft records to their BC counterparts via SystemId. -6. **Post-creation**: Copies document attachments from the E-Document to the purchase header, applies invoice discount, sets `E-Document Link` GUID on the purchase header, and validates document totals. +```mermaid +flowchart TD + A["Finish draft"] --> B{ "Document Type" } + B -->|"None"| Z["Exit without creating a BC document"] + B -->|"Purchase Invoice"| PI["Validate PO matches and draft lines"] + B -->|"Purchase Credit Memo"| PC["Validate draft lines"] + B -->|"Sales Order"| SO["Validate sales draft lines"] + PI --> L{ "Existing Doc. RecordId set?" } + PC --> L + L -->|"Yes"| X["Link E-Document to existing purchase document"] + L -->|"No invoice"| CI["Create purchase invoice"] + L -->|"No credit memo"| CC["Create purchase credit memo"] + SO --> CS["Create sales order"] + CI --> H["Set dates, posting description, currency, lines, links, attachments"] + CC --> H + X --> H + CS --> SH["Set customer, external document number, lines, link, attachments"] + H --> V["Apply discounts and VAT difference where relevant"] + SH --> E["Processed"] + V --> E +``` + +### Purchase invoice finalization + +`E-Doc. Create Purchase Invoice` first validates PO matches. Matched lines must be valid for the current receipt configuration, must not exceed invoiceable quantity, and must have required UOM information. It then either links to `Existing Doc. RecordId` or delegates invoice creation to `IEDocumentCreatePurchaseInvoice` from processing customizations. + +The default invoice creator: + +- Requires all draft lines to have both type and number. + +- Creates a purchase invoice for the resolved vendor. + +- Validates `Document Date`, `Due Date`, `Vendor Invoice No.`, currency, line quantity, direct unit cost, discounts, deferral code, dimensions, VAT product posting group, and item reference through `E-Doc. Purch. Doc. Helper`. + +- Checks duplicate posted purchase invoices by external document number after the document date is validated, so the duplicate check uses the draft document date context. + +- Applies the configured posting date default from purchase setup when it is set to Document Date. + +- Copies the staged `Posting Description` to the purchase invoice. -Alternatively, if `EDocImportParameters."Existing Doc. RecordId"` is set, no new invoice is created -- the E-Document is linked to an existing purchase document instead. +- Creates non-PO lines first, then receipt-grouped matched lines with comment separators. + +- Transfers PO match records from the E-Document to the invoice. + +- Applies invoice discount and distributes any allowed VAT difference across purchase lines. + +- Writes header and line `E-Doc. Record Link` entries for posting history. + +- Copies attachments from the E-Document to the purchase header, sets `E-Document Link`, stores document totals, and validates totals through a try function. + +Manual dimensions are preserved because the helper combines the purchase line's default dimension set with the draft line's `[BC] Dimension Set ID`, then validates both shortcut dimension codes. Additional fields are also applied through FieldRef validation, and failures are wrapped with field context by `E-Doc. Import Error Context`. + +### Purchase credit memo finalization + +`E-Doc. Create Purch. Cr. Memo` uses the same purchase helper for lines, dimensions, attachments, default posting date, currency validation, and record links. It creates a purchase credit memo, validates duplicate external document numbers against posted vendor ledger entries, carries `Posting Description`, and resolves `Applies-to Doc. No.` directly or through `Applies-to Ext. Invoice No.` when possible. + +### Sales order finalization + +`E-Doc. Create Sales Order` creates a sales order from `E-Document Sales Header` and `E-Document Sales Line`. It requires every sales draft line to have type and number, prevents duplicate sales orders by customer and external document number, sets document date and requested delivery date when extracted, copies customer reference and note, applies currency and invoice discount, creates sales lines with dimensions and item references, then sets `E-Document Link` and moves attachments to the sales order. ### Undo finish -`RevertDraftActions()` finds the purchase invoice via the `E-Document Link` GUID, transfers PO matches back to the E-Document, moves attachments back, clears the link, and clears `Document Record ID`. The purchase invoice itself is not deleted -- it must be handled separately. +Undo Finish calls the same `IEDocumentFinishDraft` implementation used to finish. Purchase invoices transfer PO matches back before attachments are moved back and the purchase header link is cleared. Purchase credit memos and sales orders move attachments back and clear the link. The helper does not delete the BC document; it detaches it so the E-Document can be reprocessed or linked again. ## The learning loop +*Updated: 2026-07-29 -- Historical matching is again part of Prepare Draft and history still graduates record links on posting.* + When a purchase invoice created by this pipeline is posted, event subscribers on `Purch.-Post` fire: -- `OnAfterPurchInvLineInsert` writes to `E-Doc. Purchase Line History` -- recording the vendor, product code, description, and the posted invoice line's SystemId. The link is found by traversing `E-Doc. Record Link` from the purchase line back to the draft line. -- `OnAfterPostPurchaseDoc` writes to `E-Doc. Vendor Assign. History` -- recording the vendor identifiers from the original E-Document draft header and the posted invoice header's SystemId. +- `OnAfterPurchInvLineInsert` writes to `E-Doc. Purchase Line History`, recording the vendor, product code, description, and the posted invoice line's SystemId. If allocation accounts replaced the original line, the code falls back through the allocation purchase line SystemId. + +- `OnAfterPostPurchaseDoc` writes to `E-Doc. Vendor Assign. History`, recording the vendor identifiers from the original draft header and the posted invoice header's SystemId. -Both event subscribers then delete the `E-Doc. Record Link` entries since the link has been "graduated" to permanent history. This means the history tables grow monotonically and future imports get progressively better at vendor and line resolution. +Both event subscribers delete the matching `E-Doc. Record Link` entries after history is created. Those links are temporary bridges from draft records to BC records; history is the permanent learning data used by future imports. diff --git a/src/Apps/W1/EDocument/App/src/Processing/Import/docs/data-model.md b/src/Apps/W1/EDocument/App/src/Processing/Import/docs/data-model.md index 56c3bea146c..95fe38991ea 100644 --- a/src/Apps/W1/EDocument/App/src/Processing/Import/docs/data-model.md +++ b/src/Apps/W1/EDocument/App/src/Processing/Import/docs/data-model.md @@ -1,8 +1,10 @@ # Import pipeline data model -## Purchase staging tables +## Draft staging tables -The staging tables hold the intermediate representation of an imported document between the Read and Finish stages. They use **dual nomenclature**: fields 2-100 store raw external data exactly as extracted from the source document, while fields 101-200 store validated BC entity references populated during Prepare Draft. +*Updated: 2026-07-29 -- Staging now covers purchase invoices, purchase credit memos, and inbound sales orders.* + +The staging tables hold the intermediate representation of an imported document between the Read and Finish stages. They use **dual nomenclature**: fields 2-100 store raw external data exactly as extracted from the source document, while fields 101-200 store validated BC entity references populated during Prepare Draft or by the user. ```mermaid erDiagram @@ -10,28 +12,61 @@ erDiagram E_DOCUMENT_PURCHASE_HEADER ||--o{ E_DOCUMENT_PURCHASE_LINE : "E-Document Entry No" E_DOCUMENT_PURCHASE_HEADER }o--o| VENDOR : "[BC] Vendor No." E_DOCUMENT_PURCHASE_HEADER }o--o| PURCHASE_HEADER : "[BC] Purchase Order No." + E_DOCUMENT ||--o| E_DOCUMENT_SALES_HEADER : "Entry No" + E_DOCUMENT_SALES_HEADER ||--o{ E_DOCUMENT_SALES_LINE : "E-Document Entry No" + E_DOCUMENT_SALES_HEADER }o--o| CUSTOMER : "[BC] Customer No." + E_DOCUMENT_PURCHASE_LINE }o--o| VAT_PRODUCT_POSTING_GROUP : "[BC] VAT Prod. Posting Group" ``` -**E-Document Purchase Header** (`Purchase/EDocumentPurchaseHeader.Table.al`, table 6100) is keyed on `E-Document Entry No.` (one-to-one with E-Document). External fields include `Vendor Company Name`, `Vendor VAT Id`, `Vendor GLN`, `Purchase Order No.`, `Sales Invoice No.`, address blocks, and monetary totals. BC fields are `[BC] Vendor No.` and `[BC] Purchase Order No.`. +**E-Document Purchase Header** (`Purchase/EDocumentPurchaseHeader.Table.al`, table 6100) is keyed on `E-Document Entry No.`. It represents both purchase invoices and purchase credit memos. External fields include supplier and customer identity, purchase order number, supplier invoice number, document date, due date, posting description, applies-to references, address blocks, currency, and monetary totals. BC fields are `[BC] Vendor No.` and `[BC] Purchase Order No.`. The table also computes the usable VAT amount difference from setup and staged line VAT amounts. + +**E-Document Purchase Line** (`Purchase/EDocumentPurchaseLine.Table.al`, table 6101) is keyed on `(E-Document Entry No., Line No.)`. External fields include product code, description, quantity, UOM, unit price, subtotal, discount, VAT rate, and currency. BC fields include purchase line type and number, UOM, deferral code, dimensions, item reference, variant, and `[BC] VAT Prod. Posting Group`. The `E-Doc. Purch. Line History Id` metadata field links to the historical match that populated the BC fields, if any. + +**E-Document Sales Header** (`Sales/EDocumentSalesHeader.Table.al`, table 6153) is keyed on `E-Document Entry No.`. It stages inbound PEPPOL orders. External fields describe buyer, seller, originator, accounting customer, buyer order number, seller sales order number, document date, requested delivery date, note, customer reference, order type code, currency, and totals. The BC field is `[BC] Customer No.`. + +**E-Document Sales Line** (`Sales/EDocumentSalesLine.Table.al`, table 6154) is keyed on `(E-Document Entry No., Line No.)`. It stages order lines with buyer, seller, and standard item identifiers plus quantities, UOM, unit price, line extension amount, discount, VAT rate, currency, requested delivery date, and validated BC sales line fields. + +The purchase draft page and sales draft page let users adjust the `[BC]` fields before finishing. On purchase lines, changing a type or number that has PO matches confirms removal of those matches before proceeding. + +## Pipeline selectors + +*Updated: 2026-07-29 -- Added current enum values and service fallback behavior.* + +```mermaid +erDiagram + E_DOCUMENT }o--|| STRUCTURE_RECEIVED_E_DOC : "Structure Data Impl." + E_DOCUMENT }o--|| E_DOC_READ_INTO_DRAFT : "Read into Draft Impl." + E_DOCUMENT }o--|| E_DOC_PROCESS_DRAFT : "Process Draft Impl." + E_DOCUMENT_SERVICE }o--|| E_DOC_READ_INTO_DRAFT : "Draft Format fallback" + E_DOC_READ_INTO_DRAFT ||--o{ E_DOC_PROCESS_DRAFT : "reader returns" +``` -**E-Document Purchase Line** (`Purchase/EDocumentPurchaseLine.Table.al`, table 6101) is keyed on `(E-Document Entry No., Line No.)`. External fields include `Product Code`, `Description`, `Quantity`, `Unit Price`, `Unit of Measure`, and `Currency Code`. BC fields include `[BC] Purchase Line Type`, `[BC] Purchase Type No.`, `[BC] Unit of Measure`, `[BC] Deferral Code`, `[BC] Item Reference No.`, `[BC] Variant Code`, `[BC] Dimension Set ID`, and shortcut dimension codes. The `E-Doc. Purch. Line History Id` metadata field links to the historical match that populated the BC fields, if any. +`E-Document` stores the chosen structuring, reader, and process-draft enum values. `ImportEDocumentProcess` fills missing values as the pipeline advances. The service-level `Read into Draft Impl.` field is the fallback reader for V2 services when neither the receiver nor the structuring step has chosen one. -The staging tables are writable by the user through the `E-Document Purchase Draft` page. When the user changes a `[BC]` field on a line that has PO matches, an `OnValidate` trigger confirms removal of those matches before proceeding. +`E-Doc. Read into Draft` now includes `Blank Draft`, `ADI`, `PEPPOL`, `MLLM`, and `Data Exchange Purchase` in the core app. Country and format apps can extend it; XRechnung and OIOUBL do so by adding enum values with `IStructuredFormatReader` implementations. `E-Doc. Process Draft` now routes to `Purchase Invoice`, `Purchase Credit Memo`, and `Sales Order`; `Purchase Document` is pending obsolete. ## Header and line mappings +*Updated: 2026-07-29 -- Flagged mapping tables for review because current draft pages mostly work directly with staging records.* + ```mermaid erDiagram E_DOCUMENT ||--o| E_DOCUMENT_HEADER_MAPPING : "E-Document Entry No." E_DOCUMENT ||--o{ E_DOCUMENT_LINE_MAPPING : "E-Document Entry No., Line No." ``` -**E-Document Header Mapping** (table 6102) stores validated BC overrides for the header -- `Vendor No.` and `Purchase Order No.` -- applied during Finish Draft. Deleted when Prepare Draft is undone. +**E-Document Header Mapping** (table 6102) stores validated BC overrides for the purchase header -- `Vendor No.` and `Purchase Order No.` -- applied during Finish Draft. Deleted when Prepare Draft is undone. + +**E-Document Line Mapping** (table 6105) stores validated BC overrides per purchase line -- purchase line type and number, UOM, deferral code, dimensions, item reference, variant code, and a history ID. These are the confirmed values that override what the provider chain suggested. -**E-Document Line Mapping** (table 6105) stores validated BC overrides per line -- purchase line type/number, UOM, deferral code, dimensions, item reference, variant code, and a history ID. These are the "confirmed" values that override what the provider chain suggested. +Both mapping tables are `Access = Internal` and are effectively legacy. Outside their own definitions, the only remaining reference in the app is the data-classification registration in `EDocumentSubscribers`, so nothing in the V2 draft pipeline reads or writes them. They are retained for compatibility with existing data rather than being part of the current draft flow. Do not build new functionality on them. + +*Updated: 2026-07-29 -- confirmed the mapping tables are retained for compatibility only* ## Purchase order matching +*Updated: 2026-07-29 -- Clarified that PO matching remains current while the old PO Matching Copilot is obsolete.* + ```mermaid erDiagram E_DOCUMENT_PURCHASE_LINE ||--o{ E_DOC_PURCHASE_LINE_PO_MATCH : "E-Doc. Purchase Line SystemId" @@ -39,12 +74,14 @@ erDiagram PURCH_RCPT_LINE ||--o{ E_DOC_PURCHASE_LINE_PO_MATCH : "Receipt Line SystemId" ``` -**E-Doc. Purchase Line PO Match** (`Purchase/PurchaseOrderMatching/EDocPurchaseLinePOMatch.Table.al`, table 6114) is the N:M junction table linking e-document draft lines to purchase order lines and optionally to receipt lines. The composite key is `(E-Doc. Purchase Line SystemId, Purchase Line SystemId, Receipt Line SystemId)` -- all three are Guid fields using SystemId references. +**E-Doc. Purchase Line PO Match** (`Purchase/PurchaseOrderMatching/EDocPurchaseLinePOMatch.Table.al`, table 6114) is the N:M junction table linking e-document draft lines to purchase order lines and optionally receipt lines. The composite key is `(E-Doc. Purchase Line SystemId, Purchase Line SystemId, Receipt Line SystemId)` -- all three are Guid fields using SystemId references. -`EDocPOMatching.Codeunit.al` manages this table: loading available PO lines for matching (filtering by vendor and optionally by order number), verifying match validity, suggesting receipts for matched lines, and transferring matches between E-Document and Purchase Invoice during Finish Draft / Undo Finish. +`EDocPOMatching.Codeunit.al` manages this table: loading available PO lines, verifying match validity, suggesting receipts, calculating warnings, and transferring matches between E-Document and Purchase Invoice during Finish Draft and Undo Finish. The older Purchase Order Matching Copilot objects outside this folder are obsolete; current import-time AI matching is line-account and history assistance, not PO line matching. ## Historical learning +*Updated: 2026-07-29 -- Clarified newest-first history matching and record-link graduation on posting.* + ```mermaid erDiagram E_DOC_PURCHASE_LINE_HISTORY }o--|| PURCH_INV_LINE : "Purch. Inv. Line SystemId" @@ -52,15 +89,19 @@ erDiagram E_DOC_PURCHASE_LINE_HISTORY }o--|| VENDOR : "Vendor No." ``` -**E-Doc. Purchase Line History** (`Purchase/History/EDocPurchaseLineHistory.Table.al`, table 6140) records what BC entities were assigned to past draft lines. Key fields: `Vendor No.`, `Product Code`, `Description`, and `Purch. Inv. Line SystemId`. Four secondary keys enable flexible lookup: by `(Vendor No., Product Code, Description)`, by `(Product Code, Description)`, by `(Vendor No., Product Code)`, and by `(Vendor No., Description)`. The history search in `EDocPurchaseHistMapping.FindRelatedPurchaseLineInHistory()` tries product code first, then exact description match, then prefix match, then substring match -- all scoped to the same vendor and sorted most-recent-first. +**E-Doc. Purchase Line History** (`Purchase/History/EDocPurchaseLineHistory.Table.al`, table 6140) records what BC entities were assigned to past draft lines. Key fields are `Vendor No.`, `Product Code`, `Description`, and `Purch. Inv. Line SystemId`. The history search in `EDocPurchaseHistMapping.FindRelatedPurchaseLineInHistory()` tries product code first, then exact description match, then prefix match, then substring match -- all scoped to the same vendor and sorted most-recent-first. -**E-Doc. Vendor Assign. History** (`Purchase/History/EDocVendorAssignHistory.Table.al`, table 6108) records past vendor identifier-to-vendor-number mappings. Key fields: `Vendor Company Name`, `Vendor Address`, `Vendor VAT Id`, `Vendor GLN`, and `Purch. Inv. Header SystemId`. The `Vendor No From Purch. Header` FlowField resolves the vendor number from the posted invoice. When the same identifier combination appears again, the existing record is updated rather than duplicated. +**E-Doc. Vendor Assign. History** (`Purchase/History/EDocVendorAssignHistory.Table.al`, table 6108) records past vendor identifier-to-vendor-number mappings. Key fields are vendor company name, address, VAT ID, GLN, and posted purchase invoice header SystemId. When the same identifier combination appears again, the existing record is updated rather than duplicated. -Both tables are populated by `EDocPurchaseHistMapping.Codeunit.al` via event subscribers on `Purch.-Post`. The `E-Doc. Record Link` entries that connect draft records to BC records are consumed during this process and then deleted -- they serve as temporary bridges that are "graduated" to permanent history on posting. +Both tables are populated by `EDocPurchaseHistMapping.Codeunit.al` via event subscribers on `Purch.-Post`. `E-Doc. Record Link` entries connect draft records to the purchase header and lines during Finish Draft; posting consumes those links to create history and then deletes them. ## Record links -**E-Doc. Record Link** (`../EDocRecordLink.Table.al`, table 6141) provides SystemId-based links between draft staging records and BC records created during Finish Draft. Each entry stores source table/SystemId and target table/SystemId. Two links are created per line (draft line --> purchase line) plus one per header (draft header --> purchase header). These links serve two purposes: they allow navigation from draft records to their BC counterparts, and they are the mechanism by which the posting event subscribers find the original draft data to populate the history tables. +*Updated: 2026-07-29 -- Record links now cover purchase and sales line helpers, but purchase history still consumes only purchase links.* + +**E-Doc. Record Link** (`../EDocRecordLink.Table.al`, table 6141) provides SystemId-based links between draft staging records and BC records created during Finish Draft. Purchase helpers insert links from purchase staging header and lines to purchase header and lines. Sales helpers insert sales line links and use `E-Document Link` on the sales header for the document-level relationship. + +For purchase documents, the links serve two purposes: navigation from draft records to BC counterparts, and the bridge by which posting subscribers find original draft data to populate history. After posting, the relevant links are deleted because the relationship has graduated to permanent history. ## Additional fields @@ -70,10 +111,14 @@ erDiagram E_DOCUMENT_PURCHASE_LINE ||--o{ E_DOCUMENT_LINE_FIELD : "E-Document Entry No., Line No." ``` -**ED Purchase Line Field Setup** (`AdditionalFields/EDPurchaseLineFieldSetup.Table.al`, table 6112) defines which `Purch. Inv. Line` fields should be tracked as additional columns on draft lines, scoped per E-Document Service. Fields that already exist on the staging tables (Type, No., UOM, etc.) are automatically omitted. +**ED Purchase Line Field Setup** (`AdditionalFields/EDPurchaseLineFieldSetup.Table.al`, table 6112) defines which `Purch. Inv. Line` fields should be tracked as additional columns on draft lines, scoped per E-Document Service. Fields that already exist on the staging tables, such as type, number, UOM, dimensions, and amounts, are automatically omitted. -**E-Document Line - Field** (`AdditionalFields/EDocumentLineField.Table.al`, table 6110) is a polymorphic value store keyed on `(E-Document Entry No., Line No., Field No.)`. It has six typed value columns: `Text Value`, `Decimal Value`, `Date Value`, `Boolean Value`, `Code Value`, and `Integer Value`. The `Get()` procedure implements a three-tier resolution: if a physical record exists, it returns `Customized`. Otherwise, it looks up the `E-Doc. Purch. Line History Id` on the draft line to find the posted invoice line and reads the value from history, returning `Historic`. If neither exists, it returns `Default` with blank values. During Finish Draft, `ApplyAdditionalFieldsFromHistoryToPurchaseLine()` validates these values onto the actual purchase line via FieldRef. +**E-Document Line - Field** (`AdditionalFields/EDocumentLineField.Table.al`, table 6110) is a polymorphic value store keyed on `(E-Document Entry No., Line No., Field No.)`. It has six typed value columns. The `Get()` procedure resolves values from customized draft data, then from posted invoice history, then from defaults. During Finish Draft, `ApplyAdditionalFieldsFromHistoryToPurchaseLine()` validates these values onto the actual purchase line via FieldRef. `E-Doc. Import Error Context` adds the field name, field ID, and value to any validation error. + +*Updated: 2026-07-29 -- Added validation context behavior for additional field failures.* ## Import parameters -**E-Doc. Import Parameters** (table 6106) is a **temporary** table that configures a single pipeline execution. Key fields: `Processing Customizations` (which provider enum to use), `Step to Run` / `Desired E-Document Status` (forward to a step or target a status), `Existing Doc. RecordId` (link to existing document instead of creating), and V1 compatibility flags. Being temporary, it exists only in memory during the import call. +*Updated: 2026-07-29 -- Added existing document linking and desired status behavior.* + +**E-Doc. Import Parameters** (table 6106) is a **temporary** table that configures a single pipeline execution. It controls processing customizations, step selection, desired status, existing document linking, and V1 compatibility behavior. Being temporary, it exists only in memory during the import call. diff --git a/src/Apps/W1/EDocument/App/src/Processing/Import/docs/extensibility.md b/src/Apps/W1/EDocument/App/src/Processing/Import/docs/extensibility.md index 92dbc635aa5..8c462b61eb2 100644 --- a/src/Apps/W1/EDocument/App/src/Processing/Import/docs/extensibility.md +++ b/src/Apps/W1/EDocument/App/src/Processing/Import/docs/extensibility.md @@ -1,129 +1,162 @@ # Import pipeline extensibility -The import pipeline is built on extensible enums backed by interfaces. Every stage dispatches through an interface, so extensions replace behavior by adding enum values with new implementations. The interfaces live in `src/Processing/Interfaces/`. +The import pipeline is built on extensible enums backed by interfaces. Every stage dispatches through an interface, so extensions replace behavior by adding enum values with implementations. The import-specific interfaces are selected by `ImportEDocumentProcess.Codeunit.al`; the contracts live in `src/Processing/Interfaces/` and, for sales-specific providers, under `Import/Sales/`. -## Add a new document format parser +*Updated: 2026-07-29 -- Added the current reader selection chain, Data Exchange bridge, purchase credit memo, and sales order extension points.* -To support a new file format (e.g. CSV invoices), implement two things: +## Add a new inbound format -**File format detection.** Extend the `"E-Doc. File Format"` enum with a new value whose `IEDocFileFormat` implementation returns the file extension, a content preview method, and a preferred structure implementation: +*Updated: 2026-07-29 -- Documented service Draft Format fallback and the process draft value returned by each reader.* -``` -interface IEDocFileFormat - procedure FileExtension(): Text - procedure PreviewContent(FileName: Text; TempBlob: Codeunit "Temp Blob") - procedure PreferredStructureDataImplementation(): Enum "Structure Received E-Doc." -``` - -The built-in implementations are in `FileFormat/`: XML returns `"Already Structured"`, PDF returns `"ADI"`, JSON returns `"Already Structured"`. Your format should point to whichever structuring implementation makes sense. +To support a new structured inbound format, the central extension point is `E-Doc. Read into Draft`. The user or service integration selects the implementation through `E-Document."Read into Draft Impl."`; if the document does not set one, V2 import falls back to `E-Document Service."Read into Draft Impl."`, shown as **Draft Format** on the service card. -**Structured format reader.** Extend the `"E-Doc. Read into Draft"` enum with a new value whose `IStructuredFormatReader` implementation parses the structured blob into `E-Document Purchase Header` / `E-Document Purchase Line` records: +Extend the enum with a value whose implementation parses the structured blob into staging records and returns the next process draft route: -``` +```al interface IStructuredFormatReader procedure ReadIntoDraft(EDocument: Record "E-Document"; TempBlob: Codeunit "Temp Blob"): Enum "E-Doc. Process Draft" procedure View(EDocument: Record "E-Document"; TempBlob: Codeunit "Temp Blob") ``` -`ReadIntoDraft` must insert the staging records and return the `"E-Doc. Process Draft"` enum value that determines which Prepare Draft implementation runs. See `EDocumentPEPPOLHandler.Codeunit.al` for a complete XML example and `EDocumentADIHandler.Codeunit.al` for a JSON example. +A purchase invoice reader inserts `E-Document Purchase Header` and `E-Document Purchase Line` records and returns `Purchase Invoice`. A purchase credit memo reader uses the same purchase staging tables and returns `Purchase Credit Memo`. A sales order reader inserts `E-Document Sales Header` and `E-Document Sales Line` records and returns `Sales Order`. + +Built-in examples: + +- PEPPOL reads invoices, credit notes, orders, and order responses in `EDocumentPEPPOLHandler.Codeunit.al`. -## Add a new structuring mechanism +- Data Exchange Purchase runs configured Data Exchange definitions and bridges intermediate data into purchase staging. -If you have a non-trivial conversion step (e.g. a custom OCR service for scanned documents), extend the `"Structure Received E-Doc."` enum with a new value whose `IStructureReceivedEDocument` implementation converts the raw blob: +- MLLM and ADI read JSON output from their structuring stage into purchase staging. +Country apps follow the same pattern. XRechnung and OIOUBL extend `E-Doc. Read into Draft`, implement `IStructuredFormatReader`, populate purchase staging, and return either `Purchase Invoice` or `Purchase Credit Memo`. + +## Add a new file type or structuring mechanism + +*Updated: 2026-07-29 -- Updated PDF default to MLLM and clarified reader override behavior from structured data.* + +Use these interfaces when your source blob is not already in the shape your reader consumes. + +**File format detection.** Extend the `E-Doc. File Format` enum with an `IEDocFileFormat` implementation: + +```al +interface IEDocFileFormat + procedure FileExtension(): Text + procedure PreviewContent(FileName: Text; TempBlob: Codeunit "Temp Blob") + procedure PreferredStructureDataImplementation(): Enum "Structure Received E-Doc." ``` + +XML and JSON return `Already Structured`. PDF now returns `MLLM`; ADI remains available but is no longer the default PDF preference in the core PDF file format. + +**Structuring.** Extend the `Structure Received E-Doc.` enum with an `IStructureReceivedEDocument` implementation: + +```al interface IStructureReceivedEDocument procedure StructureReceivedEDocument(EDocumentDataStorage: Record "E-Doc. Data Storage"): Interface IStructuredDataType ``` -Your implementation receives the raw blob and must return an `IStructuredDataType` -- a stateful object that holds the file format, the text content, and optionally specifies which `IStructuredFormatReader` to use downstream: +Your implementation returns an `IStructuredDataType` object: -``` +```al interface IStructuredDataType procedure GetFileFormat(): Enum "E-Doc. File Format" procedure GetContent(): Text procedure GetReadIntoDraftImpl(): Enum "E-Doc. Read into Draft" ``` -The ADI handler (`EDocumentADIHandler.Codeunit.al`) implements all three interfaces (`IStructureReceivedEDocument`, `IStructuredDataType`, `IStructuredFormatReader`) in a single codeunit because it owns the entire chain from PDF to staging tables. +`GetReadIntoDraftImpl()` is important. If it returns a value other than `Unspecified`, the structure step writes it to the E-Document and the read step uses it, even if the service has a different Draft Format. Use this when the conversion result implies exactly one downstream reader, such as MLLM JSON or ADI JSON. -## Customize vendor resolution +## Use Data Exchange definitions in V2 -Extend the `"E-Doc. Proc. Customizations"` enum. This is a multi-interface enum that bundles five provider interfaces with defaults from `EDocProviders.Codeunit.al`. Your new enum value provides custom implementations for any or all of: +*Updated: 2026-07-29 -- Added bridge pattern guidance.* -``` -interface IVendorProvider - procedure GetVendor(EDocument: Record "E-Document"): Record Vendor -``` +If your format is already modeled as Data Exchange Definitions, use or mimic the `Data Exchange Purchase` reader rather than duplicating the final document creation logic. The bridge pattern is: -The default implementation (`EDocProviders.GetVendor`) tries VAT ID + GLN lookup, then Service Participant matching, then name + address search. Replace it to add your own vendor matching logic -- for example, looking up a custom identifier from a localized field. +1. Configure one or more import Data Exchange Definitions on the E-Document Service. +2. Let `EDocDataExchPurchHandler` choose a definition by XML root namespace. +3. Run `DataExchDef.ProcessDataExchange()` so the standard Data Exchange framework fills `Intermediate Data Import`. +4. Map intermediate rows into `E-Document Purchase Header` and `E-Document Purchase Line`. +5. Return `Purchase Invoice` or `Purchase Credit Memo` so the normal V2 Prepare and Finish stages handle vendor resolution, matching, validation, links, attachments, and history. -## Customize line resolution +This bridge exists to preserve older Data Exchange mapping investments while still gaining the V2 draft review and finalization pipeline. It should not bypass staging tables or create the purchase document directly. -The same `"E-Doc. Proc. Customizations"` enum also controls line-level resolution: +## Customize purchase resolution -``` -interface IPurchaseLineProvider - procedure GetPurchaseLine(var EDocumentPurchaseLine: Record "E-Document Purchase Line") -``` +*Updated: 2026-07-29 -- Added current provider behavior and VAT product posting group note.* -The default implementation tries Item Reference by vendor + product code, then Text-to-Account Mapping by description. Your implementation receives the draft line with external data populated and should set `[BC] Purchase Line Type`, `[BC] Purchase Type No.`, and related fields. +Extend the `E-Doc. Proc. Customizations` enum. This multi-interface enum bundles the provider and finalizer interfaces used during Prepare Draft and Finish Draft. -Note: `IPurchaseLineAccountProvider` (same signature pattern but with explicit out-parameters for account type and number) is **obsolete as of v27** -- replaced by `IPurchaseLineProvider`. +For purchase drafts, the default value provides: + +```al +interface IVendorProvider + procedure GetVendor(EDocument: Record "E-Document"): Record Vendor + +interface IPurchaseOrderProvider + procedure GetPurchaseOrder(EDocumentPurchaseHeader: Record "E-Document Purchase Header"): Record "Purchase Header" -``` interface IUnitOfMeasureProvider procedure GetUnitOfMeasure(EDocument: Record "E-Document"; EDocumentLineId: Integer; ExternalUnitOfMeasure: Text): Record "Unit of Measure" + +interface IPurchaseLineProvider + procedure GetPurchaseLine(var EDocumentPurchaseLine: Record "E-Document Purchase Line") ``` -The default tries UOM Code, then International Standard Code, then Description. Override this if your vendors use non-standard UOM identifiers. +The default vendor provider tries VAT ID and GLN, service participant, then name and address. Purchase history can still fill a missing vendor after the provider runs. The default line provider tries item references and then Text-to-Account Mapping. VAT product posting group resolution is not an interface; it is controlled by purchase setup and uses the vendor VAT business posting group plus the extracted line VAT rate. -## Customize purchase order matching +`IPurchaseLineAccountProvider` is obsolete as of v27 and should not be used for new work. -``` -interface IPurchaseOrderProvider - procedure GetPurchaseOrder(EDocumentPurchaseHeader: Record "E-Document Purchase Header"): Record "Purchase Header" -``` +## Customize purchase finalization -The default looks up `"Purchase Order No."` from the draft header. Override to implement custom PO matching logic -- for example, matching by a combination of vendor and date range. +*Updated: 2026-07-29 -- Added credit memo creation and clarified when to override create versus finish contracts.* -## Customize invoice creation +Purchase invoice creation is controlled by `IEDocumentCreatePurchaseInvoice`, and purchase credit memo creation by `IEDocumentCreatePurchaseCreditMemo`. Both are bundled into `E-Doc. Proc. Customizations`. -``` +```al interface IEDocumentCreatePurchaseInvoice procedure CreatePurchaseInvoice(EDocument: Record "E-Document"): Record "Purchase Header" -``` -The `"E-Doc. Create Purchase Invoice"` enum is extensible and defaults to `EDocCreatePurchaseInvoice.Codeunit.al`. Override to change how purchase invoices are created -- for example, to set custom fields, apply different discount logic, or create credit memos instead. +interface IEDocumentCreatePurchaseCreditMemo + procedure CreatePurchaseCreditMemo(EDocument: Record "E-Document"): Record "Purchase Header" +``` -The Finish Draft step also uses: +The Finish Draft dispatcher itself uses `IEDocumentFinishDraft`, selected from `E-Document Type`: -``` +```al interface IEDocumentFinishDraft procedure ApplyDraftToBC(EDocument: Record "E-Document"; EDocImportParameters: Record "E-Doc. Import Parameters"): RecordId procedure RevertDraftActions(EDocument: Record "E-Document") ``` -This is controlled by the `"E-Document Type"` enum set during Prepare Draft. Currently only `"Purchase Invoice"` is implemented, routing to `EDocCreatePurchaseInvoice.Codeunit.al`. +Override the create interfaces when you only need to alter how the BC purchase document is built. Add or replace an `IEDocumentFinishDraft` implementation only when you are introducing a new document type or a different finish or undo contract. -## Register AI tools for line matching +## Customize sales order import -The Copilot matching subsystem uses `IEDocAISystem` to register AI-powered processing tools: +*Updated: 2026-07-29 -- Added inbound sales order providers and finalizer.* +Sales order import uses the same pipeline but different staging tables and providers. Extend `E-Doc. Proc. Customizations` to provide custom implementations for: + +```al +interface ICustomerProvider + procedure GetCustomer(EDocument: Record "E-Document"): Record Customer + +interface ISalesLineProvider + procedure GetSalesLine(var EDocumentSalesLine: Record "E-Document Sales Line") + +interface IEDocumentCreateSalesOrder + procedure CreateSalesOrder(EDocument: Record "E-Document"): Record "Sales Header" ``` -interface IEDocAISystem - procedure GetSystemPrompt(UserLanguage: Text): SecretText - procedure GetTools(): List of [Interface "AOAI Function"] - procedure GetFeatureName(): Text -``` -Extensions can register new AI systems by extending the `"E-Doc. AI System"` enum. Each system provides a system prompt, a set of AOAI Function tool implementations (for function-calling), and a feature name for telemetry. The built-in systems cover historical matching, GL account matching, and deferral matching. Add your own to support custom matching scenarios -- for example, matching lines to projects or jobs. +The default customer provider tries buyer GLN, service participant identifiers, buyer VAT ID, and name plus address. The default sales line provider tries seller item ID, GTIN or bar-code item reference, then buyer item reference for the resolved customer. Override these when partner identifiers or line matching rules differ from the built-in assumptions. + +The sales route itself is driven by `IProcessStructuredDataSales`, which extends `IProcessStructuredData` with the sales-specific draft operations. `Prepare Sales E-Doc. Draft` (codeunit 6429) is the built-in implementation, so a new sales reader implements the extended interface rather than the purchase one. ## Add a new draft preparation strategy -Extend the `"E-Doc. Process Draft"` enum to add a new `IProcessStructuredData` implementation: +*Updated: 2026-07-29 -- Updated current routes and obsoleted `Purchase Document` guidance.* -``` +Extend `E-Doc. Process Draft` to add a new `IProcessStructuredData` implementation: + +```al interface IProcessStructuredData procedure PrepareDraft(EDocument: Record "E-Document"; EDocImportParameters: Record "E-Doc. Import Parameters"): Enum "E-Document Type" procedure GetVendor(EDocument: Record "E-Document"; Customizations: Enum "E-Doc. Proc. Customizations"): Record Vendor @@ -131,18 +164,44 @@ interface IProcessStructuredData procedure CleanUpDraft(EDocument: Record "E-Document") ``` -Currently only `"Purchase Document"` exists. A future value could handle general journal lines or service invoices. Your `IStructuredFormatReader.ReadIntoDraft()` returns the appropriate enum value to route to your preparation logic. +Current core routes are `Purchase Invoice`, `Purchase Credit Memo`, and `Sales Order`. The obsolete `Purchase Document` value remains only behind a cleanup symbol and should not be used for new readers. + +A reader controls routing by returning the process draft enum value from `ReadIntoDraft()`. + +## Register AI tools for line matching + +*Updated: 2026-07-29 -- Added prompt-resource guidance and obsoleted PO Matching Copilot note.* + +The import-time AI matching subsystem uses `IEDocAISystem` and AOAI Function implementations. The built-in systems include historical matching, similar descriptions, GL account matching, and deferral matching. + +```al +interface IEDocAISystem + procedure GetSystemPrompt(UserLanguage: Text): SecretText + procedure GetTools(): List of [Interface "AOAI Function"] + procedure GetFeatureName(): Text +``` + +Prompts live in `App/.resources/Prompts/`. The MLLM extraction prompt now includes a content and format monitoring section that tells the model to extract only visible values, distinguish buyer from vendor using the prefilled customer party, and emit decimals in XML format. The GL account matching prompt now tells the model to process every plausible line and call one tool per line. + +The older Purchase Order Matching Copilot objects are obsolete. New AI work should plug into the import-time tool processor or a new `IEDocAISystem`, not the old PO matching Copilot pages and buffers. ## Extension patterns summary +*Updated: 2026-07-29 -- Updated current enum and interface matrix.* + | Goal | Extend this enum | Implement this interface | -|------|-----------------|------------------------| -| New file type detection | `"E-Doc. File Format"` | `IEDocFileFormat` | -| New structuring method (OCR, etc.) | `"Structure Received E-Doc."` | `IStructureReceivedEDocument` + `IStructuredDataType` | -| New format reader | `"E-Doc. Read into Draft"` | `IStructuredFormatReader` | -| New draft preparation | `"E-Doc. Process Draft"` | `IProcessStructuredData` | -| Custom providers (vendor, item, UOM, PO, invoice) | `"E-Doc. Proc. Customizations"` | Any combination of 5 provider interfaces | -| New AI matching tool | `"E-Doc. AI System"` | `IEDocAISystem` | -| Custom invoice creation | `"E-Doc. Create Purchase Invoice"` | `IEDocumentCreatePurchaseInvoice` | - -Two interfaces in `src/Processing/Interfaces/` are not part of the import pipeline: `IExportEligibilityEvaluator` (outbound filtering) and `IBlobToStructuredDataConverter` / `IBlobType` (obsolete as of v26, replaced by `IEDocFileFormat` and `IStructureReceivedEDocument`). +|------|-----------------|--------------------------| +| New file type detection | `E-Doc. File Format` | `IEDocFileFormat` | +| New structuring method | `Structure Received E-Doc.` | `IStructureReceivedEDocument` and `IStructuredDataType` | +| New structured reader or inbound format | `E-Doc. Read into Draft` | `IStructuredFormatReader` | +| Reuse Data Exchange definitions in V2 | `E-Doc. Read into Draft` or service Draft Format | `IStructuredFormatReader` that bridges intermediate data to staging | +| New draft preparation route | `E-Doc. Process Draft` | `IProcessStructuredData` | +| Custom purchase providers | `E-Doc. Proc. Customizations` | `IVendorProvider`, `IPurchaseOrderProvider`, `IPurchaseLineProvider`, `IUnitOfMeasureProvider` | +| Custom sales providers | `E-Doc. Proc. Customizations` | `ICustomerProvider`, `ISalesLineProvider` | +| Custom purchase invoice creation | `E-Doc. Proc. Customizations` | `IEDocumentCreatePurchaseInvoice` | +| Custom purchase credit memo creation | `E-Doc. Proc. Customizations` | `IEDocumentCreatePurchaseCreditMemo` | +| Custom sales order creation | `E-Doc. Proc. Customizations` | `IEDocumentCreateSalesOrder` | +| New AI matching tool | AI system implementation | `IEDocAISystem` and `AOAI Function` | +| Custom finish or undo contract | `E-Document Type` | `IEDocumentFinishDraft` | + +Two interfaces in `src/Processing/Interfaces/` are not part of the current import route: `IExportEligibilityEvaluator` is for outbound filtering, and `IBlobToStructuredDataConverter` / `IBlobType` are obsolete as of v26, replaced by `IEDocFileFormat` and `IStructureReceivedEDocument`. diff --git a/src/Apps/W1/EDocument/App/src/Processing/docs/CLAUDE.md b/src/Apps/W1/EDocument/App/src/Processing/docs/CLAUDE.md index 76651f60800..b46f14781b4 100644 --- a/src/Apps/W1/EDocument/App/src/Processing/docs/CLAUDE.md +++ b/src/Apps/W1/EDocument/App/src/Processing/docs/CLAUDE.md @@ -1,29 +1,45 @@ # Processing -The Processing module orchestrates what happens to E-Documents between creation and delivery (outbound) or between receipt and BC document creation (inbound). It owns the export pipeline, event subscribers that hook into BC posting, background job scheduling, order matching for purchase documents, and AI-assisted line matching via Copilot. The import pipeline lives in `Import/` and has its own docs. +The Processing module orchestrates what happens to E-Documents between creation and delivery on the outbound side and between receipt and BC document creation on the general inbound side. It owns the export pipeline, event subscribers that hook into BC posting and release, background job scheduling, order matching for purchase documents, outbound response messages, and AI-assisted draft line matching. The detailed import pipeline lives in `Import/` and has its own docs. ## How it works -**Outbound flow.** `EDocumentSubscribers` listens to `OnAfterPostSalesDoc`, `OnAfterPostPurchaseDoc`, `OnAfterPostServiceDoc`, and similar events. When a document posts and its Document Sending Profile is set to `"Extended E-Document Service Flow"`, the subscriber calls `EDocExport.CreateEDocument()`. This creates an E-Document record, evaluates export eligibility per service via `IExportEligibilityEvaluator`, runs field mapping, invokes the format interface's `Create()` method (via `EDocumentCreate.Codeunit.al`) to produce a TempBlob, and logs the result. Finally, `EDocumentBackgroundJobs.StartEDocumentCreatedFlow()` enqueues a job that triggers the workflow -- which in turn decides whether to send, email, or route for approval. +*Updated: 2026-07-29 -- added purchase order export, message responses, and current AI matching boundaries.* + +**Outbound flow.** `EDocumentSubscribers` listens to posted sales, service, finance charge, reminder, warehouse completion, and transfer shipment events. It also listens to purchase order release. When the relevant customer, vendor, or location resolves to a Document Sending Profile with `"Extended E-Document Service Flow"`, the subscriber calls `EDocExport.CreateEDocument()`. This creates or reuses an E-Document record, evaluates export eligibility per service via `IExportEligibilityEvaluator`, runs field mapping, invokes the format interface's `Create()` method through `EDocumentCreate.Codeunit.al` to produce a TempBlob, and logs the result. Finally, `EDocumentBackgroundJobs.StartEDocumentCreatedFlow()` enqueues a job that triggers the workflow -- which in turn decides whether to send, email, or route for approval. + +**PEPPOL EDI flow.** Purchase orders now enter the same outbound pipeline when a purchase order is released. `EDocExport` understands `"Purchase Order"` as an E-Document type and reads live `Purchase Line` records for export. The core `"PEPPOL BIS 3.0"` format value still dispatches through `EDoc PEPPOL BIS 3.0`, but the PEPPOL XML builders it calls live in the standalone PEPPOL app. For inbound sales orders and order responses, Processing only owns the parent dispatch and message storage; the sales draft details live under `Import/`. **Batch processing.** When a service has `"Use Batch Processing"` enabled, individual documents are not exported immediately. Instead they get status `Created` and wait. If the batch mode is `Recurrent`, a scheduled job (`"E-Doc. Recurrent Batch Send"`) collects all pending-batch documents grouped by document type, exports them as a batch, and sends them together. -**Order matching.** For incoming purchase orders, `EDocLineMatching.Codeunit.al` matches imported e-document lines to existing purchase order lines. Automatic matching filters on UOM, unit cost, and discount, then uses `CalculateStringNearness()` above 80% for description matching, plus Item Reference and Text-to-Account Mapping lookups. The Copilot subfolder adds AI-assisted matching via Azure OpenAI when automatic matching leaves unmatched lines. +**Order matching.** For incoming purchase orders, `EDocLineMatching.Codeunit.al` matches imported e-document lines to existing purchase order lines. Automatic matching filters on UOM, unit cost, and discount, then uses `CalculateStringNearness()` above 80% for description matching, plus Item Reference and Text-to-Account Mapping lookups. The old matching-page Copilot action is now hidden and obsolete; AI-assisted matching has moved to import draft processing through `EDocAIToolProcessor.Codeunit.al` and the tool codeunits in `AI/Tools/`. -**AI tools.** `EDocAIToolProcessor.Codeunit.al` is a generic Copilot orchestrator that configures Azure OpenAI (GPT-4.1), registers AI tools as function calls, and processes responses. The `Tools/` subfolder provides implementations for historical matching, G/L account matching, deferral matching, and similar-description lookups. +**Message responses.** `E-Document Message` records store message payloads related to an existing E-Document, such as PEPPOL Order Responses. `EDocumentSubscribers.OnAfterReleaseSalesDoc()` creates an Accepted response for an inbound sales order when the document format reports a response message type. `EDocumentProcessing.SendOrderRejection()` builds the same kind of message with a Rejected response. Both paths use `IEDocResponseProvider` to ask the format which message applies, then `IEDocMessageBuilder` to build and persist the payload. + +**AI tools.** `EDocAIToolProcessor.Codeunit.al` is a generic Copilot orchestrator that registers the E-Document Matching Assistance capability in SaaS, configures Azure OpenAI with the latest GPT-4.1 chat deployment, registers AI tools as function calls, and processes responses. The `Tools/` subfolder provides implementations for historical matching, G/L account matching, deferral matching, and similar-description lookups. Historical matching now tries direct history before asking the model. ## Things to know -- Export eligibility is pluggable: the `"Export Eligibility Evaluator"` enum on the service record controls which `IExportEligibilityEvaluator` runs. The default implementation allows all documents. Extend the enum to filter by document attributes, customer, or any other criteria. +*Updated: 2026-07-29 -- refreshed gotchas for PEPPOL EDI, error handling, attachments, and obsolete Copilot matching.* + +- Export eligibility is pluggable: the `"Export Eligibility Evaluator"` enum on the service record controls which `IExportEligibilityEvaluator` runs. The default implementation allows all documents. Extend the enum to filter by document attributes, customer, vendor, or any other criteria. - `EDocumentCreate.Codeunit.al` is a thin runner that delegates to the format interface's `Create()` or `CreateBatch()`. It exists solely to be wrapped in `Codeunit.Run()` for error isolation. -- `EDocumentSubscribers` also subscribes to release events (`OnBeforeReleaseSalesDoc`, etc.) and posting-check events to run `CheckEDocument()` before the document is committed, ensuring format-specific validation happens early. +- `EDocumentSubscribers` subscribes to release events (`OnBeforeReleaseSalesDoc`, `OnBeforeReleasePurchaseDoc`, etc.) and posting-check events to run `CheckEDocument()` before the document is committed. Purchase order release also creates or re-exports an outbound E-Document when the profile is electronic. -- Order matching only applies to incoming purchase orders (`"Document Type" = "Purchase Order"`, `Direction = Incoming`, `Status = "Order Linked"`). The matching page lets users match manually, run automatic matching, or invoke Copilot. Accepted matches persist to the `"E-Doc. Order Match"` table and update `"Qty. to Invoice"` on purchase lines. +- Manual creation from a posted document validates the resolved Document Sending Profile before export and errors if the source record cannot be mapped to an E-Document type. That avoids silently doing nothing when setup is missing or the record type is unsupported. -- The Copilot PO matching (`EDocPOCopilotMatching.Codeunit.al`) builds a user prompt from imported line and PO line descriptions, sends it to GPT-4.1 with function-calling tools, and grounds the result by verifying cost/quantity thresholds before surfacing proposals. +- Error messages are part of the document lifecycle. Processing actions clear existing errors before retrying, import step failures log the wrapped error text and mark the service status as `"Imported Document Processing Error"`, and E-Document cleanup clears linked Error Message records along with logs, service statuses, attachments, imported lines, messages, and draft data. + +- Order matching only applies to incoming purchase orders (`"Document Type" = "Purchase Order"`, `Direction = Incoming`, `Status = "Order Linked"`). The matching page lets users match manually or run automatic matching. Accepted matches persist to the `"E-Doc. Order Match"` table and update `"Qty. to Invoice"` on purchase lines. + +- The legacy Copilot PO matching code remains behind `#if not CLEAN29` as obsolete and hidden UI. Do not build new flows on it. New AI matching work should use `EDocAIToolProcessor` and import draft tools instead. - `EDocumentBackgroundJobs` manages three job types: the one-shot "created flow" trigger, the recurring 5-minute `GetResponse` poller, and the recurrent batch send/import jobs with configurable frequency. -- Do not confuse `EDocImport.Codeunit.al` in this folder with the full import pipeline -- it is the entry point that delegates to `Processing/Import/` for V2 import processing. See the Import docs for that pipeline. +- Document attachments saved against an E-Document are marked with `"E-Document Attachment"` and `"E-Document Entry No."`. This lets the attachment factbox recover the E-Document context, supports Digital Voucher attachment scenarios, and lets cleanup remove only attachments that belong to that E-Document. + +- `E-Document Message` is not a BC source document. It is a child payload for lifecycle messages such as an outgoing PEPPOL Order Response, with its XML stored in `E-Doc. Data Storage` and surfaced by the messages factbox. + +- Do not confuse `EDocImport.Codeunit.al` in this folder with the full import pipeline -- it is the entry point that delegates to `Processing/Import/` for V2 import processing. New services default to the V2 import architecture; see the Import docs for stage-by-stage inbound behavior. diff --git a/src/Apps/W1/EDocument/App/src/Processing/docs/business-logic.md b/src/Apps/W1/EDocument/App/src/Processing/docs/business-logic.md index c42bd1c89d9..5041cb11a1a 100644 --- a/src/Apps/W1/EDocument/App/src/Processing/docs/business-logic.md +++ b/src/Apps/W1/EDocument/App/src/Processing/docs/business-logic.md @@ -2,38 +2,50 @@ ## Overview -Processing owns two major flows: outbound export (posting a BC document into an E-Document and handing it to the Integration layer) and inbound order matching (reconciling imported e-document lines against purchase order lines). The export flow is event-driven and largely automatic; order matching is interactive with optional AI assistance. Error handling throughout uses the `Commit(); if not Codeunit.Run()` pattern -- every interface call is isolated so a failure logs an error without rolling back the surrounding transaction. +*Updated: 2026-07-29 -- added purchase order export, sales order response messaging, and current error-handling boundaries.* + +Processing owns the outbound export flow, the parent entry point for inbound processing, interactive V1 purchase order matching, E-Document message responses, attachment handling, and shared AI tooling. The export flow is event-driven and now covers both posted documents and released purchase orders. Inbound sales and purchase draft creation is delegated to `Import/`; this parent doc only calls out the boundaries where Processing dispatches into that pipeline or reacts after a draft creates a BC document. + +Error handling throughout still uses the `Commit(); if not Codeunit.Run()` pattern -- every interface call is isolated so a failure logs an error without rolling back the surrounding transaction. Page actions that retry incoming processing now surface a message when processing leaves errors on the E-Document, and document cleanup clears linked error messages with the rest of the E-Document-owned data. ## Export flow -The outbound pipeline starts when a BC document is posted and ends when the E-Document is queued for sending. +*Updated: 2026-07-29 -- added released purchase orders and PEPPOL purchase order export.* + +The outbound pipeline starts from a document event and ends when the E-Document is queued for sending. ```mermaid flowchart TD - A[BC document posted] --> B{Document Sending Profile = Extended E-Document Service Flow?} - B -- No --> Z[No E-Document created] - B -- Yes --> C[EDocExport.CreateEDocument] - C --> D[Create E-Document record with Status = In Progress] - D --> E{For each service in workflow} - E --> F{IExportEligibilityEvaluator.ShouldExport?} - F -- No --> E - F -- Yes --> G{Service uses batch processing?} - G -- Yes --> H[Set status = Created, wait for batch job] - G -- No --> I[MapEDocument + format.Create = TempBlob] - I --> J{Export succeeded?} - J -- No --> K[Status = Export Error, log error] - J -- Yes --> L[Status = Exported, store blob in log] - L --> M[StartEDocumentCreatedFlow -- enqueue background job] - M --> N[Workflow evaluates -- triggers Send / Email / Approval] - H --> O[Recurrent batch job collects pending-batch docs] - O --> P[ExportEDocumentBatch + SendBatch] + A[BC document event] --> B{Event source} + B -- Posted sales, service, finance charge, reminder, transfer, or shipment --> C{Document Sending Profile = Extended E-Document Service Flow?} + B -- Purchase Order released --> C + C -- No --> Z[No E-Document created] + C -- Yes --> D[EDocExport.CreateEDocument] + D --> E[Resolve enabled workflow services] + E --> F{Service supports document type and eligibility evaluator allows it?} + F -- No supported services --> Z + F -- Yes --> G[Create or re-export E-Document with service status Created] + G --> H{Service uses batch processing?} + H -- Yes --> I[Leave status Created for recurrent batch job] + H -- No --> J[MapEDocument + format.Create = TempBlob] + J --> K{Export succeeded?} + K -- No --> L[Status = Export Error, log error] + K -- Yes --> M[Status = Exported, store blob in log] + M --> N[StartEDocumentCreatedFlow -- enqueue background job] + N --> O[Workflow evaluates -- triggers Send / Email / Approval] + I --> P[Recurrent batch job groups pending docs] + P --> Q[ExportEDocumentBatch + SendBatch] ``` Key decision points in the flow: -- **Document Sending Profile.** The subscriber checks whether the customer/vendor has a profile with `"Electronic Document" = "Extended E-Document Service Flow"` and a valid, enabled workflow. If not, no E-Document is created. +- **Document Sending Profile.** The subscriber checks whether the customer, vendor, or location has a profile with `"Electronic Document" = "Extended E-Document Service Flow"` and a valid, enabled workflow. If not, no E-Document is created. The page action for manual creation validates this setup before it calls the subscriber path. -- **Export eligibility.** Each service in the workflow is checked individually via `IExportEligibilityEvaluator.ShouldExport()`. The default implementation allows all documents, but extensions can filter by document type, amount, customer attributes, or any other criteria. The service's `"E-Doc. Service Supported Type"` table is also checked before the evaluator runs. +- **Purchase order send path.** `EDocumentSubscribers.OnAfterReleasePurchaseDoc()` handles released purchase orders, not posted purchase invoices. It resolves the vendor's electronic profile, then calls `CreateEDocumentFromPostedDocument()` with `"Purchase Order"` and `AllowReExport = true`. `EDocExport.GetLines()` reads `Purchase Line` records for the order, and `IsDocumentTypeSupported()` accepts a service that supports either Purchase Order or Purchase Invoice for this source type. + +- **PEPPOL format boundary.** The core `"E-Document Format"` enum still contains `"PEPPOL BIS 3.0"` and binds it to `EDoc PEPPOL BIS 3.0`. That codeunit is a bridge: for purchase orders it calls PEPPOL app code such as `Export Purchase Order PEPPOL30`, using the PEPPOL setup's purchase format. Do not describe the Processing module as owning the PEPPOL XML builders themselves. + +- **Export eligibility.** Each service in the workflow is checked individually via `IExportEligibilityEvaluator.ShouldExport()`. The default implementation allows all documents, but extensions can filter by document type, amount, customer/vendor attributes, or any other criteria. The service's `"E-Doc. Service Supported Type"` table is also checked before the evaluator runs. - **Batch vs. immediate.** If the service has `"Use Batch Processing"` enabled, the document gets status `Created` and is not exported inline. A recurrent job queue entry (`"E-Doc. Recurrent Batch Send"`) picks up all pending-batch documents at the configured interval, groups them by document type, exports them as a single batch blob, and sends the batch. @@ -41,29 +53,55 @@ Key decision points in the flow: - **Error isolation.** `EDocumentCreate.Codeunit.al` is a runner codeunit invoked with `Codeunit.Run()`. If the format interface throws, `GetLastErrorText()` is captured and logged against the E-Document without aborting the caller. -## Order matching (two separate systems) +## E-Document message responses -There are two distinct order matching systems in the codebase. They serve different purposes and use different data models. Do not confuse them. +*Updated: 2026-07-29 -- documented PEPPOL Order Response handling and response message storage.* + +Processing stores lifecycle messages separately from the parent E-Document. The first built-in message type is `"PEPPOL Order Response"`, used when an inbound order becomes a sales order and BC needs to send an acceptance or rejection response back through the service flow. + +```mermaid +flowchart TD + A[Incoming E-Document] --> B[V2 import pipeline in Import] + B --> C{Draft process creates which BC document?} + C -- Sales Order --> D[Sales Header linked to E-Document] + C -- Purchase document or journal --> X[Handled by Import docs] + D --> E{Sales Order released?} + E -- Yes --> F[EDocumentSubscribers.OnAfterReleaseSalesDoc] + E -- User rejects inbound order --> G[EDocumentProcessing.SendOrderRejection] + F --> H[Ask document format through IEDocResponseProvider] + G --> H + H --> I{Response message type returned?} + I -- Unknown --> Z[No message] + I -- PEPPOL Order Response --> J[IEDocMessageBuilder.BuildMessage] + J --> K[E-Doc. PEPPOL Msg. Builder delegates XML to PEPPOL app] + K --> L[E-Doc. Message Mgt. creates E-Document Message + Data Storage] +``` -### V2 import pipeline PO matching (automatic, during Prepare Draft) +Key points: -This is the **newer** system, part of the V2.0 import pipeline in `Import/Purchase/PurchaseOrderMatching/`. It runs automatically during the "Prepare draft" stage when an incoming e-document references a purchase order number. +- `IEDocResponseProvider` is implemented by the document format enum value, so a format can decide whether the current E-Document should emit a response message. The PEPPOL handler returns the PEPPOL Order Response message type for applicable inbound sales orders. -The flow: -1. During Prepare Draft, `PreparePurchaseEDocDraft` calls `IPurchaseOrderProvider.GetPurchaseOrder()` to look up a PO by order number from the e-document header. -2. If found, `EDocPOMatching.MatchPOLinesToEDocumentLine()` matches e-document purchase lines to PO lines. -3. After matching, `CalculatePOMatchWarnings()` generates warnings for over-receipt, under-receipt, quantity mismatches, etc. -4. During Finish Draft, `SuggestReceiptsForMatchedOrderLines()` proposes receipt lines, and `TransferPOMatchesFromEDocumentToInvoice()` writes matches to the created purchase invoice. +- `IEDocMessageBuilder` is implemented by the message type enum value. Core's PEPPOL builder reads the E-Document sales draft header and passes only primitive values to the standalone PEPPOL app's `PEPPOL Order Resp. Builder`. -**Key data:** Matches are stored in `"E-Doc. Purchase Line PO Match"` (table 6114) -- a junction table linking e-document lines to PO lines and receipt lines via SystemIds. Warnings go in `"E-Doc. PO Match Warning"` (table 6115). Receipt behavior is configurable per vendor in `"E-Doc. PO Matching Setup"` (Always Ask / Always Receive / Never Receive). +- Accepted responses are created automatically after the linked sales order is released. Rejected responses are created by the E-Document page's Reject Order action through `SendOrderRejection()`. -**Main codeunit:** `EDocPOMatching.Codeunit.al` (codeunit 6196) in `Import/Purchase/PurchaseOrderMatching/`. +- Messages are child records. `E-Document Message` stores message type, direction, response type, service, status, and a pointer to the XML blob in `E-Doc. Data Storage`; the messages factbox lets users download the raw XML. -**Extensibility:** Override `IPurchaseOrderProvider.GetPurchaseOrder()` to customize how POs are looked up (e.g., match by vendor + date range instead of order number). +## Order matching (two separate systems) + +*Updated: 2026-07-29 -- narrowed this parent section to boundaries and pointed V2 details to Import docs.* + +There are two distinct order matching systems in the codebase. They serve different purposes and use different data models. Do not confuse them. + +### V2 import pipeline PO matching (automatic, during Prepare draft) + +This is the newer system, part of the V2.0 import pipeline in `Import/Purchase/PurchaseOrderMatching/`. It runs automatically during the Prepare draft and Finish draft stages when an incoming e-document references a purchase order number. + +This parent Processing doc should not duplicate the V2 matching details. See the Import docs for the stage flow, PO match tables, warnings, receipt suggestion behavior, and `IPurchaseOrderProvider` customization point. From this level, the important boundary is that `EDocImport.Codeunit.al` drives the configured import steps toward the desired processing status, while `Import/` owns the actual draft preparation and PO match data model. ### V1 interactive order matching (user-driven, post-import) -This is the **older** system in `OrderMatching/`. It applies after the import pipeline has linked an E-Document to a purchase order (status `"Order Linked"`). The goal is to interactively reconcile imported e-document lines with PO lines so that `"Qty. to Invoice"` is set correctly before posting. +This is the older system in `OrderMatching/`. It applies after the import pipeline has linked an E-Document to a purchase order (status `"Order Linked"`). The goal is to interactively reconcile imported e-document lines with PO lines so that `"Qty. to Invoice"` is set correctly before posting. **Automatic matching** (`EDocLineMatching.MatchAutomatically`) filters PO lines to those with the same unit of measure, direct unit cost, and line discount as the imported line, then applies three matching strategies in order: @@ -75,27 +113,37 @@ Each successful match creates an `"E-Doc. Order Match"` record linking the e-doc **Manual matching** lets users select one or more imported lines and one or more PO lines on the `"E-Doc. Order Line Matching"` page. The framework validates that all selected lines share the same unit cost, discount, and UOM before creating the match. -**Learn matching rule.** When a match is accepted with the "Learn" flag, the framework creates an Item Reference (for items) or a Text-to-Account Mapping (for G/L accounts) so future automatic matching will recognize the same pattern. +**Learn matching rule.** When a match is accepted with the Learn flag, the framework creates an Item Reference for items or a Text-to-Account Mapping for G/L accounts so future automatic matching will recognize the same pattern. **Apply to purchase order.** `ApplyToPurchaseOrder()` validates that all imported lines are fully matched, then writes the matched unit costs and discounts to the actual PO lines and links the purchase header to the E-Document via `"E-Document Link"`. ### Common gotcha: which matching system applies? -If you are working on the V2.0 import pipeline (Prepare Draft / Finish Draft stages, `"E-Doc. Purchase Line PO Match"` table), you are in the **new** system. If you are working with `"E-Doc. Order Match"` records or the `"E-Doc. Order Line Matching"` page, you are in the **old** system. The two do not share data models, codeunits, or flow paths. Code changes to one should not be applied to the other without understanding which pipeline the document is going through. +If you are working on the V2.0 import pipeline (Prepare Draft / Finish Draft stages, `"E-Doc. Purchase Line PO Match"` table), you are in the new system. If you are working with `"E-Doc. Order Match"` records or the `"E-Doc. Order Line Matching"` page, you are in the old system. The two do not share data models, codeunits, or flow paths. Code changes to one should not be applied to the other without understanding which pipeline the document is going through. ## Copilot PO matching -When automatic matching (V1 interactive system) leaves unmatched lines, users can invoke Copilot from the matching page. `EDocPOCopilotMatching.MatchWithCopilot()` builds a prompt containing imported line and PO line descriptions, sends it to Azure OpenAI (GPT-4.1 via the `"E-Document Matching Assistance"` Copilot capability), and interprets the response through function-calling tools. +*Updated: 2026-07-29 -- marked the matching-page Copilot path obsolete and hidden.* -The Copilot result is **grounded** before being shown: the framework verifies that proposed matches respect the cost difference threshold configured in `"Purchases & Payables Setup"."E-Document Matching Difference"`. Proposals that exceed the threshold are discarded. Accepted proposals are surfaced on a proposal page where the user reviews and confirms. +The V1 matching-page Copilot path is obsolete behind `#if not CLEAN29`. The Match with Copilot action and promoted action are hidden, and the obsolete reason points developers to import-time AI matching in `E-Doc. AI Tool Processor`. The old code still exists for compatibility until cleanup, including prompt building and grounding against purchase line cost and quantity, but it is not the place for new matching work. ## AI tools for import processing -`EDocAIToolProcessor` is a reusable Copilot orchestrator used during import processing (not order matching). It configures Azure OpenAI with a system prompt, registers tools from `IEDocAISystem` implementations, and executes function calls from the model's response. The `Tools/` subfolder provides four tools: +*Updated: 2026-07-29 -- refreshed current model, capability registration, and historical matching behavior.* + +`EDocAIToolProcessor` is a reusable Copilot orchestrator used during import processing. It registers the E-Document Matching Assistance capability in SaaS, checks that the capability is active, configures Azure OpenAI with the latest GPT-4.1 chat deployment, and registers tools from `IEDocAISystem` implementations. The `Tools/` subfolder provides four tools: -- **Historical matching** -- suggests line mappings based on previously accepted matches for the same vendor. -- **G/L account matching** -- proposes G/L accounts based on description similarity to the chart of accounts. +- **Historical matching** -- first tries direct history for the same vendor and product history; if unresolved lines remain, it builds a bounded historical candidate set and asks the model to choose matches. +- **G/L account matching** -- proposes G/L accounts using company/vendor context, posting accounts, and line descriptions. - **Deferral matching** -- suggests deferral codes for lines that appear to represent recurring charges. - **Similar descriptions** -- finds items or G/L accounts with descriptions similar to the imported line text. -Each tool implements the `IEDocAISystem` interface and registers via the `OnAfterRegister*` event pattern. The `EDocAIToolProcessor.Process()` method handles token counting (125k input limit), API error handling, and function call dispatch. +Each tool implements the `IEDocAISystem` interface and registers via the `OnAfterRegister*` event pattern. The `EDocAIToolProcessor.Process()` method handles language-aware system prompts, token counting (125k input limit), API error handling, function call dispatch, and telemetry. + +## Attachment and cleanup handling + +*Updated: 2026-07-29 -- added Digital Voucher attachment context and E-Document cleanup behavior.* + +Attachments created against an E-Document are stamped with `"E-Document Attachment"` and `"E-Document Entry No."` by `EDocAttachmentProcessor`. The same codeunit helps the document attachment factbox recover the E-Document record when the normal record reference is missing, which is important for Digital Voucher and factbox attachment flows. During import finalization, attachments can be moved from the E-Document to the created purchase document; during cleanup, only attachments tied to the E-Document entry are deleted. + +E-Document cleanup is intentionally broad because the E-Document is the owner of processing artifacts. It clears Error Message records, logs, integration logs, service statuses, E-Document messages, mapping logs, imported lines, E-Document attachments, and V2 draft data through the configured process draft implementation. This keeps deleted or orphaned E-Documents from leaving stale processing state behind. diff --git a/src/Apps/W1/EDocument/App/src/Service/docs/data-model.md b/src/Apps/W1/EDocument/App/src/Service/docs/data-model.md index 428a9b83a8a..5dda1801a3e 100644 --- a/src/Apps/W1/EDocument/App/src/Service/docs/data-model.md +++ b/src/Apps/W1/EDocument/App/src/Service/docs/data-model.md @@ -4,10 +4,32 @@ This describes the service configuration data model. For the full cross-module d ## Service configuration and type bridge +*Updated: 2026-07-29 -- Added service-owned import v2 and draft-format configuration.* + The `E-Document Service` table (6103) is the configuration hub. The `E-Doc. Service Supported Type` table (6122) creates an N:M bridge between services and the `E-Document Type` enum, with a composite primary key of `(E-Document Service Code, Source Document Type)`. This means a single service can handle Sales Invoices, Purchase Invoices, and Credit Memos, while the same document type can be handled by multiple services. +The service also owns the default inbound import strategy. `Import Process` now initializes to Version 2.0 for new services, `Automatic Import Processing` decides whether the v2 pipeline continues automatically or stops for review, and `Read into Draft Impl.` is the service-level Draft Format used to read structured content into a draft when the integration or structuring step has not selected a reader. + ```mermaid erDiagram + E-Document-Service { + Code Code PK + Enum Document_Format + Enum Service_Integration_V2 + Enum Import_Process + Enum Automatic_Import_Processing + Enum Read_into_Draft_Impl + } + E-Doc-Service-Supported-Type { + Code E-Document_Service_Code PK + Enum Source_Document_Type PK + } + Service-Participant { + Code Service PK + Enum Participant_Type PK + Code Participant PK + Text Participant_Identifier + } E-Document-Service ||--o{ E-Doc-Service-Supported-Type : "accepts document types" E-Document-Service ||--o{ E-Document-Service-Status : "tracks per-document state" E-Document-Service ||--o{ Service-Participant : "has participants" @@ -36,8 +58,12 @@ The batch and import jobs use separate scheduling configurations: batch jobs hav ## Design decisions and gotchas +*Updated: 2026-07-29 -- Clarified why import reader selection belongs to the service record.* + - The `Service Integration` field (v1) is being replaced by `Service Integration V2`. The old field is pending removal (CLEAN26/29 tags). During the transition, both fields may coexist, and code paths check both. The v2 field uses the `Service Integration` enum which implements `IConsentManager` for privacy consent. +- The import version and Draft Format live on the service rather than on the connector. This lets the same integration receive different structured payloads while the service record tells the v2 import pipeline which `E-Doc. Read into Draft` implementation should consume the structured data. + - The supported type bridge table has `ReplicateData = false`, meaning it is not replicated across environments. This is intentional -- service configurations are environment-specific. - The service carries no direct reference to E-Document records. The relationship is always mediated through `E-Document Service Status`, which is owned by the Document module. This keeps the service as a pure configuration entity. diff --git a/src/Apps/W1/EDocument/App/src/Setup/docs/CLAUDE.md b/src/Apps/W1/EDocument/App/src/Setup/docs/CLAUDE.md index d4ba7e7f41f..92b0e2268a6 100644 --- a/src/Apps/W1/EDocument/App/src/Setup/docs/CLAUDE.md +++ b/src/Apps/W1/EDocument/App/src/Setup/docs/CLAUDE.md @@ -4,17 +4,21 @@ Installation, upgrade, and consent management for the E-Document Core app. This ## How it works +*Updated: 2026-07-29 -- Added the new upgrade actions for v2 Data Exchange definitions, process-draft enum migration, and purchase VAT defaults.* + **`EDocumentSetup`** (Install codeunit) runs `OnInstallAppPerCompany` and does two things: (1) creates the Workflow Table Relation records that link `E-Document` and `E-Document Service Status` tables bidirectionally by entry number -- this is required for the Workflow engine to navigate between the two tables during event processing; (2) registers four tables (`E-Document Log`, `E-Document Integration Log`, `E-Doc. Data Storage`, `E-Doc. Mapping Log`) with the Retention Policy framework so administrators can configure automatic cleanup of historical data. -**`EDocumentUpgrade`** runs on version upgrades. Currently it has one migration: `UpgradeLogURLMaxLength`, which copies the old `URL` field to a new `"Request URL"` field on `E-Document Integration Log` using `DataTransfer` (bulk copy, no row-by-row loop). The upgrade is gated by an upgrade tag (`MS-540448-LogURLMaxLength-20240813`) and registered via `OnGetPerCompanyUpgradeTags`. +**`EDocumentUpgrade`** runs on version upgrades. It still performs `UpgradeLogURLMaxLength`, which copies the old `URL` field to a new `"Request URL"` field on `E-Document Integration Log` using `DataTransfer` (bulk copy, no row-by-row loop). It now also imports the v2 PEPPOL invoice and credit memo Data Exchange definitions, migrates existing E-Documents from the old generic purchase-draft enum value to `Purchase Invoice` while the compatibility code is present, and enables the purchase E-Document VAT difference and VAT group resolution options in Purchases & Payables Setup. **`ConsentManagerDefaultImpl`** implements the `IConsentManager` interface with a standard privacy consent dialog. When a user first configures an E-Document connector, this prompts them to acknowledge that third-party systems may have different compliance and privacy standards. Connector implementations can substitute their own consent manager via the interface. ## Things to know +*Updated: 2026-07-29 -- Listed the additional upgrade tags now registered per company.* + - The install codeunit uses `if Insert() then;` (swallow-failure pattern) for workflow table relations because the install runs on every upgrade, not just first install, and the records may already exist. - Retention policies are created disabled by default (`Enabled = false`) with `"Apply to all records" = true`. Administrators must explicitly enable them and set retention periods. -- The upgrade tag format includes a work item number and date (`MS-540448-LogURLMaxLength-20240813`), following BC's standard upgrade tag conventions. +- The upgrade tag format includes a work item number and date (`MS-540448-LogURLMaxLength-20240813`), following BC's standard upgrade tag conventions. Newer tags follow the same purpose-and-date style, for example `MS-EDoc-DataExchV2Defs-20260521` and `MS-EDoc-EnableVATOptionsForPurchEDoc-20260520`. - `DataTransfer.CopyFields` is used instead of record-by-record modification for the URL migration -- this is significantly faster on large log tables. - The consent text is a single hardcoded label. It does not vary by connector -- connectors that need specific consent language should implement their own `IConsentManager`. diff --git a/src/Apps/W1/EDocument/App/src/Workflow/docs/CLAUDE.md b/src/Apps/W1/EDocument/App/src/Workflow/docs/CLAUDE.md index 07bb44aa38a..c634b65e18f 100644 --- a/src/Apps/W1/EDocument/App/src/Workflow/docs/CLAUDE.md +++ b/src/Apps/W1/EDocument/App/src/Workflow/docs/CLAUDE.md @@ -4,19 +4,23 @@ Workflow orchestration for outbound E-Documents -- wires the E-Document lifecycl ## How it works +*Updated: 2026-07-29 -- Added idempotent export-step behavior.* + E-Documents use Document Sending Profile to select a Workflow Code, which defines the processing chain. When a document is posted, `EDocumentCreatedFlow` runs as a Job Queue Entry, firing the `EDOCCREATEDEVENT` workflow event. The workflow engine then walks the step chain, executing responses in order. `EDocumentWorkFlowSetup` registers four events (`EDocCreated`, `EDocStatusChanged`, `EDocImported`, `EDocExported`) and five responses (`Send`, `Export`, `Import`, `EmailEDoc`, `EmailPDFAndEDoc`) into the BC workflow library via event subscribers. It also installs two built-in templates: "Send to one service" (EDOCTOS) and "Send to multiple services" (EDOCTOM). The multi-service template chains two Send responses off the same entry point event, each targeting a different service. -`EDocumentWorkFlowProcessing` is the response executor. When the workflow engine dispatches a response, the `ExecuteEdocWorkflowResponses` subscriber routes to the appropriate method: `SendEDocument`, `ExportEDocument`, or `SendEDocFromEmail`. Each method resolves the E-Document Service from the `Workflow Step Argument` (extended with an `"E-Document Service"` field via table extension) and delegates to the export/integration layer. After execution, `HandleNextEvent` fires `EDocStatusChanged` to advance the workflow to the next step, enabling multi-step chains like Export -> Send -> Email. +`EDocumentWorkFlowProcessing` is the response executor. When the workflow engine dispatches a response, the `ExecuteEdocWorkflowResponses` subscriber routes to the appropriate method: `SendEDocument`, `ExportEDocument`, or `SendEDocFromEmail`. Each method resolves the E-Document Service from the `Workflow Step Argument` (extended with an `"E-Document Service"` field via table extension) and delegates to the export/integration layer. After execution, `HandleNextEvent` fires `EDocStatusChanged` to advance the workflow to the next step, enabling multi-step chains like Export -> Send -> Email. The Export response is idempotent for an already-exported service status: it raises the exported workflow event and exits instead of generating a new export blob. For async services, after a send returns `IsAsync = true`, a background job is scheduled to poll `GetResponse` until the service confirms delivery. ## Things to know +*Updated: 2026-07-29 -- Clarified that both Export and Send avoid duplicate exports.* + - The `Workflow Step Argument` table extension (field `"E-Document Service"`) is the critical link between a workflow response step and the E-Document Service it targets. Each response step in the chain can target a different service. - `ValidateFlowStep` ensures the workflow step instance matches the E-Document's stored `"Workflow Code"` -- this prevents cross-workflow execution when multiple workflows are enabled. -- The `DoSend` path has an optimization: if the document was already exported (status = `Exported` from a prior Export step), it skips re-export and goes straight to the integration send. +- The `DoSend` path and the explicit Export response both check `E-Document Service Status`. If the service is already `Exported`, Send goes straight to the integration send and Export advances the workflow without re-exporting. - Batch processing has three modes: `Recurrent` (exits immediately, a separate scheduled job handles it), `Threshold` (waits until N documents of the same type accumulate), and custom (raises `OnBatchSendWithCustomBatchMode` for extensions). - Email responses (`SendEDocFromEmail`) look up the previous Send or Export response in the workflow to find which service produced the document -- they don't require their own service argument. - The `EDocWorkflowStepArgumentArch` table extension mirrors the argument field to the archive table, preserving which service was used after workflow archival. diff --git a/src/System Application/App/Privacy Notice/CLAUDE.md b/src/System Application/App/Privacy Notice/CLAUDE.md index 9fefbd763e6..365309c21d4 100644 --- a/src/System Application/App/Privacy Notice/CLAUDE.md +++ b/src/System Application/App/Privacy Notice/CLAUDE.md @@ -1,9 +1,12 @@ # Privacy Notice Consent management layer for Business Central integrations that access external -services (Teams, Power Automate, Bing, Microsoft Learn). Handles the full -lifecycle: creating notices, prompting users for consent, storing approval state, -and enforcing admin-level decisions that override individual user choices. +services (Teams, Power Automate, Bing, Microsoft Learn, Microsoft Copilot). +Handles the full lifecycle: creating notices, prompting users for consent, +storing approval state, and enforcing admin-level decisions that override +individual user choices. + +*Updated: 2026-07-29 -- Microsoft Copilot privacy notice registered* ## Quick reference @@ -33,8 +36,19 @@ non-admins are not stored -- they just keep getting prompted). Privacy notices are registered via the `OnRegisterPrivacyNotices` event. All subscribers contribute to a temp table, and the module bulk-inserts any that don't already exist. The `ShouldApproveByDefault` mechanism auto-approves specific -integrations on first creation (currently only Microsoft Learn when in-geo support -is detected via a DotNet call). +integrations on first creation. It is region-conditional, and each case is a +separate DotNet probe wrapped in a `[TryFunction]` so a platform failure degrades +to "not approved" rather than erroring: + +- **Microsoft Learn** -- approved when in-geo support is detected. +- **Microsoft Copilot** -- approved when the environment is *not* within the EU + Data Boundary (`not ALCopilotFunctions.IsWithinEUDB()`). EEA environments + therefore start unapproved and require an explicit admin decision. + +Default approval only applies at registration time. An admin can change it +afterwards on the Privacy Notices page, and that stored decision wins. + +*Updated: 2026-07-29 -- Microsoft Copilot default approval keyed to EU Data Boundary* ## Structure @@ -75,3 +89,10 @@ Two integration events on codeunit 1563: (empty GUID) so it only shows org-wide decisions, not per-user ones. - `CreateDefaultPrivacyNoticesInSeparateThread` wraps `Codeunit.Run` to isolate registration failures -- if any subscriber errors, the calling flow isn't blocked. +- Every codeunit in the module declares `InherentEntitlements = X` and + `InherentPermissions = X`, including the public facade (1563). This is what lets + service-invoked callers reach the facade without the caller holding Privacy + Notice permissions. It does not weaken the admin check, which still comes from + `PrivacyNoticeApproval.WritePermission()`. + +*Updated: 2026-07-29 -- inherent permissions on the facade for service-invoked callers* From 2b72269851f45788533bb15d1dc6e791047e6eaa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Magnus=20Hartvig=20Gr=C3=B8nbech?= Date: Wed, 29 Jul 2026 10:42:46 +0200 Subject: [PATCH 3/3] Add .docs-updated markers for the remaining documented apps Only the Shopify app and test app carried a `.docs-updated` marker. Every other documented app relied on the git history of its docs folder to work out a baseline, which is easy to miss: E-Document had drifted 55 commits behind before anyone noticed, because nothing pointed at it. Adds markers for the four remaining documentation roots so the next refresh has an explicit baseline: - src/Apps/W1/EDocument/App - src/Apps/W1/APIReportsFinance/App - src/Apps/W1/DataCorrectionFA/App - src/System Application/App/Privacy Notice All 82 doc-bearing folders in the repository now roll up under one of six markers. DataCorrectionFA has had no AL changes since its docs were written, so its docs were already current and its marker records that rather than any edit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/Apps/W1/APIReportsFinance/App/.docs-updated | 4 ++++ src/Apps/W1/DataCorrectionFA/App/.docs-updated | 4 ++++ src/Apps/W1/EDocument/App/.docs-updated | 4 ++++ src/System Application/App/Privacy Notice/.docs-updated | 4 ++++ 4 files changed, 16 insertions(+) create mode 100644 src/Apps/W1/APIReportsFinance/App/.docs-updated create mode 100644 src/Apps/W1/DataCorrectionFA/App/.docs-updated create mode 100644 src/Apps/W1/EDocument/App/.docs-updated create mode 100644 src/System Application/App/Privacy Notice/.docs-updated diff --git a/src/Apps/W1/APIReportsFinance/App/.docs-updated b/src/Apps/W1/APIReportsFinance/App/.docs-updated new file mode 100644 index 00000000000..ff8167354eb --- /dev/null +++ b/src/Apps/W1/APIReportsFinance/App/.docs-updated @@ -0,0 +1,4 @@ +# Documentation last updated +commit: 1b8465eff867e0da8910d924f7643e9db51717bc +date: 2026-07-29 +scope: full diff --git a/src/Apps/W1/DataCorrectionFA/App/.docs-updated b/src/Apps/W1/DataCorrectionFA/App/.docs-updated new file mode 100644 index 00000000000..ff8167354eb --- /dev/null +++ b/src/Apps/W1/DataCorrectionFA/App/.docs-updated @@ -0,0 +1,4 @@ +# Documentation last updated +commit: 1b8465eff867e0da8910d924f7643e9db51717bc +date: 2026-07-29 +scope: full diff --git a/src/Apps/W1/EDocument/App/.docs-updated b/src/Apps/W1/EDocument/App/.docs-updated new file mode 100644 index 00000000000..ff8167354eb --- /dev/null +++ b/src/Apps/W1/EDocument/App/.docs-updated @@ -0,0 +1,4 @@ +# Documentation last updated +commit: 1b8465eff867e0da8910d924f7643e9db51717bc +date: 2026-07-29 +scope: full diff --git a/src/System Application/App/Privacy Notice/.docs-updated b/src/System Application/App/Privacy Notice/.docs-updated new file mode 100644 index 00000000000..ff8167354eb --- /dev/null +++ b/src/System Application/App/Privacy Notice/.docs-updated @@ -0,0 +1,4 @@ +# Documentation last updated +commit: 1b8465eff867e0da8910d924f7643e9db51717bc +date: 2026-07-29 +scope: full