Task t66bef/catalog auto discovery feature documentation

docs/build-your-software-catalog/catalog-auto-discovery.md

@@ -0,0 +1,125 @@
+import ThemedImage from "@theme/ThemedImage"
+
+# Catalog auto discovery
+
+:::info Open beta
+This feature is currently in open beta and available to all organizations.
+:::
+
+The **auto discovery** capability uses [Port AI](/ai-interfaces/port-ai/overview) to discover entities and their relations.
+This helps you maintain a complete and accurate catalog, especially for entities that are not automatically created through integrations (see common use-cases below).
+
+<h3>Common use cases</h3>
+
+- **Services**: Service blueprint centralizes different components of a service like its repository, incidents for example. For that reason, unlike GitHub repositories or PagerDuty services that sync automatically from integrations, services are typically created manually. Auto discovery helps you identify and create these missing services.
+- **Users**: Discover users from related entities. For instance, if you have GitHub repositories synced, we can analyze pull requests and issues entities to suggest users who contributed to them but do not yet exist in your catalog.
+
+## How to use catalog auto discovery
+
+**Run the discovery:**
+
+1. Navigate to the [Catalog](https://app.port.io/organization/catalog) page of your portal.
+
+2. Open the catalog page of the blueprint for which you want to discover new entities.
+
+3. Click on the <ThemedImage sources={{light: "/img/icons/AI-icon.svg", dark: "/img/icons/AI-dark-icon.svg"}} style={{"vertical-align": "text-top"}} className="not-zoom" /> button in the top right corner of the page.
+
+4. For the best results, we recommend providing the definition of the blueprint you want to discover, along with clear instructions for patterns or specific properties that should be considered.
+
+    For example:
+
+    - **Mono-repo microservices:** 
+        ```
+        Services are represented as code in a repository.  
+        Check the file structure of each repository to identify services.  
+        Services may be found in specific folders, such as "apps" or "services".
+        ```
+    - **Service repository identification:** 
+        ```
+        Focus on repos that have keywords that can indicate they are services 
+        (e.g., "service", "ms", "srv").  
+        Ignore repos of libraries and packages. Having also a PagerDuty service 
+        with a similar name as a repo is a strong indication that this is a service.
+        ```
+    - **Identify users:** 
+        ```
+        Check "Jira issues" assignees and "pull requests" to identify developers in the organization.
+        ```
+
+5. Select related blueprints to analyze. The entities from these blueprints will be used to identify patterns and suggest new entities for your target blueprint. This field is mandatory and is automatically filled with all directly related blueprints.
+
+6. Click on the `Discover` button.
+
+    <img src="/img/software-catalog/pages/catalogDiscoveryForm.png" border='1px' width='50%' style={{borderRadius:'8px'}} />
+
+**Review and edit suggestions:**
+
+Once the process is complete, a list of suggested entities will be displayed, divided into two sections: **Create** and **Update**.  
+
+You can:
+
+- Edit individual entity suggestions.
+- Approve or decline suggestions individually or in bulk.
+- View the proposed updates to existing entities by clicking the <ThemedImage sources={{light: "/img/icons/Diff-icon.svg", dark: "/img/icons/Diff-dark-icon.svg"}} style={{"vertical-align": "text-top"}} className="not-zoom" /> button.
+
+Suggested entities persist until they are approved or declined. You can close the discovery results window and return to review pending suggestions at any time by accessing the discovery results from the blueprint's catalog page using the <ThemedImage sources={{light: "/img/icons/AI-icon.svg", dark: "/img/icons/AI-dark-icon.svg"}} style={{"vertical-align": "text-top"}} className="not-zoom" /> button.
+
+<img src="/img/software-catalog/pages/catalogDiscoveryResultsWindow.png" border='1px' width='80%' style={{borderRadius:'8px'}} />
+<br/><br/>
+
+**Re-run the discovery**
+
+You can re-run the discovery process at any time to generate additional or different suggestions. Each discovery run analyzes the current state of your catalog and may produce new suggestions based on newly added entities, updated relationships, or refined patterns. Re-running the discovery does not affect previously approved or declined suggestions.
+
+:::info Using auto discovery via the API
+This feature is also available via API for a more programatic execution of the process. Refer to the [API reference](/api-reference/port-api) catalog auto discovery for the full list of paths.
+:::
+
+## Permissions
+
+The permissions are derived from the blueprint permissions.  
+You can approve suggested entities only if you have write access to the blueprint.  
+For more information about blueprint permissions, see the [set catalog RBAC](/build-your-software-catalog/set-catalog-rbac/) documentation.
+
+To learn more about how Port AI uses your data, see the [security and data controls](/ai-interfaces/port-ai/security-and-data-controls) documentation.
+
+## Limitations
+
+- **Entity evaluation limit**: Discovery evaluates only the 500 most recently added entities from each related blueprint.
+- **Property truncation**: Only the first 100 characters of each property value are analyzed. Longer content (such as large markdown fields) will be truncated.
+- **LLM provider**: This feature currently uses Port's LLM and does not support [Bring Your Own LLM (BYOLLM)](/ai-interfaces/port-ai/overview#bring-your-own-llm-byollm).
+
+## FAQs
+
+<details>
+<summary><b>Which LLM model is used? (click to expand)</b></summary>
+
+The AI uses the default LLM defined in Port. To learn more, see the [LLM models and providers](/ai-interfaces/port-ai/overview#llm-models-and-providers) documentation.  
+Bring Your Own LLM (BYOLLM) is not currently supported for catalog auto discovery.
+
+</details>
+
+<details>
+<summary><b>Are there usgae limits? (click to expand)</b></summary>
+
+It depends on your LLM setup. To learn more, see the [limits and usage](/ai-interfaces/port-ai/overview#limits-and-usage) documentation.
+
+</details>
+
+<details>
+<summary><b>Is the AI trained based on my data? (click to expand)</b></summary>
+
+No, Port AI does not use your data to train models. To learn more, see the [security and data controls](/ai-interfaces/port-ai/security-and-data-controls) documentation.
+
+</details>
+
+<details>
+<summary><b>How can I improve the auto discovery results? (click to expand)</b></summary>
+
+You can improve results by:
+- Providing more specific instructions in the prompt about patterns to look for.
+- Including clear definitions of what constitutes your target entity type.
+- Selecting the most relevant related blueprints for analysis.
+- Re-running the discovery after refining your prompt based on initial results.
+
+</details>

docs/customize-pages-dashboards-and-plugins/page/catalog-page.md

@@ -170,7 +170,7 @@ Other properties will be hidden by default.
 You can always customize the table to [hide/show columns](/customize-pages-dashboards-and-plugins/page/catalog-page?create-page=ui#hideshow-columns).
 :::
 
-### Description
+<h3>Description</h3>
 
 You can provide additional context to your developers by using the `Description` field when creating a catalog page.  
 This field supports adding links in markdown format: `[link text](https://www.address.com)`.
@@ -183,7 +183,7 @@ The description will be displayed at the top of the page, under the page title:
 
 <img src='/img/software-catalog/pages/catalogPageDescription.png' width='80%' border='1px' />
 
-## Performance
+## Filters and performance
 
 Large entity tables can result in long loading times. Use the following tips and best practices to improve performance.
 
@@ -244,19 +244,23 @@ We highly recommend using these customizations to provide a clean and accurate v
 
 All table customizations are available on the top bar of the table:
 
-![Table operations bar](/img/software-catalog/pages/TableOperationsBar.png)
+<img src='/img/software-catalog/pages/TableOperationsBar.png' width='100%' border='1px' />
 
-### Filter
+### Table filters
+
+:::info Table filters vs initial filters
+Unlike the filters described in the [scection above](#filter-and-performance), this filter does not affect performance. It filters entities that have already been loaded and only affects what is displayed in the table view.
+:::
 
 You can filter the table by using the following menu:
 
-![Table filter menu marked](/img/software-catalog/pages/TableFilterMenu.png)
+<img src='/img/software-catalog/pages/TableFilterMenu.png' width='100%' border='1px' />
 
 You can define any filtering operator with a suitable value.
 
 You can filter one or more values while setting the relation between each field with a `And/Or`.
 
-#### `My Teams` filter
+**`My Teams` filter**
 
 By using the `My Teams` filter you will only see entities that belong to one of your teams. This means you will only see entities from teams that you are a member of.
 
@@ -265,21 +269,22 @@ This filter works on:
 - `string` properties with the format `team`.
 - The [meta property](/build-your-software-catalog/customize-integrations/configure-data-model/setup-blueprint/properties/meta-properties) `Team`.
 
-![My Teams Filter](/img/software-catalog/pages/MyTeamsFilter.png)
+<img src='/img/software-catalog/pages/MyTeamsFilter.png' width='100%' border='1px' />
+<br></br><br></br>
 
-#### `Me` filter
+**`Me` filter**s
 
 By using the `Me` filter you will only see entities that belong to the logged-in user.
 
 This filter works on [`User`](/build-your-software-catalog/customize-integrations/configure-data-model/setup-blueprint/properties/user) properties.
 
-![Me Filter](../../../static/img/software-catalog/pages/meFilter.png)
+<img src='/img/software-catalog/pages/meFilter.png' width='90%' border='1px' />
 
 ### Sort
 
 You can sort the table by using the following menu:
 
-![Table sort menu marked](/img/software-catalog/pages/TableSortMenu.png)
+<img src='/img/software-catalog/pages/TableSortMenu.png' width='100%' border='1px' />
 
 You can sort by one or more fields of any kind.
 
@@ -325,10 +330,18 @@ Just create your `group-by` view (and any other table customizations you desire)
 Port provides a free-text search option on tables. This will search all of the entities' properties and display the entities that match the query.  
 If the query contains multiple words, entities that contain all of these words will be displayed, even if they are spread across different properties. 
 
-![Table search bar marked](/img/software-catalog/pages/TableSearchBar.png)
+<img src='/img/software-catalog/pages/TableSearchBar.png' width='100%' border='1px' />
 
 -> [Explore how to control page visibility and permissions](./page-permissions.md)
 
+
+## Catalog auto discovery
+
+The **auto discovery** capability uses AI to analyze your existing catalog data and suggests missing entities based on existing relationships and patterns.
+This helps you maintain a complete and accurate catalog, especially for entities that are not automatically created through integrations (see common use-cases below).
+
+To learn more about the auto discovery capability, refer to the [catalog auto discovery](/build-your-software-catalog/catalog-auto-discovery) page.
+
 ## Page operations
 
 Pages have a set of operations that can be performed from the UI.  
@@ -357,28 +370,28 @@ You can edit, lock or delete a page by clicking the `...` button in the top righ
 
 <center>
 
-![Page menu](/img/software-catalog/pages/PageMenu.png)
+<img src='/img/software-catalog/pages/PageMenu.png' width='30%' border='1px' />
 
 </center>
 
-#### Editing pages
+**Editing pages**
 
 Editing a page allows you to change various properties:
 
-![Edit Page popup window](/img/software-catalog/pages/EditPageForm.png)
+<img src='/img/software-catalog/pages/EditPageForm.png' width='100%' border='1px' />
 
-#### Locking pages
+**Locking pages**
 
 Locking a catalog page disables the option to hide columns or apply filters to modify the displayed data.
 
 Locking pages gives you a way to specifically curate pages to your developers' needs. This ensures that they can't modify the views or see data that isn't relevant to them.
 
 To learn how to lock pages, refer to [page permissions](./page-permissions.md#lock-pages).
 
-#### Deleting pages
+**Deleting pages**
 
 Any page (whether created automatically or manually) can be deleted by clicking the `Delete page` button.
 
 :::warning Default pages
 When deleting a blueprint from your portal, all pages tied to that blueprint (including the default page that was created for it) will be deleted as well.
-:::
+:::

docs/getting-started/set-up-automatic-discovery.md

@@ -7,6 +7,7 @@ import PortTooltip from "/src/components/tooltip/tooltip.jsx"
 import Tabs from "@theme/Tabs"
 import TabItem from "@theme/TabItem"
 import LogoImage from '/src/components/guides-section/LogoImage/LogoImage.jsx';
+import ThemedImage from "@theme/ThemedImage"
 
 # Set up automatic data mapping
 
@@ -23,12 +24,13 @@ Not sure what these entities mean? See their definitions [here](/getting-started
 
 ## Methods
 
-This page describes two methods you can use to automatically map and update entities in your catalog:
+This page describes three methods you can use to automatically map and update entities in your catalog:
 
-1. Define one of your external tools as a "source of truth" for the resources you want to ingest.
-2. Use metadata from your external tools (e.g. labels, naming conventions) to identify and update an <PortTooltip id="entity">entity</PortTooltip> in Port.
+1. Use the catalog auto discovery which utilizes AI to analyze your existing catalog data and suggest missing entities.
+2. Define one of your external tools as a "source of truth" for the resources you want to ingest.
+3. Use metadata from your external tools (e.g. labels, naming conventions) to identify and update an <PortTooltip id="entity">entity</PortTooltip> in Port.
 
-All methods require modifying the [mapping configuration](/build-your-software-catalog/customize-integrations/configure-mapping) of the relevant integration.  
+Methods two and three require modifying the [mapping configuration](/build-your-software-catalog/customize-integrations/configure-mapping) of the relevant integration.  
 
 <details>
 <summary>**How to modify a mapping configuration (click to expand)**</summary>
@@ -38,7 +40,71 @@ All methods require modifying the [mapping configuration](/build-your-software-c
 4. Click on the "Save & Resync" button to save the changes and resync the integration.
 </details>
 
-## 1. Define a source of truth
+## 1. Catalog auto discovery
+
+The **auto discovery** capability uses [Port AI](/ai-interfaces/port-ai/overview) to discover entities and their relations.
+It is particularly useful for discovering entities that are not automatically created through integrations, such as `services` and `users`.  
+
+To learn more about how Port AI uses your data, see the [security and data controls](/ai-interfaces/port-ai/security-and-data-controls) documentation.
+
+### Examples
+
+To use catalog auto discovery, navigate to the [Catalog](https://app.port.io/organization/catalog) page, open the catalog page of the blueprint you want to discover, click the <ThemedImage sources={{light: "/img/icons/AI-icon.svg", dark: "/img/icons/AI-dark-icon.svg"}} style={{"vertical-align": "text-top"}} className="not-zoom" /> button, and select related blueprints to analyze.
+
+**Providing a prompt is highly recommended** and gives the best and most accurate results.  
+When you enable `Advanced configuration`, you can provide specific instructions that guide the AI to identify patterns relevant to your organization.
+
+**Example 1: Discover services from a monorepo**
+
+If you have a monorepo structure with multiple services, you can discover `service` entities by analyzing your GitHub repositories.
+
+**Recommended prompt:**
+
+```
+Services are represented as code in a repository.  
+Check the file structure of each repository to identify services.  
+Services may be found in specific folders, such as "apps" or "services".
+```
+
+Add the relevant blueprints to base the auto discovery on, for example: `GitHub Repository`.
+
+___
+
+**Example 2: Discover services from individual repositories**
+
+If you have individual repositories and want to identify which ones represent services, you can use catalog auto discovery with both GitHub and PagerDuty data.
+
+**Recommended prompt:**
+
+```
+Focus on repos that have keywords that can indicate they are services (e.g., "service", "ms", "srv").  
+Ignore repos of libraries and packages. Having also a PagerDuty service with a similar name as a repo  
+is a strong indication that this is a service.
+```
+
+Add the relevant blueprints to base the auto discovery on, for example: `GitHub Repository`, `PagerDuty Service`.
+
+___
+
+**Example 3: Discover users from development activity**
+
+If you want to identify users who contribute to your codebase but don't yet exist in your catalog, you can analyze pull requests and issues.
+
+**Recommended prompt:**
+
+```
+Check "Jira issues" assignees and "pull requests" to identify developers in the organization.
+```
+
+Add the relevant blueprints to base the auto discovery on, for example: `Jira Issue`, `Jira User` ,`Pull Requests`.
+
+___
+
+Once the discovery process is complete, you can review, edit, approve, or decline the suggested entities individually or in bulk.
+
+For more information, see the [catalog auto discovery](/build-your-software-catalog/catalog-auto-discovery) documentation.
+
+## 2. Define a source of truth
 
 This approach is useful when you want to create entities of a specific type (e.g. services, environments, teams, users) based on resources from a specific external tool.  
 
@@ -581,7 +647,7 @@ Common examples for resources that can be used as a source of truth for `teams`:
 
 </Tabs>
 
-## 2. Use predefined metadata
+## 3. Use predefined metadata
 
 In addition to defining "sources of truth", you can use data from your external tools to identify and update entities in Port.  
 This is useful when you want to update entities of a specific type (e.g. services, environments, teams, users) based on a label, naming convention, or other piece of metadata in a specific external tool.