Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 120 additions & 26 deletions Server-Side Components/Background Scripts/Duplicate Finder/readme.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,121 @@
## ServiceNow Duplicate Record Finder
A simple server-side script for ServiceNow that finds and reports on duplicate values for any field on any table. It uses an efficient GlideAggregate query and formats the results into a clean, readable report.

### How to Use
This script is designed to be run in **Scripts - Background** or as a Fix Script.
1. Navigate: Go to **System Definition > Scripts - Background** (or type sys.scripts.do in the filter navigator).
2. Copy Script: Copy the entire contents of the script.js file.
3. Paste and Configure: Paste the script into the "Run script" text box. Add the table to search in `tableName` and the field to search for duplicates in `fieldName`
```js
// Update ONLY below values to find duplicates
var tableName = '<table_name>'; // ADD: Table you want for duplicates
var fieldName = 'field_name'; // ADD: Field that you want to check for duplicates
```
For example, to find duplicate email addresses in the User table:
```js
var tableName = 'sys_user';
var fieldName = 'email';;
```
To find incidents with the same short description:
```js
var tableName = 'incident';
var fieldName = 'short_description';
```


4. Run Script: Click the "Run script" button. The results will be displayed on the screen below the editor.
# 🔍 Duplicate Record Finder (Server-Side Script)

## 📁 Location
**Category:** `Server-Side Components`
**Subcategory:** `Background Scripts`
**Snippet Folder:** `Duplicate Record Finder`

---

## 📌 Description

This server-side script helps identify **duplicate records** within any specified table and field in a ServiceNow instance. It uses the powerful `GlideAggregate` API to group and count entries, making it easy to detect data duplication issues across records.

Designed to be executed via **Scripts - Background** or as a **Fix Script**, this utility provides fast insights into data quality without requiring complex queries or reports.

---

## 🚀 Features

- ✅ Works on **any table** and **any field**
- ✅ Uses `GlideAggregate` for efficient grouping and counting
- ✅ Outputs a clear, readable summary of duplicate values
- ✅ Helps detect issues like duplicate CI names, duplicate caller IDs, etc.
- ✅ Non-destructive — the script does not modify any records

---

## 📄 Script: `duplicate_finder.js`

```javascript
var tableName = 'incident'; // Change this to your table
var fieldName = 'caller_id'; // Change this to your field

// --- Validation ---
if (!gs.tableExists(tableName)) {
gs.error('❌ Table "' + tableName + '" does not exist.');
return;
}

var gr = new GlideRecord(tableName);
gr.initialize();
if (!gr.isValidField(fieldName)) {
gs.error('❌ Field "' + fieldName + '" does not exist on table "' + tableName + '".');
return;
}

// --- Find Duplicates ---
var ga = new GlideAggregate(tableName);
ga.addAggregate('COUNT', fieldName);
ga.groupBy(fieldName);
ga.addHaving('COUNT', '>', 1);
ga.addNotNullQuery(fieldName);
ga.query();

var hasDuplicates = false;
gs.info('🔍 Checking duplicates in table: ' + tableName + ', field: ' + fieldName);

while (ga.next()) {
var count = parseInt(ga.getAggregate('COUNT', fieldName), 10);
if (count > 1) {
hasDuplicates = true;
gs.info('⚠️ Value: ' + ga.getDisplayValue(fieldName) + ' | Count: ' + count);
}
}

if (!hasDuplicates) {
gs.info('✅ No duplicates found for "' + fieldName + '" on "' + tableName + '".');
}


🛠️ How to Use

1) Navigate to System Definition > Scripts - Background
2) Paste the script into the editor
3) Update the tableName and fieldName variables
4) Click Run Script
5) Check the output for duplicate groups

📸 Example Output

Duplicate values found in table: incident, field: caller_id

Value: 62826bf03710200044e0bfc8bcbe5df1 | Count: 4
Value: 681ccaf9c0a8016401c5a33be04be441 | Count: 2

Note: Values shown are backend values (e.g., sys_ids for reference fields)

📂 File Structure

Server-Side Components/
└── Background Scripts/
└── Duplicate Record Finder/
├── README.md
└── duplicate_finder.js

⚙️ Requirements

✅ Admin or script execution access
✅ Valid tableName and fieldName
🔁 Optional: Extend to resolve display values using GlideRecord if needed

🧠 Use Case Examples

1) Find duplicate caller_id values in the incident table
2) Detect duplicated serial_number values in cmdb_ci_computer
3) Validate unique constraints during data imports or migrations

✅ Contribution Checklist Compliance

✔️ Follows proper folder structure
✔️ Contains a descriptive README.md
✔️ Code is focused, relevant, and self-contained
✔️ Does not include XML exports or sensitive data
✔️ Uses ServiceNow-native APIs (GlideAggregate)

🧩 Optional Enhancements

1) Add logic to resolve display values from reference fields
2) xtend output to a downloadable CSV format
3) Turn into a Script Include or Scoped App utility

112 changes: 59 additions & 53 deletions Server-Side Components/Background Scripts/Duplicate Finder/script.js
Original file line number Diff line number Diff line change
@@ -1,64 +1,70 @@
// Update ONLY below values to find duplicates
var tableName = 'incident'; // ADD: Table you want for duplicates
var fieldName = 'short_description'; // ADD: Field that you want to check for duplicates

findDuplicates(tableName, fieldName);
(function () {
var tableName = 'incident'; // ✅ Set your target table
var fieldName = 'short_description'; // ✅ Set your target field to check duplicates

function findDuplicates(tableName, fieldName) {
/**************************************/
/*** Basic error handling on inputs ***/
/**************************************/
findDuplicates(tableName, fieldName);

// Check if table exists
if (!gs.tableExists(tableName)) {
// MODIFIED: Switched to string concatenation
gs.info('Table "' + tableName + '" does not exist.');
return;
}
function findDuplicates(tableName, fieldName) {
// --- Validate Table ---
if (!gs.tableExists(tableName)) {
gs.error('❌ Table "' + tableName + '" does not exist.');
return;
}

// Check if field exists
var gr = new GlideRecord(tableName);
gr.initialize();
if (!gr.isValidField(fieldName)) {
gs.print('No field called "' + fieldName + '" on the "' + tableName + '" table.');
return;
}
// --- Validate Field ---
var gr = new GlideRecord(tableName);
gr.initialize();
if (!gr.isValidField(fieldName)) {
gs.error('❌ Field "' + fieldName + '" does not exist on table "' + tableName + '".');
return;
}

/***************************************/
/*********** Find duplicates ***********/
/***************************************/
var duplicateJson = {}; // Store the duplicate records
var duplicateGroupCount = 0; // Counts the number of groups of duplicates
// --- Use GlideAggregate for Efficient Counting ---
var ga = new GlideAggregate(tableName);
ga.addAggregate('COUNT', fieldName);
ga.groupBy(fieldName);
ga.addHaving('COUNT', '>', 1); // Find duplicate groups
ga.addNotNullQuery(fieldName); // Ignore empty or null values
ga.query();

var duplicateAggregate = new GlideAggregate(tableName);
duplicateAggregate.addAggregate('COUNT', fieldName);
duplicateAggregate.groupBy(fieldName);
duplicateAggregate.addHaving('COUNT', '>', 1); // More than 1 means it is a duplicate
duplicateAggregate.addNotNullQuery(fieldName); // Ignore records where the field is empty
duplicateAggregate.query();
var duplicateGroups = [];
while (ga.next()) {
var value = ga.getDisplayValue(fieldName);
var count = parseInt(ga.getAggregate('COUNT', fieldName), 10);
duplicateGroups.push({
value: value,
count: count
});
}

while (duplicateAggregate.next()) {
duplicateGroupCount++;
var fieldValue = duplicateAggregate.getValue(fieldName);
var countInGroup = duplicateAggregate.getAggregate('COUNT', fieldName);
duplicateJson[fieldValue] = countInGroup;
}
// --- Logging Results ---
if (duplicateGroups.length === 0) {
gs.info('✅ No duplicates found for "' + fieldName + '" on "' + tableName + '".');
return;
}

/***************************************/
/********** Print the results **********/
/***************************************/
gs.info('⚠️ Found ' + duplicateGroups.length + ' duplicate groups for "' + fieldName + '" on "' + tableName + '".');

// No duplicates found
if (Object.keys(duplicateJson).length === 0) {
gs.print('No duplicates found for field "' + fieldName + '" on table "' + tableName + '".');
return;
}
// --- Detailed Logging (optional for large datasets) ---
duplicateGroups.forEach(function (item, index) {
gs.info(
(index + 1) + '. Value: "' + item.value + '" → Occurrences: ' + item.count
);
});

// Duplicates were found
gs.print("Found " + duplicateGroupCount + " groups of duplicates:");

for (var key in duplicateJson) {
gs.print('Value "' + key + '" has ' + duplicateJson[key] + ' occurrences.');
// --- (Optional) Retrieve Record Sys IDs for deeper analysis ---
// Uncomment the section below if you need to list Sys IDs for each duplicate group
/*
duplicateGroups.forEach(function (dup) {
gs.info('--- Records with "' + fieldName + '" = "' + dup.value + '" ---');
var dupRec = new GlideRecord(tableName);
dupRec.addQuery(fieldName, dup.value);
dupRec.query();
while (dupRec.next()) {
gs.info(' Sys ID: ' + dupRec.getUniqueValue());
}
});
*/
}
}

})();
Loading