Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mapping Jira Custom Fields to Azure DevOps History #1017

Open
masteragi opened this issue Apr 11, 2024 · 11 comments
Open

Mapping Jira Custom Fields to Azure DevOps History #1017

masteragi opened this issue Apr 11, 2024 · 11 comments
Labels
enhancement New feature or request

Comments

@masteragi
Copy link

masteragi commented Apr 11, 2024

Dear Community,

I hope you are doing well! 🤓

In our Jira we have some custom fields which we don't want to replicate in the Azure DevOps instance but would like to keep as historical data. For this reason it would be sufficient to store their values (e.g. in the format "field name: field value") in the History/Discussion as separate comments.

I have defined the mappings like following:

      {
        "original-name": "Geplanter Aufwand h",
	"source": "customfield_16862",
        "target": "Microsoft.VSTS.Scheduling.Effort"
      },
      {
        "original-name": "Melder",
	"source": "customfield_10160",
        "target": "System.History"
      },
      {
        "original-name": "Fehlerart",
	"source": "customfield_16324",
        "target": "System.History"
      },
      {
        "original-name": "Fachbereich",
	"source": "customfield_10460",
        "target": "System.History"
      },
      {
        "original-name": "Anforderungsart",
	"source": "customfield_16324",
        "target": "System.History"
      },

However the tool seems to ignore the mappings (or at least I am not able to find the error) and I can see following entries in the log:
[I][15:13:38] Connecting to Jira...
[I][15:13:38] Retrieving Jira fields...
[I][15:13:38] Retrieving Jira link types...
[I][15:13:38] Export started. Selecting 1 items.
[I][15:13:38] Initializing Jira field mapping...
[W][15:13:42] Ignoring target mapping with key: 'System.History', because it is already configured.
[W][15:16:28] Ignoring target mapping with key: 'System.History', because it is already configured.
[W][15:16:32] Ignoring target mapping with key: 'System.History', because it is already configured.
[W][15:16:33] Ignoring target mapping with key: 'System.History', because it is already configured.

Should my approach work or I am doing something completely wrong?

Thank you in advance for your help. 🙏

@masteragi masteragi changed the title Mapping Jira Custom Fields to Azure DevOps History/Discussion Mapping Jira Custom Fields to Azure DevOps History Apr 11, 2024
@Alexander-Hjelm Alexander-Hjelm self-assigned this Apr 11, 2024
@Alexander-Hjelm
Copy link
Collaborator

However the tool seems to ignore the mappings (or at least I am not able to find the error) and I can see following entries in the log:
[I][15:13:38] Connecting to Jira...
[I][15:13:38] Retrieving Jira fields...
[I][15:13:38] Retrieving Jira link types...
[I][15:13:38] Export started. Selecting 1 items.
[I][15:13:38] Initializing Jira field mapping...
[W][15:13:42] Ignoring target mapping with key: 'System.History', because it is already configured.
[W][15:16:28] Ignoring target mapping with key: 'System.History', because it is already configured.
[W][15:16:32] Ignoring target mapping with key: 'System.History', because it is already configured.
[W][15:16:33] Ignoring target mapping with key: 'System.History', because it is already configured.

Migrating multiple source fields to the same target fields is not supported. I will mark this as a feature request.

:)

@Alexander-Hjelm Alexander-Hjelm added enhancement New feature or request and removed support labels Apr 11, 2024
@Alexander-Hjelm Alexander-Hjelm removed their assignment Apr 11, 2024
@masteragi
Copy link
Author

masteragi commented Apr 11, 2024

OK, thank you. I need to think of a workaround here... Is a change required only on the export side or also on the import side?

@Alexander-Hjelm
Copy link
Collaborator

Alexander-Hjelm commented Apr 11, 2024

Pretty sure it would only be on the export side!

Sure! A temporary but rather annoying workaround would be to set the target field as a temporary token and then search-replace all such tokens with "System.History".

For example I could see you doing something like this:

      {
        "original-name": "Melder",
	"source": "customfield_10160",
        "target": "UniqueReplace1"
      },
      {
        "original-name": "Fehlerart",
	"source": "customfield_16324",
        "target": "UniqueReplace2"
      },
      {
        "original-name": "Fachbereich",
	"source": "customfield_10460",
        "target": "UniqueReplace3"
      },
      {
        "original-name": "Anforderungsart",
	"source": "customfield_16324",
        "target": "UniqueReplace4"
      },

Then after the Export, go inside an IDE like VS Code and do a folderwide search-replace in your folder (with regex enabled):

  • UniqueReplace\d+ -> System.History

Finally go ahead with the Import.

This should take care of it in theory, but I have never tried before, so I cannot say with 100% certainty. You are welcome to try though!

@masteragi
Copy link
Author

masteragi commented Apr 11, 2024

First thing I tried is to add another History entry in the exported Jira json file, like this:

        {
          "ReferenceName": "System.History",
          "Value": "Value 1"
        },
	{
          "ReferenceName": "System.History",
          "Value": "Value 2"
        },

And after the import just "Value 2" gets written in the Azure DevOps work item.

It doesn't seem to be a big code-change on the export side so I might take a swing at it but I haven't checked the import side yet...

@Alexander-Hjelm
Copy link
Collaborator

Correct, each update would need to be in it's own separate revisions! It can only write to any single Work Item Field once in each revision.

@masteragi
Copy link
Author

Correct, each update would need to be in it's own separate revisions! It can only write to any single Work Item Field once in each revision.

I see... In this case it is a bigger code-change, but I can only say for sure after I understand the code bahind revisions. 😄

@masteragi
Copy link
Author

@Alexander-Hjelm
I am trying to make this work and have made following changes in the code to make this work:

Added the option to mapping configuration in the config.json file (additional "customFieldsList" with the list of my custom fields, mapped to "System.History" with the new "MapMultiple" mapper):

{
"source": "multiple",
"customFieldsList": [
	"Melder",
	"Fehlerart",
	"Fachbereich",
	"Anforderungsart"
],
"target": "System.History",
"mapper": "MapMultiple"
}

Then I added the "customFieldsList" as a property in the Field class:

[JsonProperty("customFieldsList")]
public List<string> CustomFieldsList { get; set; }

Added the handling of the "MapMultiple" mapper in JiraMapper::InitializeFieldMappings:

case "MapMultiple":
    value = r => FieldMapperUtils.MapMultipleValues(r, item.CustomFieldsList, item.Target, _config, _jiraProvider);
    break;

Implemented the concatenation of custom field "name: value" pairs in the MapMultipleValues method:

public static (bool, object) MapMultipleValues(JiraRevision r, List<string> customFields, string targetItem, ConfigJson config, IJiraProvider jiraProvider)
{
    if (r == null)
        throw new ArgumentNullException(nameof(r));

    if (config == null)
        throw new ArgumentNullException(nameof(config));

    var mappedValues = string.Empty;
    var customFieldId = string.Empty;
    var customFieldValue = string.Empty;
    //go through the list of source fields
    foreach (var customFieldName in customFields)
    {
        //fetch the Jira Custom Field ID via the Jira Rest API
        customFieldId = jiraProvider.GetCustomId(customFieldName);
        //fetch the field value from the list of fields in the Jira Revision
        customFieldValue = r.GetFieldValue(customFieldId);

        //add an entry only if the custom Field has a value in the Jira Revision
        if (customFieldValue != null)
        {
            //add a breakpoint before every new key-value pair (except for the first one)
            mappedValues += (mappedValues != string.Empty) ? "</br>" : "";
            //put name and value together as "name: value"
            mappedValues += customFieldName + ": " + customFieldValue;
        }
    }

    return (true, mappedValues);
}

I've probably butchered the code a lot and left some loose ends unhandled but this is as far as I got so far to get multiple custom fields mapped to the System.History field.

What I am experiencing at the moment is that all the original Jira comments get overwritten by those same custom field values:
image

Before I go and lose a lot of time debugging the code, do you know where I got it wrong and where I might need to adjust the code (or maybe add some additional forks/checks)?

Thank you in advance for your support. 🙏

Cheers,
Marko

@Alexander-Hjelm
Copy link
Collaborator

My best guess is that it would be the JiraMapper blocking the old field mapping for System.History because it contains the same target property as your new field mapping. This will cause one to override the other. Currently only one field mapping is allowed per field per Work Item Type.

foreach (var wit in currentWorkItemTypes)
{
try
{
if (wit == "All" || wit == "Common")
{
commonFields.Add(item.Target, value);
}
else
{
// If we haven't mapped the Type then we probably want to ignore the field
if (typeFields.TryGetValue(wit, out FieldMapping<JiraRevision> fm))
{
fm.Add(item.Target, value);
}
else
{
Logger.Log(LogLevel.Warning, $"No target type '{wit}' is set, field {item.Source} cannot be mapped.");
}
}
}
catch (Exception)
{
Logger.Log(LogLevel.Warning, $"Ignoring target mapping with key: '{item.Target}', because it is already configured.");
}
}

@masteragi
Copy link
Author

masteragi commented May 4, 2024

Thank you, @Alexander-Hjelm. Yes, that's what I first thought as well - that it is caused by the limitation of the Dictionary to hold one mapping for each key (target work item type). What would you recommend as the easiest workaround?

@Alexander-Hjelm
Copy link
Collaborator

I can think of a simple workaround which would be best implemented as a post-migration task. I would have mapped Comment to let's say "target": "System.History" and your new aggregated field to a unique token like "target": "System.History1". You can then do a folder-wide search replace in the workspace folder using e.g. an IDE or a script. You would replace all ocurences of System.History1" with "System.History". I think that this should adequately solve your problem, unless there are any instances where the the Jira Comment and your aggregated field are updated on the same revision, which I assume is not the case.

We would probably need to do a major redesign of the JiraMapper component if we want to properly support this scenario, plus think about all possible conflict scenarios. I am thus leaving this issue as a Feature Request for when we can commit to it! :)

@masteragi
Copy link
Author

That's a good idea and easy to implement - as one additional line of code before the WiItem gets serialized to the JSON file. Will do that... Thank you so much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants