[PackageTool.ps1] Conversion from JSON failed with error: Unterminated string. #197

LanceMcCarthy · 2020-08-11T23:06:20Z

There's an error running Update-ApplicationSubmission -ReplacePackages regarding notes for certification.

##[error]ConvertFrom-Json : Conversion from JSON failed with error: Unterminated string. Expected delimiter: ". Path 'appSubmission.notesForCertification', line 1, position 2059.
At D:\a\MediaFileManager\MediaFileManager\broker\StoreBroker\PackageTool.ps1:3276 char:22

I am not making any custom package setup; just a clone of previous submission and uploading new packages with the -ReplacePackages flag.

Script

Here's my Powershell script, running in a GitHub Action on a windows machine. Disregard any envVars you dont recognize, they're set earlier on the VM.

If you want to see the rest of it, it is public here MediaFileManager CD.yaml.

$uploadPackage = "D:\a\MediaFileManager\MediaFileManager\src\MediaFileManager\PackageProject\StoreUploadPackages\PackageProject_" + $env:UWP_VERSION + "_" + $env:UwpBundlePlatform + "_bundle.appxupload"
$username = $env:PartnerCenterClientId
$password = ConvertTo-SecureString $env:PartnerCenterClientSecret -AsPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential ($username, $password)

# install StoreBroker
Set-ExecutionPolicy RemoteSigned -Force
git clone https://github.com/Microsoft/StoreBroker.git 'D:\a\MediaFileManager\MediaFileManager\broker\'
Import-Module -Force 'D:\a\MediaFileManager\MediaFileManager\broker\StoreBroker'

# Authenticating with DevCenter
Set-StoreBrokerAuthentication -TenantId $env:PartnerCenterTenantId -Credential $cred

# Generating PDP
$pdpFolderPath = [System.Environment]::GetFolderPath('Desktop')
$pdpFileName = 'Pdp.xml' 
D:\a\MediaFileManager\MediaFileManager\broker\\Extensions\ConvertFrom-ExistingSubmission.ps1 -AppId $env:PartnerCenterStoreId -Release $env:UWP_VERSION -PdpFileName $pdpFileName -OutPath $pdpFolderPath

# Generating SBConfig file
$brokerConfigFilePath = Join-Path -Path ([System.Environment]::GetFolderPath('Desktop')) -ChildPath 'SBConfig.json'
New-StoreBrokerConfigFile -AppId $env:PartnerCenterStoreId -Path $brokerConfigFilePath

# Generate submission Data package
$submissionDataFilePath = Join-Path -Path ([System.Environment]::GetFolderPath('Desktop')) -ChildPath 'submission.json'
New-SubmissionPackage -ConfigPath (Join-Path -Path ([System.Environment]::GetFolderPath('Desktop')) -ChildPath 'SBConfig.json') -PDPRootPath ([System.Environment]::GetFolderPath('Desktop')) -PDPInclude 'Pdp.xml' -ImagesRootPath 'D:\a\MediaFileManager\MediaFileManager\src\MediaFileManager\PackageProject\Images' -AppxPath $uploadPackage -OutPath $pdpFolderPath -OutName $submissionDataFilePath

# submitting app
Update-ApplicationSubmission -ReplacePackages -AppId $env:PartnerCenterStoreId -SubmissionDataPath $submissionDataFilePath -PackagePath $uploadPackage -AutoCommit -Force

Output

Here is the output of the run. To keep it short, I removed the duplicate "I'm busy" updates.

Cloning into 'D:\a\MediaFileManager\MediaFileManager\broker'...

|  Elapsed: 2 second(s) [Getting access token]
DONE - Operation took 2 second(s) [Getting access token]

/  Elapsed: 2 second(s) [Getting data for AppId: 9PD3JFK7W5MB]
DONE - Operation took 2 second(s) [Getting data for AppId: 9PD3JFK7W5MB]
2020-08-11 22:34:07 : runneradmin : Telemetry is currently enabled.  It can be disabled by setting "$global:SBDisableTelemetry = $true". Refer to USAGE.md#telemetry for more information.


-  Elapsed: 4 second(s) [Retrieving nuget package: Microsoft.Bcl.Async]
DONE - Operation took 4 second(s) [Retrieving nuget package: Microsoft.Bcl.Async]
2020-08-11 22:34:13 : runneradmin : To avoid this download delay in the future, copy the following file:
  [C:\Users\RUNNER~1\AppData\Local\Temp\e229bc7e-692a-44fd-8ba5-82b48e7b62d7\Microsoft.Bcl.Async.1.0.168\lib\net40\Microsoft.Threading.Tasks.dll]
either to:
  [D:\a\MediaFileManager\MediaFileManager\broker\StoreBroker]
or to:
  a directory of your choosing, and save that directory path to $SBAlternateAssemblyDir


\  Elapsed: 2 second(s) [Retrieving nuget package: Microsoft.Diagnostics.Tracing.EventSource.Redist]
DONE - Operation took 2 second(s) [Retrieving nuget package: Microsoft.Diagnostics.Tracing.EventSource.Redist]
2020-08-11 22:34:15 : runneradmin : To avoid this download delay in the future, copy the following file:
  [C:\Users\RUNNER~1\AppData\Local\Temp\e229bc7e-692a-44fd-8ba5-82b48e7b62d7\Microsoft.Diagnostics.Tracing.EventSource.Redist.1.1.24\lib\net35\Microsoft.Diagnostics.Tracing.EventSource.dll]
either to:
  [D:\a\MediaFileManager\MediaFileManager\broker\StoreBroker]
or to:
  a directory of your choosing, and save that directory path to $SBAlternateAssemblyDir

-  Elapsed: 2 second(s) [Retrieving nuget package: Microsoft.ApplicationInsights]
DONE - Operation took 2 second(s) [Retrieving nuget package: Microsoft.ApplicationInsights]
2020-08-11 22:34:18 : runneradmin : To avoid this download delay in the future, copy the following file:
  [C:\Users\RUNNER~1\AppData\Local\Temp\e229bc7e-692a-44fd-8ba5-82b48e7b62d7\Microsoft.ApplicationInsights.2.0.1\lib\net45\Microsoft.ApplicationInsights.dll]
either to:
  [D:\a\MediaFileManager\MediaFileManager\broker\StoreBroker]
or to:
  a directory of your choosing, and save that directory path to $SBAlternateAssemblyDir

/  Elapsed: 14 second(s) [Getting data for AppId: 9PD3JFK7W5MB SubmissionId: 1152921505688863279]
DONE - Operation took 14 second(s) [Getting data for AppId: 9PD3JFK7W5MB SubmissionId: 1152921505688863279]
2020-08-11 22:34:36 : runneradmin : PDP's have been created here: C:\Users\runneradmin\Desktop
2020-08-11 22:34:36 : runneradmin : You now need to find all of your assets and place them here: <ImagesRootPath>\2020.811.75.0\<langcode>\...
  where <ImagesRootPath> is the path defined in your config file,
  and <langcode> is the same langcode for the directory of the PDP file referencing those assets.
2020-08-11 22:34:36 : runneradmin : Every language that has a PDP references the following assets:
	MFM1.png
	MFM2.png
	MFM3.png


/  Elapsed: 1 second(s) [Getting data for AppId: 9PD3JFK7W5MB]
DONE - Operation took 2 second(s) [Getting data for AppId: 9PD3JFK7W5MB]

/  Elapsed: 6 second(s) [Getting data for AppId: 9PD3JFK7W5MB SubmissionId: 1152921505688863279]
DONE - Operation took 6 second(s) [Getting data for AppId: 9PD3JFK7W5MB SubmissionId: 1152921505688863279]
Write-Error: D:\a\MediaFileManager\MediaFileManager\broker\StoreBroker\PackageTool.ps1:3725
Line |
3725 |          Write-Log -Exception $_ -Level Error
     |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | 2020-08-11 22:34:48 : runneradmin :  ConvertFrom-Json:
     | D:\a\MediaFileManager\MediaFileManager\broker\StoreBroker\PackageTool.ps1:3276 Line |
     | 3276 |      return ($lines | ConvertFrom-Json)      |                     
     | ~~~~~~~~~~~~~~~~      | Conversion from JSON failed with error: Unterminated string.
     | Expected delimiter: ". Path      | 'appSubmission.notesForCertification', line 1, position
     | 2059. 

##[error]ConvertFrom-Json : Conversion from JSON failed with error: Unterminated string. Expected delimiter: ". Path 'appSubmission.notesForCertification', line 1, position 2059.
At D:\a\MediaFileManager\MediaFileManager\broker\StoreBroker\PackageTool.ps1:3276 char:22
+     return ($lines | ConvertFrom-Json)
+                      ~~~~~~~~~~~~~~~~
+ CategoryInfo          : NotSpecified: (:) [ConvertFrom-Json], ArgumentException
+ FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.Commands.ConvertFromJsonCommand

However, if you would rather see the entire step's output, it is also public here. Media File Manager - Run 204743631 - Fail.

Be sure to expand the arrows:

The text was updated successfully, but these errors were encountered:

HowardWolosky · 2020-08-13T05:35:40Z

Thanks for the report, @LanceMcCarthy.

Would you be willing to share out the config file that was generated at $brokerConfigFilePath? The issue is with the notesForCertification that was injected into there, and so I'd like to understand what's going on there.

As an aside, you can completely skip the "Generating PDP" step. Just don't include the PDPRootPath parameter for New-SubmissionPackage. Since you're just doing package updates, there's no need to include any of the metadata. The metadata is only useful when you're actually trying to update it.

HowardWolosky · 2020-08-13T05:46:33Z

Note for maintainers...relevant code for inspection:

StoreBroker/StoreBroker/PackageTool.ps1

Lines 363 to 365 in 0fbb6b9

    
           # NOTES FOR CERTIFICATION 
        
           $notesForCertification = Get-EscapedJsonValue -Value $sub.notesForCertification 
        
           $updated = $updated -replace '"notesForCertification": ""', "`"notesForCertification`": `"$notesForCertification`""

StoreBroker/StoreBroker/Helpers.ps1

Lines 359 to 372 in 0fbb6b9

    
           # The syntax of -replace is a bit confusing, so it's worth a note here. 
        
           # The first parameter is a regular expression match pattern.  The second parameter is a replacement string. 
        
           # So, when we try to do "-replace '\\', '\\", that's matching a single backslash (which has to be 
        
           # escaped within the match regular expression as a double-backslash), and replacing it with a 
        
           # string containing literally two backslashes. 
        
           # (And as a reminder, PowerShell's escape character is actually the backtick (`) and not backslash (\).) 
        
           # \, ", <tab> 
        
           $escaped = $Value -replace '\\', '\\' -replace '"', '\"' -replace '\t', '\t' 
        
           # Now normalize actual CR's and LF's with their control codes.  We'll ensure all variations are uniformly formatted as \r\n 
        
           $escaped = $escaped -replace '\r\n', '\r\n' -replace '\r', '\r\n' -replace '\n', '\r\n' 
        
           return $escaped

HowardWolosky · 2020-08-13T05:55:28Z

It's not clear to me why this isn't calling ConvertTo-Json on the result of Get-EscapedJsonValue (or instead of calling that entirely).

LanceMcCarthy · 2020-08-13T14:02:14Z

Thanks @HowardWolosky

I made the suggested changes and do not generate a PDP, but the failure still occurs. See https://github.com/LanceMcCarthy/MediaFileManager/runs/980649893?check_suite_focus=true

In that same run, I did attempt to output the SBconfig.json contents so I could copy/paste it, but I'm not sure it is even reaching that step because I'm not seeing the output. I will try to upload the file to a blob instead.

Metadata

You're right that I dont have any changes to the metadata in this current iteration, but I plan on building it in because that repo is intended as a "gold standard" for dev community to reference for when they want to see a real-world GitHub Action for deploying a WPF/UWP app to both side-load distributions.

The step prior to the StoreBroker is my own custom GH Action that uploads the .appinstaller (and associated packages) to Azure Storage blob. The Store Broker step is the next phase that distributes to the Store (I'll add a flights branch when this works).

Side Note - Building a new Broker

I am also trying to understand exactly how the API works and what it expects. I'm about 80% done with developing a TypeScript version of the Store Broker that is on npm. Here are the tasks https://github.com/LanceMcCarthy/PartnerCenterBroker/blob/main/src/index.ts

I have studied the API docs and understand how complicated the payloads can get (the number of nested objects gets gnarly at times https://github.com/LanceMcCarthy/PartnerCenterBroker/blob/main/src/interfaces.ts) so I like the idea of cloning a previous submission and have done that here https://github.com/LanceMcCarthy/PartnerCenterBroker/blob/36901851ace5126d562ed619a6f3c2446831aa85/src/helpers.ts#L25-L47 .

I aim to provide developers with options, no matter what their environment is (at least until PS Core works 100%)

HowardWolosky · 2020-08-13T15:29:45Z

I made the suggested changes and do not generate a PDP, but the failure still occurs. See https://github.com/LanceMcCarthy/MediaFileManager/runs/980649893?check_suite_focus=true

That's what I would have expected. That suggestion wasn't to fix the problem, I was simply trying to simplify your script; as you indicated below however, that goes against the goal you were trying to achieve.

I am also trying to understand exactly how the API works and what it expects. I'm about 80% done with developing a TypeScript version of the Store Broker that is on npm. Here are the tasks https://github.com/LanceMcCarthy/PartnerCenterBroker/blob/main/src/index.ts

There is an Azure DevOps task written in TypeScript that you could look at for reference (or use).

Something for you to consider: As you might have noticed with this project, there has been minimal activity on the master (v1) branch. The same can't be said of StoreBroker v2. StoreBroker v2 is built on top of the next version of the Windows Submission API (v2) which is substantially different than v1, and intended to eventually replace it. Unfortunately, v2 isn't publicly available yet, and we still don't know what that timeframe would be for its public release. StoreBroker v2 has been in-use internally now for over 1.5 years, and there's a new version of the Azure DevOps task that is built on top of StoreBroker v2, but that too is only available internally until such time as the new API is made widely available publicly.

LanceMcCarthy · 2020-08-13T15:48:45Z

That's what I would have expected. That suggestion wasn't to fix the problem, I was simply trying to simplify your script; as you indicated below however, that goes against the goal you were trying to achieve.

Understood, I was just sharing the end result. Thank you for the suggestion as it saves another possible point of failure.

There is an Azure DevOps task written in TypeScript that you could look at for reference (or use).

Thank you! Although it seems I should not put too much effort into using the v1 endpoints, but if v2 endpoints aren't public yet my options are limited and might as well keep moving forward with it. I can then do a v2 of my package later.

I reviewed the code and I do already have everything they implemented, and more :) What it did show me is that I can simplify the metadata operations and just do the clone+add package and forget about giving the user granular capability to change metadata.

Something for you to consider: As you might have noticed with this project, there has been minimal activity on the master (v1) branch. The same can't be said of StoreBroker v2. StoreBroker v2 is built on top of the next version of the Windows Submission API (v2) which is substantially different than v1, and intended to eventually replace it. Unfortunately, v2 isn't publicly available yet, and we still don't know what that timeframe would be for its public release. StoreBroker v2 has been in-use internally now for over 1.5 years, and there's a new version of the Azure DevOps task that is built on top of StoreBroker v2, but that too is only available internally until such time as the new API is made widely available publicly.

In my initial research, I did find the Partner Center APIs which heavily use the MS Graph, but do not see any endpoints for App submission :( This is why my current efforts are still using manage.devcenter.microsoft.com/v1/ instead

Is it going to be manage.devcenter.microsoft.com/v2/ or is it going to use https://api.partnercenter.microsoft.com endpoints?

NDA Help

Although it doesn't sound like you need the testing for v2 of the submission API, but if you do, we can use my MVP NDA for my private Actions 😉 (I would help write day1 blog (similar to my Windows 10 on ARM: Day-One Developer Experience post). I am also under several other TAP programs and have pre-approved clearance for future TAPs/NDAs.

This would not only help me, but I can also have working examples for the community ready-to-go.

LanceMcCarthy · 2020-08-13T23:10:09Z

@HowardWolosky Okay, I tried making this as simple and straightforward as possible by using all full strings where possible and broke out the steps to different stages to catch where it is failing

The Store Broker now seems to hang when trying to generate the SBConfig file (for 26 minutes until the VM quits)

************************* GETS STUCK HERE *******************************
# ---- Create config file  
New-StoreBrokerConfigFile -AppId $env:PartnerCenterStoreId -Path 'D:\a\MediaFileManager\MediaFileManager\SBTemp\SBConfig.json'

Here are the relevant steps, you can see the offender in Part 2

 # ************ PART 1  ************ # 
- name: Install Store Broker (SB)
  uses: Amadevus/pwsh-script@v1
  id: install-store-broker
  with:
    script: |
      # ---- Install StoreBroker
      Set-ExecutionPolicy RemoteSigned -Force
      git clone https://github.com/Microsoft/StoreBroker.git 'D:\a\MediaFileManager\MediaFileManager\SBGitRoot\'

# ************ PART 2 ************ #
- name: Authenticate SB and Generate SBConfig.json
  uses: Amadevus/pwsh-script@v1
  id: authenticate-store-brokwer
  with:
    script: |
      $username = $env:PartnerCenterClientId
      $password = ConvertTo-SecureString $env:PartnerCenterClientSecret -AsPlainText -Force
      $cred = New-Object System.Management.Automation.PSCredential ($username, $password)
      Import-Module -Force 'D:\a\MediaFileManager\MediaFileManager\SBGitRoot\StoreBroker'

      # ---- Authenticating with Service Principal Credentials
      Set-StoreBrokerAuthentication -TenantId $env:PartnerCenterTenantId -Credential $cred
      
      # ---- Making a new directory for Store Broker generated files
      New-Item -Path 'D:\a\MediaFileManager\MediaFileManager\SBTemp' -ItemType Directory

******** GETS STUCK HERE (for 26 minutes) ****************
      # ---- Create config file  
      New-StoreBrokerConfigFile -AppId $env:PartnerCenterStoreId -Path 'D:\a\MediaFileManager\MediaFileManager\SBTemp\SBConfig.json'


# ************ PART 3 (debug step to upload the sbconfig.json file to my blob) ************
- name: Uploading SBconfig.json to Azure Blob (temp)
  id: blob-upload-sbconfig
  uses: LanceMcCarthy/Action-AzureBlobUpload@v1.3
  with:
    connection_string: ${{ secrets.AZURE_DVLUP_BLOB_CONNECTION_STRING }}
    container_name: general-app-files
    source_folder: D:\a\MediaFileManager\MediaFileManager\SBTemp\
    destination_folder: SBTemp

# ************ PART 4 do the actual submission ************
- name: Store Broker (Upload the Store build to Microsoft Partner Center)
  uses: Amadevus/pwsh-script@v1
  id: store-broker-submit-app
  with:
    script: |
       $uploadPackage = "D:\a\MediaFileManager\MediaFileManager\src\MediaFileManager\PackageProject\StoreUploadPackages\PackageProject_" + 
$env:UWP_VERSION + "_" + $env:UwpBundlePlatform + "_bundle.appxupload"
        $username = $env:PartnerCenterClientId
        $password = ConvertTo-SecureString $env:PartnerCenterClientSecret -AsPlainText -Force
        $cred = New-Object System.Management.Automation.PSCredential ($username, $password)
          
        Import-Module -Force 'D:\a\MediaFileManager\MediaFileManager\SBGitRoot\StoreBroker'
          
          # ---- Authenticating with Service Principal Credentials
        Set-StoreBrokerAuthentication -TenantId $env:PartnerCenterTenantId -Credential $cred
    
        # ---- Create submissionPackage
        New-SubmissionPackage -ConfigPath 'D:\a\MediaFileManager\MediaFileManager\SBTemp\submission.json' -ImagesRootPath 'D:\a\MediaFileManager\MediaFileManager\src\MediaFileManager\PackageProject\Images' -AppxPath $uploadPackage -OutPath 'D:\a\MediaFileManager\MediaFileManager\SBTemp\' -OutName 'submission.json'
      
        # ---- Submitting app, we only want to close last submission and just update package
        Update-ApplicationSubmission -ReplacePackages -AppId $env:PartnerCenterStoreId -SubmissionDataPath 'D:\a\MediaFileManager\MediaFileManager\SBTemp\submission.json' -PackagePath $uploadPackage -AutoCommit -Force

Output

Here is the out put from step 2 (with repeated lines removed)

- Elapsed: 0 second(s) [Getting access token]
**** REDACTED ****
DONE - Operation took 2 second(s) [Getting access token]

Elapsed: 0 second(s) [Getting data for AppId: 9PD3JFK7W5MB]
**** REDACTED ****
DONE - Operation took 3 second(s) [Getting data for AppId: 9PD3JFK7W5MB]
2020-08-13 22:19:15 : runneradmin : Telemetry is currently enabled.  It can be disabled by setting "$global:SBDisableTelemetry = $true". Refer to USAGE.md#telemetry for more information.


-  Elapsed: 0 second(s) [Retrieving nuget package: Microsoft.Bcl.Async]
**** REDACTED ****
DONE - Operation took 4 second(s) [Retrieving nuget package: Microsoft.Bcl.Async]
2020-08-13 22:19:21 : runneradmin : To avoid this download delay in the future, copy the following file:
  [C:\Users\RUNNER~1\AppData\Local\Temp\ecf83896-abd9-4c21-b5eb-ea9e170df8ff\Microsoft.Bcl.Async.1.0.168\lib\net40\Microsoft.Threading.Tasks.dll]
either to:
  [D:\a\MediaFileManager\MediaFileManager\SBGitRoot\StoreBroker]
or to:
  a directory of your choosing, and save that directory path to $SBAlternateAssemblyDir
  

|  Elapsed: 0 second(s) [Retrieving nuget package: Microsoft.Diagnostics.Tracing.EventSource.Redist]
**** REDACTED ****
DONE - Operation took 2 second(s) [Retrieving nuget package: Microsoft.Diagnostics.Tracing.EventSource.Redist]
2020-08-13 22:19:23 : runneradmin : To avoid this download delay in the future, copy the following file:
  [C:\Users\RUNNER~1\AppData\Local\Temp\ecf83896-abd9-4c21-b5eb-ea9e170df8ff\Microsoft.Diagnostics.Tracing.EventSource.Redist.1.1.24\lib\net35\Microsoft.Diagnostics.Tracing.EventSource.dll]
either to:
  [D:\a\MediaFileManager\MediaFileManager\SBGitRoot\StoreBroker]
or to:
  a directory of your choosing, and save that directory path to $SBAlternateAssemblyDir
  

|  Elapsed: 2 second(s) [Retrieving nuget package: Microsoft.ApplicationInsights]
**** REDACTED ****
DONE - Operation took 2 second(s) [Retrieving nuget package: Microsoft.ApplicationInsights]
2020-08-13 22:19:26 : runneradmin : To avoid this download delay in the future, copy the following file:
  [C:\Users\RUNNER~1\AppData\Local\Temp\ecf83896-abd9-4c21-b5eb-ea9e170df8ff\Microsoft.ApplicationInsights.2.0.1\lib\net45\Microsoft.ApplicationInsights.dll]
either to:
  [D:\a\MediaFileManager\MediaFileManager\SBGitRoot\StoreBroker]
or to:
  a directory of your choosing, and save that directory path to $SBAlternateAssemblyDir
  
 -  Elapsed: 0 second(s) [Getting data for AppId: 9PD3JFK7W5MB SubmissionId: 1152921505688863279]
**** REDACTED ****
DONE - Operation took 9 second(s) [Getting data for AppId: 9PD3JFK7W5MB SubmissionId: 1152921505688863279]

Here's the run if you want to see the whole thing https://github.com/LanceMcCarthy/MediaFileManager/runs/982539639?check_suite_focus=true

HowardWolosky · 2020-08-13T23:42:05Z

Thanks @LanceMcCarthy. I think the easiest way to debug what might be going on is to understand your submission that it's trying to convert into a config file. Can you run the following?

$appId = '9PD3JFK7W5MB'
$app = Get-Application -AppId $appId
$sub = Get-ApplicationSubmission -AppId $appId -SubmissionId $($app.lastPublishedApplicationSubmission.id)
$subJsonFilePath = New-TemporaryFile
ConvertTo-Json -InputObject $sub -Depth 20 | Out-File -FilePath $subJsonFilePath
Write-Information -MessageData "Please upload $($subJsonFilePath.FullName) to a new gist at https://gist.github.com/LanceMcCarthy and then reference that gist in microsoft/StoreBroker#197" -InformationAction Continue

That would allow us to reproduce the behavior of Get-StoreBrokerConfigFileContentForAppId without needing access to your developer account. Something weird is going on there, as that command should take ~5 seconds to complete.

As an aside, the general usage pattern for StoreBroker would be to generate the config file once and check that in with your source code. Same with the PDP file content. The intention of the config file is two fold:

Let you define common parameters to New-SubmissionPackage so that they don't need to be specified at the command-line
Allow you to alter app-wide settings that have no command-line component (like changing the pricing, category, etc...).

If you're always generating it fresh, you're not getting any of those benefits. That's fine if that functionality isn't desired, but wanted to make sure that was an explicit decision on your part.

And as for the PDP content, like I said before, you'd convert an existing published submission into the individual components (XML and image files) once and then store them with your code, and then update the listing metadata in those files locally when you've made changes to the app that you're intending to publish. Then you'd generate a submission package with the updated PDP content and make use of -UpdateListings when calling Update-AppSubmission (with or without -AddPackages/-ReplacePackages).

You may just want to add a comment to that effect (that those command are intended to only be run once when initially configuring StoreBroker).

LanceMcCarthy · 2020-08-14T04:10:50Z

It definitely a problem inside the submission data being downloaded from DevCenter.

See the latest runs output, even after trying to copy out the data https://github.com/LanceMcCarthy/MediaFileManager/actions/runs/208040011

I'm going to try this on my normal pc and get real files I can include with source control. I'll let you know how that goes tomorrow

HowardWolosky · 2020-08-14T04:31:00Z

No, that error indicates a poorly formatted config file. That means it already successfully downloaded the data and successfully created a config file. Now it's trying to read and covert the config file back into an object during New-SubmissionPackage, and it's failing at that conversion step:

StoreBroker/StoreBroker/PackageTool.ps1

Lines 3274 to 3276 in 0fbb6b9

    
           $lines = (Get-Content -Path $ConfigPath -Encoding UTF8 | Remove-Comment) -join '' 
        
           return ($lines | ConvertFrom-Json)

Like I said, if you just run the commands in my previous response, I'll be able to see your submission and understand what's going on when the config file is getting created and why it's malformed.

LanceMcCarthy · 2020-08-14T15:39:12Z

@HowardWolosky I just couldn't get GitHub actions to cooperate and give me the files, so I did everything locally on my dev machine instead.

You can get the generated file here https://dvlup.blob.core.windows.net/ci-cd/broker-output/tmp828.tmp

LanceMcCarthy · 2020-08-14T17:31:42Z

@HowardWolosky Discovered the issue!

I think I found the culprit. In the notes for certification, I see a < character that is encoded in one place, but not the other.

SubmissionData.json

Notice the \u003c character in the release notes.

SBConfig.json

If you look at the sbconfig.json, the < character is not encoded.

Source Control-based Config

As you recommended, I checked in the config.json and submission.json files into my source. I also moved my Store Broker script into its own ps1 file. https://github.com/LanceMcCarthy/MediaFileManager/tree/main/.scripts.

My question now is: If I use the checked-in SubmissionData.json file, how does Update-ApplicationSubmission know to use the new APPX? Does it automatically change the packages list when I use -ReplacePackages flag?

For example:

# Generated (STILL FAILS b/c of BAD CHAR)
$newSubmissionDataPath = Join-Path -Path $sbTempFolderPath -ChildPath 'submissionData.json'

# Checked in with source control and has manually fixed notes
$originalSubmissionDataFilePath = 'D:\a\MediaFileManager\MediaFileManager\.scripts\submissionData.json'

$packagePath = 'D:\a\MediaFileManager\MediaFileManager\src\MediaFileManager\PackageProject\StoreUploadPackages\StoreUploadPackages.zip'

Update-ApplicationSubmission -ReplacePackages -AppId $env:PartnerCenterStoreId -SubmissionDataPath $originalSubmissionDataFilePath -PackagePath $packagePath -AutoCommit -Force

I did try using $originalSubmissionDataFilePath for the submission, but it failed as well with the same error. I'm not sure what went wrong there.

Next Steps

After multiple attempts to get past this, I think I'm going to submit a new version via DevCenter and clean out the cert notes After that is published, I can generate a new submissionData file for future use.

PS - Despite these issues in GitHub Actions (PS core?) StoreBroker is an absolute pleasure in user-mode locally.

HowardWolosky · 2020-08-14T18:59:14Z

Thanks Lance. I have a fix out now (#198) that fixes that issue and another issue that would have been there due to the URL's in certification notes.

My question now is: If I use the checked-in SubmissionData.json file, how does Update-ApplicationSubmission know to use the new APPX? Does it automatically change the packages list when I use -ReplacePackages flag?

Don't check-in the submission data. Only check-in the config file. You should generate the "submission package" (which outputs a .json and .zip file) each time you want to do a submission. You reference the .json and .zip file in Update-ApplicationSubmission, and if you specify -ReplacePackages, it will replace the packages on PartnerCenter with the ones that are defined in the .json file. You may want to review USAGE.md again for a bit more clarity on this step.

PS - Despite these issues in GitHub Actions (PS core?) StoreBroker is an absolute pleasure in user-mode locally.

Thanks for the feedback! If you like StoreBroker, you may be interested in looking at PowerShellForGitHub as well. It's next major release (0.15.0) should be happening this weekend and has a ton of improvements (including full pipeline support).

LanceMcCarthy · 2020-08-14T19:23:33Z

I was just looking closely at the PR with the fix, good stuff! Now I understand why there were so many comments in the config.json file :D

You should generate the "submission package" (which outputs a .json and .zip file) each time you want to do a submission. You reference the .json and .zip file in Update-ApplicationSubmission, and if you specify -ReplacePackages, it will replace the packages on PartnerCenter with the ones that are defined in the .json file.

This is one of the key places that I was misunderstanding, I have read USAGE.md several times, but it gets a little confusing about where the ZIP file comes from. The submission data path is clear because you set the name with -OutName, but I'm still not sure what the file name is for the ZIP?

To illustrate the question using the snippet below, is the generated ZIP file path MyFolder\package.zip?

New-SubmissionPackage -OutPath 'MyFolder\' -OutName 'submission.json'

In any case, this thread is not the right place to discuss this, I'll keep trying and open a new Issue if I need to. Here's what I settled on for upcoming builds:

# Temp folder for all StoreBroker artifacts
$sbTempFolderPath = New-Item -Type Directory -Force -Path (Join-Path -Path 'D:\a\MediaFileManager\MediaFileManager\' -ChildPath 'SBTemp')

# Use the committed config file to generate a new submission.json file
$configFilePath = 'D:\a\MediaFileManager\MediaFileManager\.scripts\storeBrokerConfig.json'
New-SubmissionPackage -ConfigPath $configFilePath -AppxPath $appxUploadFilePath -OutPath $sbTempFolderPath -OutName 'submission.json'

# Identify the ZIP and submission.json file paths
$submissionDataPath = Join-Path -Path $sbTempFolderPath -ChildPath 'submission.json'
$submissionPackagePath = Join-Path -Path $sbTempFolderPath -ChildPath 'package.zip'

Update-ApplicationSubmission -ReplacePackages -AppId $appStoreId -SubmissionDataPath $submissionDataPath -PackagePath $submissionPackagePath -AutoCommit -Force

Maybe it will be useful to add a new doc page for complete examples (instead of partial snippets) so the reader can see the use in entirety? This usually helps make everything click (after being taken through the individual steps).

I'd be happy to contribute to it with GitHub Actions example.

LanceMcCarthy · 2020-08-19T21:21:49Z

Is this planned to merge soon? I noticed you have 2 PRs open.

I have an opportunity to try out the Broker again in a workflow because my last manual publish has passed certification (I would like to give the Action a shot for real).

HowardWolosky · 2020-08-19T21:27:44Z

Is this planned to merge soon? I noticed you have 2 PRs open.

Done. One PR was for master (StoreBroker v1), and the other was for the v2 branch (which targets the v2 API).

HowardWolosky added the bug label Aug 13, 2020

HowardWolosky mentioned this issue Aug 14, 2020

Fix character escaping issues during config generation #198

Merged

HowardWolosky closed this as completed in #198 Aug 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PackageTool.ps1] Conversion from JSON failed with error: Unterminated string. #197

[PackageTool.ps1] Conversion from JSON failed with error: Unterminated string. #197

LanceMcCarthy commented Aug 11, 2020 •

edited

Loading

HowardWolosky commented Aug 13, 2020

HowardWolosky commented Aug 13, 2020

HowardWolosky commented Aug 13, 2020

LanceMcCarthy commented Aug 13, 2020

HowardWolosky commented Aug 13, 2020

LanceMcCarthy commented Aug 13, 2020 •

edited

Loading

LanceMcCarthy commented Aug 13, 2020

HowardWolosky commented Aug 13, 2020 •

edited

Loading

LanceMcCarthy commented Aug 14, 2020

HowardWolosky commented Aug 14, 2020

LanceMcCarthy commented Aug 14, 2020 •

edited

Loading

LanceMcCarthy commented Aug 14, 2020 •

edited

Loading

HowardWolosky commented Aug 14, 2020

LanceMcCarthy commented Aug 14, 2020 •

edited

Loading

LanceMcCarthy commented Aug 19, 2020

HowardWolosky commented Aug 19, 2020

[PackageTool.ps1] Conversion from JSON failed with error: Unterminated string. #197

[PackageTool.ps1] Conversion from JSON failed with error: Unterminated string. #197

Comments

LanceMcCarthy commented Aug 11, 2020 • edited Loading

Script

Output

HowardWolosky commented Aug 13, 2020

HowardWolosky commented Aug 13, 2020

HowardWolosky commented Aug 13, 2020

LanceMcCarthy commented Aug 13, 2020

Metadata

Side Note - Building a new Broker

HowardWolosky commented Aug 13, 2020

LanceMcCarthy commented Aug 13, 2020 • edited Loading

LanceMcCarthy commented Aug 13, 2020

Output

HowardWolosky commented Aug 13, 2020 • edited Loading

LanceMcCarthy commented Aug 14, 2020

HowardWolosky commented Aug 14, 2020

LanceMcCarthy commented Aug 14, 2020 • edited Loading

LanceMcCarthy commented Aug 14, 2020 • edited Loading

SubmissionData.json

SBConfig.json

Source Control-based Config

Next Steps

HowardWolosky commented Aug 14, 2020

LanceMcCarthy commented Aug 14, 2020 • edited Loading

LanceMcCarthy commented Aug 19, 2020

HowardWolosky commented Aug 19, 2020

LanceMcCarthy commented Aug 11, 2020 •

edited

Loading

LanceMcCarthy commented Aug 13, 2020 •

edited

Loading

HowardWolosky commented Aug 13, 2020 •

edited

Loading

LanceMcCarthy commented Aug 14, 2020 •

edited

Loading

LanceMcCarthy commented Aug 14, 2020 •

edited

Loading

LanceMcCarthy commented Aug 14, 2020 •

edited

Loading