[BUG] Job list command is limited to 50 items

[rebeccaalpert]

Describe the bug
The job list command is limited to 50 items. We regularly have more than 50 jobs in a project.

I see that the Memsource API limits you to 50 items per page, but at least it lets you see the total number of elements and total number of pages (i.e. ..."numberOfElements":50,"totalElements":91,"pageSize":50,"totalPages":2}). The Memsource CLI client doesn't seem to provide this information.

I'm working on a tool to automate job downloads. I can increment the number of pages I request until I get back an empty array, but it's a pain and not what I expected initially (I was unfamiliar with this Memsource API limitation).

Affected version
Latest.

To Reproduce
memsource job download --project-id tCQYoSuMvoUxatSpkj0On3 --output-dir downloaded_po_files/ja --job-id $(memsource job list --project-id tCQYoSuMvoUxatSpkj0On3 --target-lang 'ja' -f value -c uid | tr '\n' ' ')

I used https://cloud.memsource.com/web/api2/v2/projects/tCQYoSuMvoUxatSpkj0On3/jobs?targetLang=ja to find the total number of pages/elements.

Expected behavior
It should return all jobs, not just the first 50, or at least give the user information on how many additional items and/or pages there are.

[ludekjanda]

Hi @rebeccaalpert I am hitting the same problem in my projects as well. Let's see what we can do about it.

[ludekjanda]

Hi @rebeccaalpert ,

probably the best workaround is to increase the page numbers until you'll get a list of jobs which is lower than 50:

#!/bin/bash

# Set the page number to first page
page_number=1
output=$(memsource job list --project-id tCQYoSuMvoUxatSpkj0On3 --page-number ${page_number} -f value -c uid | tr '\n' ' ')
((page_number++))

# Execute command until there are less than 50 jobs
while [ $(memsource job list --project-id tCQYoSuMvoUxatSpkj0On3 --page-number ${page_number} -f value -c uid | wc -l) -eq 50 ]; do
  # Save output into temporary variable
  temp=$(memsource job list --project-id tCQYoSuMvoUxatSpkj0On3 --page-number ${page_number} -f value -c uid | tr '\n' ' ')
  output=${output}${temp}
  ((page_number++))
done

echo ${output}

However when I checked your project in Memsource and your API call: https://cloud.memsource.com/web/api2/v2/projects/tCQYoSuMvoUxatSpkj0On3/jobs?targetLang=ja

I noticed that it has duplicated jobs. The total for each language is 41, your list is showing 82. In your Memsource project I can see that you have created 2 workflow steps there. Memsource CLI works by default with one step only. You can specify the workflow step with a --workflow-level parameter.
E.g.

memsource job list --project-id tCQYoSuMvoUxatSpkj0On3 --page-number 7 -f value -c uid --workflow-level 1 | wc -l

Please DM me if you need more explanation or help.

[zerodayz]

Hi @rebeccaalpert ,

probably the best workaround is to increase the page numbers until you'll get a list of jobs which is lower than 50:

#!/bin/bash



# Set the page number to first page

page_number=1

output=$(memsource job list --project-id tCQYoSuMvoUxatSpkj0On3 --page-number ${page_number} -f value -c uid | tr '\n' ' ')

((page_number++))



# Execute command until there are less than 50 jobs

while [ $(memsource job list --project-id tCQYoSuMvoUxatSpkj0On3 --page-number ${page_number} -f value -c uid | wc -l) -eq 50 ]; do

  # Save output into temporary variable

  temp=$(memsource job list --project-id tCQYoSuMvoUxatSpkj0On3 --page-number ${page_number} -f value -c uid | tr '\n' ' ')

  output=${output}${temp}

  ((page_number++))

done



echo ${output}

However when I checked your project in Memsource and your API call: https://cloud.memsource.com/web/api2/v2/projects/tCQYoSuMvoUxatSpkj0On3/jobs?targetLang=ja

I noticed that it has duplicated jobs. The total for each language is 41, your list is showing 82. In your Memsource project I can see that you have created 2 workflow steps there. Memsource CLI works by default with one step only. You can specify the workflow step with a --workflow-level parameter.

E.g.

memsource job list --project-id tCQYoSuMvoUxatSpkj0On3 --page-number 7 -f value -c uid --workflow-level 1 | wc -l

Please DM me if you need more explanation or help.

Or you can do 1 extra API call with curl and grab the totalPages and then execute loop with job list for each page.

Regarding the script, you may need to add the one more line at the end

temp=$(memsource job list --project-id tCQYoSuMvoUxatSpkj0On3 --page-number ${page_number} -f value -c uid | tr '\n' ' ')

output=${output}${temp}

Because while wont execute if you have less than 50 (last page would be missing)

[rebeccaalpert]

Like I said in my initial post, I already have a workaround, but I think the library needs the additional info I outlined or to return info without additional work from the user.

I noticed that it has duplicated jobs. The total for each language is 41, your list is showing 82. In your Memsource project I can see that you have created 2 workflow steps there. Memsource CLI works by default with one step only. You can specify the workflow step with a --workflow-level parameter.

Another team created these and I don't have control over their workflow. :) I just need to download whatever is there.

[zerodayz]

@ludekjanda @rebeccaalpert Added that in 0.3.8

The meta is hidden but you can access it with:

 memsource job list --project-id 9iFwa1rk54RL3WWSi9hVf5 -f value -c total_elements -c total_pages -c page_size -c page_number -c number_of_elements

1 1 50 0 1

[zerodayz]

Feel free to update to 0.3.8 and test it out.

Note this is hidden from the output unless you specifically access with -c as above.

[zerodayz]

@rebeccaalpert If you have installed the memsource cli from github, you can just copy & paste https://github.com/zerodayz/memsource-cli-client#install-from-github to update.

[zerodayz]

Closing this issue as resolved. If you still have problem, please feel free to re-open this issue.