Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

In 10.15.0 the json output from 'azcopy.exe jobs list --output-type json' does not work #1777

Closed
gregpakes opened this issue May 12, 2022 · 8 comments · Fixed by #1795
Closed

In 10.15.0 the json output from 'azcopy.exe jobs list --output-type json' does not work #1777

gregpakes opened this issue May 12, 2022 · 8 comments · Fixed by #1795
Assignees
@nakulkar-msft
Labels
bug fixed-in-dev
Milestone
10.15.1

Comments

@gregpakes
Copy link

Which version of the AzCopy was used?

10.15.0

Note: The version is visible when running AzCopy without any argument

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

azcopy.exe jobs list --output-type json

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

What problem was encountered?

There is no output.

How can we reproduce the problem in the simplest way?

Run azcopy.exe jobs list --output-type json

Have you found a mitigation/solution?

It works fine in 10.14.1 - we have changed our build/release pipelines to forcefully use that version.

Link to static version is here if anyone else needs it:
https://azcopyvnext.azureedge.net/release20220315/azcopy_windows_amd64_10.14.1.zip
@JasonYeMSFT
Copy link

Have you tried adding the flag by "--output-type=json" instead of "--output-type json"?

@gregpakes
Copy link
Author

Hi @JasonYeMSFT - Has this changed then? We've been using it like this for years.

@gregpakes
Copy link
Author

@JasonYeMSFT - I have just tested this. It does not resolve the issue.

@nakulkar-msft nakulkar-msft self-assigned this May 17, 2022
@nakulkar-msft
Copy link
Member

Hi @gregpakes, can you confirm if you're seeing this?
0 0x000000000043a9e0 in runtime.fatalpanic
at /usr/local/go/src/runtime/panic.go:1271
1 0x000000000043a286 in runtime.gopanic
at /usr/local/go/src/runtime/panic.go:1147
2 0x0000000000ed7005 in github.com/Azure/azure-storage-azcopy/v10/common.PanicIfErr
at ./common/lifecyleMgr.go:674
3 0x0000000000ed7eec in github.com/Azure/azure-storage-azcopy/v10/common.(*jobLogger).CloseLog
at ./common/logger.go:168

@gregpakes
Copy link
Author

Yes - that looks very familiar

@nakulkar-msft nakulkar-msft linked a pull request May 23, 2022 that will close this issue
Fix jobs list #1795
Merged
@geekzter
Copy link
Member

geekzter commented May 27, 2022 •
edited

It's not just the json output. A plain azcopy jobs list will fail as well:

~/src/github/geekzter/files-sync/scripts [main ≡]> azcopy jobs list                   
panic: close /var/folders/11/g1qq6mx94fqby61trclr71gm0000gn/T/22436eb2-525b-6e41-4b0c-7e535053e21a.log: file already closed

goroutine 1 [running]:
github.com/Azure/azure-storage-azcopy/v10/common.PanicIfErr(...)
        github.com/Azure/azure-storage-azcopy/v10/common/lifecyleMgr.go:674
github.com/Azure/azure-storage-azcopy/v10/common.(*jobLogger).CloseLog(0x140001abbc0)
        github.com/Azure/azure-storage-azcopy/v10/common/logger.go:168 +0xc0
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).CloseLog(0x14000451180)
        github.com/Azure/azure-storage-azcopy/v10/ste/mgr-JobMgr.go:671 +0x34
github.com/Azure/azure-storage-azcopy/v10/jobsAdmin.ListJobs(0x64)
        github.com/Azure/azure-storage-azcopy/v10/jobsAdmin/init.go:766 +0x300
github.com/Azure/azure-storage-azcopy/v10/cmd.inprocSend({0x1015d2c15, 0x8}, {0x10191d0c0?, 0x101f4f800?}, {0x1017cc960?, 0x14000544270?})
        github.com/Azure/azure-storage-azcopy/v10/cmd/rpc.go:50 +0x21c
github.com/Azure/azure-storage-azcopy/v10/cmd.glob..func6({0x1015d2c15?, 0x101945cc0?}, {0x10191d0c0?, 0x101f4f800?}, {0x1017cc960?, 0x14000544270?})
        github.com/Azure/azure-storage-azcopy/v10/cmd/rpc.go:36 +0x34
github.com/Azure/azure-storage-azcopy/v10/cmd.HandleListJobsCommand(0x3a7b8c?)
        github.com/Azure/azure-storage-azcopy/v10/cmd/jobsList.go:87 +0x74
github.com/Azure/azure-storage-azcopy/v10/cmd.init.7.func2(0x14000532f00?, {0x101fe4cb8?, 0x0?, 0x0?})
        github.com/Azure/azure-storage-azcopy/v10/cmd/jobsList.go:67 +0xc4
github.com/spf13/cobra.(*Command).execute(0x14000532f00, {0x101fe4cb8, 0x0, 0x0})
        github.com/spf13/cobra@v1.4.0/command.go:860 +0x4ac
github.com/spf13/cobra.(*Command).ExecuteC(0x101fa70e0)
        github.com/spf13/cobra@v1.4.0/command.go:974 +0x354
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.4.0/command.go:902
github.com/Azure/azure-storage-azcopy/v10/cmd.Execute({0x140001a6750?, 0x140001920e0?}, {0x1400003a1f4, 0x30}, {0x1400003a069?, 0x101f3fc00?}, 0x14000085e28?, {0x22436eb2, 0x525b, 0x6e41, ...})
        github.com/Azure/azure-storage-azcopy/v10/cmd/root.go:167 +0x234
main.main()
        github.com/Azure/azure-storage-azcopy/v10/main.go:83 +0x360

If I perform azcopy jobs clean the command will work, but once an azcopy job has run (i.e. the jobs list is not empty) it will fail.

This is azcopy 10.15.0 on macOS 12.4, installed via Homebrew.

@nakulkar-msft nakulkar-msft added this to the 10.15.1 milestone Jun 14, 2022
@riezebosch
Copy link

On 10.14.1 it also produces crap:

{
    "TimeStamp": "2022-06-29T14:26:39.784762+02:00",
    "MessageType": "EndOfJob",
    "MessageContent": "{\"ErrorMessage\":\"\",\"JobIDDetails\":[{\"JobId\":\"ea3228e2-3006-e742-75bb-35951c462092\",\"CommandString\":\"copy --recursive --preserve-permissions --overwrite ifSourceNewer https://azuritereplacement.blob.core.windows.net/source?se=2023-05-31t19%3A24%3A45z\\u0026sig=REDACTED\\u0026sp=rl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A24%3A45z\\u0026sv=2020-08-04 https://azuritereplacement.blob.core.windows.net/sink?se=2023-02-01t20%3A59%3A32z\\u0026sig=REDACTED\\u0026sp=racwdl\\u0026spr=https\\u0026sr=c\\u0026st=2022-06-03t11%3A59%3A32z\\u0026sv=2020-08-04 \",\"StartTime\":1654695011970066000,\"JobStatus\":\"CompletedWithSkipped\"},{\"JobId\":\"cd5511ba-bf34-a643-5a49-3baac1456204\",\"CommandString\":\"copy --recursive https://azuritereplacement.blob.core.windows.net/source?se=2023-05-31t19%3A24%3A45z\\u0026sig=REDACTED\\u0026sp=rl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A24%3A45z\\u0026sv=2020-08-04 https://azuritereplacement.blob.core.windows.net/sink?se=2023-02-01t20%3A59%3A32z\\u0026sig=REDACTED\\u0026sp=racwdl\\u0026spr=https\\u0026sr=c\\u0026st=2022-06-03t11%3A59%3A32z\\u0026sv=2020-08-04 \",\"StartTime\":1654693631908732000,\"JobStatus\":\"Completed\"},{\"JobId\":\"71855c7c-dc1c-7b4a-42bd-91482a0e60b1\",\"CommandString\":\"copy . https://azuritereplacement.blob.core.windows.net/source?se=2022-09-28t20%3A35%3A11z\\u0026sig=REDACTED\\u0026sp=racwdl\\u0026spr=https\\u0026sr=c\\u0026st=2022-06-03t12%3A35%3A11z\\u0026sv=2020-08-04 --recursive \",\"StartTime\":1654259780498162000,\"JobStatus\":\"Completed\"},{\"JobId\":\"14b7b578-6bc8-da4d-6347-58a229031b5d\",\"CommandString\":\"sync https://azuritereplacement.blob.core.windows.net/source?se=2023-05-31t19%3A24%3A45z\\u0026sig=REDACTED\\u0026sp=rl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A24%3A45z\\u0026sv=2020-08-04 https://azuritereplacement.blob.core.windows.net/sink?se=2023-05-31t19%3A25%3A39z\\u0026sig=REDACTED\\u0026sp=rcwl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A25%3A39z\\u0026sv=2020-08-04 --delete-destination true --preserve-permissions --log-level NONE \",\"StartTime\":1654077789080913000,\"JobStatus\":\"Failed\"},{\"JobId\":\"b0e27b7a-96b2-7546-69a0-d4e19db4cad2\",\"CommandString\":\"sync https://azuritereplacement.blob.core.windows.net/source?se=2023-05-31t19%3A24%3A45z\\u0026sig=REDACTED\\u0026sp=rl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A24%3A45z\\u0026sv=2020-08-04 https://azuritereplacement.blob.core.windows.net/sink?se=2023-05-31t19%3A25%3A39z\\u0026sig=REDACTED\\u0026sp=rcwl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A25%3A39z\\u0026sv=2020-08-04 --delete-destination true --preserve-permissions \",\"StartTime\":1654077764449885000,\"JobStatus\":\"Failed\"},{\"JobId\":\"c358de7d-fcfa-224a-7d09-46cd8048049a\",\"CommandString\":\"sync https://azuritereplacement.blob.core.windows.net/source?se=2023-05-31t19%3A24%3A45z\\u0026sig=REDACTED\\u0026sp=rl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A24%3A45z\\u0026sv=2020-08-04 https://azuritereplacement.blob.core.windows.net/sink?se=2023-05-31t19%3A25%3A39z\\u0026sig=REDACTED\\u0026sp=rcwl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A25%3A39z\\u0026sv=2020-08-04 --delete-destination true --preserve-permissions \",\"StartTime\":1654077741463447000,\"JobStatus\":\"Completed\"},{\"JobId\":\"78227ebc-7e55-0a47-6981-1536c0b78b6b\",\"CommandString\":\"sync https://azuritereplacement.blob.core.windows.net/source?se=2023-05-31t19%3A24%3A45z\\u0026sig=REDACTED\\u0026sp=rl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A24%3A45z\\u0026sv=2020-08-04 https://azuritereplacement.blob.core.windows.net/sink?se=2023-05-31t19%3A25%3A39z\\u0026sig=REDACTED\\u0026sp=rcwl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A25%3A39z\\u0026sv=2020-08-04 --delete-destination true --preserve-permissions --log-level NONE \",\"StartTime\":1654077691968667000,\"JobStatus\":\"Completed\"},{\"JobId\":\"d8e5e86a-00f2-9147-7a33-6d807b2e5af9\",\"CommandString\":\"sync https://azuritereplacement.blob.core.windows.net/source?se=2023-05-31t19%3A24%3A45z\\u0026sig=REDACTED\\u0026sp=rl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A24%3A45z\\u0026sv=2020-08-04 https://azuritereplacement.blob.core.windows.net/sink?se=2023-05-31t19%3A25%3A39z\\u0026sig=REDACTED\\u0026sp=rcwl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A25%3A39z\\u0026sv=2020-08-04 --delete-destination true --preserve-permissions \",\"StartTime\":1654077667592650000,\"JobStatus\":\"Completed\"},{\"JobId\":\"dc8dd1d8-9159-8f4a-4ef9-664dc6f94480\",\"CommandString\":\"sync https://azuritereplacement.blob.core.windows.net/source?se=2023-05-31t19%3A24%3A45z\\u0026sig=REDACTED\\u0026sp=rl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A24%3A45z\\u0026sv=2020-08-04 https://azuritereplacement.blob.core.windows.net/sink?se=2023-05-31t19%3A25%3A39z\\u0026sig=REDACTED\\u0026sp=rcwl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A25%3A39z\\u0026sv=2020-08-04 --delete-destination true --preserve-permissions \",\"StartTime\":1654077536150650000,\"JobStatus\":\"Completed\"},{\"JobId\":\"3a66ba96-bb64-8d47-73ca-fded2b01588a\",\"CommandString\":\"sync https://azuritereplacement.blob.core.windows.net/source?se=2023-05-31t19%3A24%3A45z\\u0026sig=REDACTED\\u0026sp=rl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A24%3A45z\\u0026sv=2020-08-04 https://azuritereplacement.blob.core.windows.net/sink?se=2023-05-31t19%3A25%3A39z\\u0026sig=REDACTED\\u0026sp=rcwl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A25%3A39z\\u0026sv=2020-08-04 --delete-destination true --preserve-permissions --log-level NONE \",\"StartTime\":1654077515224586000,\"JobStatus\":\"Failed\"},{\"JobId\":\"34ac5a24-b72f-4246-48d4-805ef8dbdea4\",\"CommandString\":\"sync https://azuritereplacement.blob.core.windows.net/source?se=2023-05-31t19%3A24%3A45z\\u0026sig=REDACTED\\u0026sp=rl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A24%3A45z\\u0026sv=2020-08-04 https://azuritereplacement.blob.core.windows.net/sink?se=2023-05-31t19%3A25%3A39z\\u0026sig=REDACTED\\u0026sp=rcwl\\u0026spr=https\\u0026sr=c\\u0026st=2022-05-31t11%3A25%3A39z\\u0026sv=2020-08-04 --delete-destination true --preserve-permissions \",\"StartTime\":1654077445899291000,\"JobStatus\":\"Completed\"}]}",
    "PromptDetails": {
        "PromptType": "",
        "ResponseOptions": null,
        "PromptTarget": ""
    }
}

I really need to do a json decode on the MessageContent property?! Not sure if this is the same for other commands when producing json?

@nakulkar-msft
Copy link
Member

Fixed in 10.16.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nakulkar-msft nakulkar-msft

Labels
bug fixed-in-dev
Projects
None yet
Milestone
10.15.1
Development

Successfully merging a pull request may close this issue.

Fix jobs list Azure/azure-storage-azcopy
5 participants
@gregpakes @riezebosch @geekzter @JasonYeMSFT @nakulkar-msft