File Sync fails when syncing thousands of files: kubectl argument list too long

[dfernandezm]

This is issue has been created off the back of a comment in another:

#531 (comment)

Expected behavior

FileSync should work without error

Actual behavior

Errors happen during the File Sync:

time="2020-08-03T15:54:49+01:00" level=warning msg="Skipping deploy due to sync error: deleting files: starting command /usr/local/bin/kubectl --context gke_qumu-dev_europe-west1-b_qumu-dev-dev1 exec qumucloud-dfernandez-85dc9c9d85-xn9xk --namespace default -c qumucloud-dfernandez -- rm -rf -- /usr/local/tomcat/webapps/ROOT/WEB-INF/classes/com/kuluvalley/guru/server/api/cache/Delegate.class /usr/local/tomcat/webapps/ROOT/WEB-INF/classes/com/kuluvalley/guru/server/api/content/chapter/ChapterApi.class /usr/local/tomcat/webapps/ROOT/WEB-INF/classes/com/kuluvalley/guru/server/integration/stoneshot/GetSessionsInvitedForContactResponse.class /usr/local/tomcat/webapps/ROOT/WEB-INF/classes/com/kuluvalley/guru/server/security/SsoUserMapper.class /usr/local/tomcat/webapps/ROOT/WEB-INF/classes/com/kuluvalley/guru/server/security/jwt/JwtSigningException.class /usr/local/tomcat/webapps/ROOT/WEB-INF/classes/com/kuluvalley/guru/server/api/dto/AsyncJobDto.class /usr/local/tomcat/webapps/ROOT/WEB-INF/classes/com/kuluvalley/guru/server/integration/stone...
-- MANY MORE FILE PATHS --
INF/classes/com/kuluvalley/guru/server/presentation/controller/admin/MetadataTypeEnumController.class /usr/local/tomcat/webapps/ROOT/WEB-INF/classes/com/kuluvalley/guru/server/security/KeyStoreConfig.class: fork/exec /usr/local/bin/kubectl: argument list too long"

Information

• Skaffold version: 1.13.1
• Operating system: Mac OSX 10.15.5
• Contents of skaffold.yaml:

artifacts:
        - image: "skaffold-qc"
          context: '..'
          sync:
            manual:
            - src: 'KVP/**/*.class'
              dest: /usr/local/tomcat/webapps/ROOT
              strip: 'KVP'
            - src: 'KVP/client/**/*.*'
              dest: /usr/local/tomcat/webapps/ROOT
              strip: 'KVP'
            - src: 'deploy/external-configuration/app/external-configuration/*.*'
              dest: /app/external-configuration
              strip: 'deploy/external-configuration/app/external-configuration'

Steps to reproduce the behavior

Run skaffold command like this:

skaffold dev --filename skaffold.yml --profile qc --trigger=polling --cleanup=false --auto-build=false --auto-deploy=false --default-repo repo  --port-forward=true --status-check=false

[MarlonGamez]

Hi @dfernandezm , thanks for opening this issue! I'm currently wondering what could be done about this. It seems like we're simply running in to the hard limit that's defined by your OS (more context here).

Perhaps the kubectl call could be split into multiple calls based off the user's ARG_MAX or equivalent for other OS's?

Would it be possible for you to create some kind of public repo that we could use to recreate this issue?

[dfernandezm]

Hi @MarlonGamez, thanks for the attention. I can definitely try.

Would it be enough the following?

• Manifests of sample app (whoami?)
• Folder with thousands of files
• Skaffold yaml with profile that syncs that folder so it shows the issue

I won't be able to provide a testing kubernetes cluster, is that ok?

Also, even I am quite a newbie in Go, I'll be glad to help with a fix if pointed in the right area of the code and provided with an approach.

Thanks a lot.

[briandealwis]

@dfernandezm could you describe what was going on at the time? Skaffold has a file-system event debouncer that should try to wait for a quiet period before perfoming a sync. It looks like Skaffold is trying to delete hundreds of .class files. Why did these class files disappear?

[dfernandezm]

@briandealwis thanks for your reply.

There is a build happening, affecting thousands of files, the build is 'cleaning' and recreating the files in that folder. This process takes 10-20 seconds in total, during which Skaffold tries to sync and fails.

I remember all this syncing was working at some point, this is why it was implemented like this, but it's possible there are more files now.

[briandealwis]

Do you think this sync is happening mid-way during your rebuild? If so, you could try using --trigger=manual to control rebuilding using the keyboard (basically: Skaffold waits for an Return to perform rebuilds/redeploys/syncs).

I think Skaffold should support alternative sync strategies, such as using rsync or mutagen, but that's a longer-term change.

[dfernandezm]

This is a good point, maybe a manual trigger would work for backend engineers (lots of automated deletion, batched builds) while leaving frontend with polling (immediate changes). Will test and feed back.

[briandealwis]

@dfernandezm what kind of builder are you using? A jib builder or a custom builder?

[dfernandezm]

This app uses a customised 'jib' build, which in turn fires a gradle build with multiple steps, one of them does a custom clean task very time that deletes all those files. On top of that, IntelliJ IDEA may clear them too. I have to still test the option of the manual trigger, but that'd be a workaround.