Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

need to implement new features to script for packages pushes to S3 #4839

Closed
avtikhon opened this issue Apr 3, 2020 · 1 comment
Closed

need to implement new features to script for packages pushes to S3 #4839

avtikhon opened this issue Apr 3, 2020 · 1 comment
Assignees
@avtikhon
Labels
feature A new functionality qa Issues related to tests or testing subsystem

Comments

@avtikhon
Copy link
Contributor

avtikhon commented Apr 3, 2020 •
edited by ligurio

Bug description:
Need to add more features:

  • create test to check availability and correctness of pushed packages to S3 just after the pushing
  • add ability to save packages with only single package type binaries either sources
  • implement ability to fix metadata files on S3 with missed packages
  • implement ability to fix metadata files on S3 with new packages
  • implement ability to fix metadata files on S3 with changed packages
  • implement ability to remove needed packages from S3
@avtikhon avtikhon added the feature A new functionality label Apr 3, 2020
@avtikhon avtikhon self-assigned this Apr 3, 2020
@avtikhon avtikhon added the qa Issues related to tests or testing subsystem label Apr 3, 2020
@avtikhon avtikhon mentioned this issue Apr 3, 2020
Closed
avtikhon added a commit that referenced this issue Apr 3, 2020
@avtikhon
Add metafiles cleanup routines at S3 pack script
4affbfd 
Added cleanup functionality for the meta files.
Script may have the following situations:

 - package files removed at S3, but it still registered:
   Script stores and registers the new packages at S3 and
   removes all the other registered blocks for the sames
   files in meta files.

 - package files already exists at S3 with the same hashes:
   Script passes it with warning message.

 - package files already exists at S3 with the old hashes:
   Script fails w/o force flag, otherwise it stores and
   registers the new packages at S3 and removes all the other
   registered blocks for the sames files in meta files.

Added '-s|skip_errors' option flag to skip errors on changed
packages to avoid of exits on script run.

Part of #4839
avtikhon added a commit that referenced this issue Apr 3, 2020
@avtikhon
Enable script for saving packages in S3 for modules
2bcbe39 
Found that modules may have only binaries packages w/o sources
packages. Script changed to be able to work with only binaries
either sources packages.

Part of #4839
avtikhon added a commit that referenced this issue Apr 3, 2020
@avtikhon
Add help instruction on 'product' option
7733ab8 
Added instructions on 'product' option with examples.

Part of #4839
avtikhon added a commit that referenced this issue Apr 4, 2020
@avtikhon
Enable script for saving packages in S3 for modules
700bc0d 
Found that modules may have only binaries packages w/o sources
packages. Script changed to be able to work with only binaries
either sources packages.

Part of #4839
avtikhon added a commit that referenced this issue Apr 4, 2020
@avtikhon
Add help instruction on 'product' option
1924243 
Added instructions on 'product' option with examples.

Part of #4839
avtikhon added a commit that referenced this issue Apr 6, 2020
@avtikhon
Implement ability to remove needed packages from S3
4fc4d93 
Added ability to remove given in options package from S3.
Currently only for DEB repositories.

Part of #4839
avtikhon added a commit that referenced this issue Apr 6, 2020
@avtikhon
Implement ability to remove needed packages from S3
b99202b 
Added ability to remove given in options package from S3.
Currently only for DEB repositories.

Part of #4839
avtikhon added a commit that referenced this issue Apr 6, 2020
@avtikhon
Implement ability to remove needed packages from S3
848247a 
Added ability to remove given in options package from S3.
Currently only for DEB repositories.

Part of #4839
avtikhon added a commit that referenced this issue Apr 7, 2020
@avtikhon
Implement ability to remove needed packages from S3
8a7e811 
Added ability to remove given in options package from S3.
Currently only for DEB repositories.

Part of #4839
avtikhon added a commit that referenced this issue Apr 7, 2020
@avtikhon
Implement ability to remove needed packages from S3
22be97e 
Added ability to remove given in options package from S3.
Currently only for DEB repositories.

Part of #4839
avtikhon added a commit that referenced this issue Apr 7, 2020
@avtikhon
Implement ability to remove needed packages from S3
6073a92 
Added ability to remove given in options package from S3.
Currently only for DEB repositories.

Part of #4839
@avtikhon avtikhon added this to DOING in Quality Assurance Apr 7, 2020
avtikhon added a commit that referenced this issue Apr 8, 2020
@avtikhon
Implement ability to remove needed packages from S3
8971371 
Added ability to remove given in options package from S3.

Part of #4839
avtikhon added a commit that referenced this issue Apr 9, 2020
@avtikhon
Added ability to remove packages from S3
c25839c 
Added ability to remove given in options package from S3. TO remove the
needed package need to set '-r=<package name with version>' option,
like:
  ./tools/update_repo.sh -o=<OS> -d=<DIST> -b=<S3 repo> \
    -r=tarantool-2.2.2.0
it will remove all found appropriate source and binaries packages from
the given S3 repository, also the meta files will be corrected there.

Close #4839
avtikhon added a commit that referenced this issue Apr 9, 2020
@avtikhon
Add metafiles cleanup routines at S3 pack script
08ae339 
Added cleanup functionality for the meta files.
Script may have the following situations:

 - package files removed at S3, but it still registered:
   Script stores and registers the new packages at S3 and
   removes all the other registered blocks for the sames
   files in meta files.

 - package files already exists at S3 with the same hashes:
   Script passes it with warning message.

 - package files already exists at S3 with the old hashes:
   Script fails w/o force flag, otherwise it stores and
   registers the new packages at S3 and removes all the other
   registered blocks for the sames files in meta files.

Added '-s|skip_errors' option flag to skip errors on changed
packages to avoid of exits on script run.

Part of #4839
avtikhon added a commit that referenced this issue Apr 9, 2020
@avtikhon
Enable script for saving packages in S3 for modules
e303fe2 
Found that modules may have only binaries packages w/o sources
packages. Script changed to be able to work with only binaries
either sources packages.

Part of #4839
avtikhon added a commit that referenced this issue Apr 9, 2020
@avtikhon
Add help instruction on 'product' option
936fbdb 
Added instructions on 'product' option with examples.

Part of #4839
avtikhon added a commit that referenced this issue Apr 9, 2020
@avtikhon
Added ability to remove packages from S3
2d12e43 
Added ability to remove given in options package from S3. TO remove the
needed package need to set '-r=<package name with version>' option,
like:
  ./tools/update_repo.sh -o=<OS> -d=<DIST> -b=<S3 repo> \
    -r=tarantool-2.2.2.0
it will remove all found appropriate source and binaries packages from
the given S3 repository, also the meta files will be corrected there.

Close #4839
@avtikhon avtikhon moved this from DOING to ON REVIEW in Quality Assurance Apr 9, 2020
avtikhon added a commit that referenced this issue Apr 9, 2020
@avtikhon
Add metafiles cleanup routines at S3 pack script
6617291 
Added cleanup functionality for the meta files.
Script may have the following situations:

 - package files removed at S3, but it still registered:
   Script stores and registers the new packages at S3 and
   removes all the other registered blocks for the sames
   files in meta files.

 - package files already exists at S3 with the same hashes:
   Script passes it with warning message.

 - package files already exists at S3 with the old hashes:
   Script fails w/o force flag, otherwise it stores and
   registers the new packages at S3 and removes all the other
   registered blocks for the sames files in meta files.

Added '-s|skip_errors' option flag to skip errors on changed
packages to avoid of exits on script run.

Part of #4839
avtikhon added a commit that referenced this issue Apr 9, 2020
@avtikhon
Enable script for saving packages in S3 for modules
73f3d24 
Found that modules may have only binaries packages w/o sources
packages. Script changed to be able to work with only binaries
either sources packages.

Part of #4839
avtikhon added a commit that referenced this issue Apr 9, 2020
@avtikhon
Add help instruction on 'product' option
27febf7 
Added instructions on 'product' option with examples.

Part of #4839
avtikhon added a commit that referenced this issue Apr 9, 2020
@avtikhon
Added ability to remove packages from S3
db93852 
Added ability to remove given in options package from S3. TO remove the
needed package need to set '-r=<package name with version>' option,
like:
  ./tools/update_repo.sh -o=<OS> -d=<DIST> -b=<S3 repo> \
    -r=tarantool-2.2.2.0
it will remove all found appropriate source and binaries packages from
the given S3 repository, also the meta files will be corrected there.

Close #4839
@sergos sergos changed the title need to implement new futures to script for packages pushes to S3 need to implement new features to script for packages pushes to S3 Apr 10, 2020
avtikhon added a commit that referenced this issue Apr 11, 2020
@avtikhon
Added ability to remove packages from S3
097b03d 
Added ability to remove given in options package from S3. TO remove the
needed package need to set '-r=<package name with version>' option,
like:
  ./tools/update_repo.sh -o=<OS> -d=<DIST> -b=<S3 repo> \
    -r=tarantool-2.2.2.0
it will remove all found appropriate source and binaries packages from
the given S3 repository, also the meta files will be corrected there.

Close #4839
avtikhon added a commit that referenced this issue Apr 11, 2020
@avtikhon
Added ability to remove packages from S3
d370d53 
Added ability to remove given in options package from S3. TO remove the
needed package need to set '-r=<package name with version>' option,
like:
  ./tools/update_repo.sh -o=<OS> -d=<DIST> -b=<S3 repo> \
    -r=tarantool-2.2.2.0
it will remove all found appropriate source and binaries packages from
the given S3 repository, also the meta files will be corrected there.

Close #4839
kyukhin pushed a commit that referenced this issue Apr 15, 2020
@avtikhon @kyukhin
Enable script for saving packages in S3 for modules
c725805 
Found that modules may have only binaries packages w/o sources
packages. Script changed to be able to work with only binaries
either sources packages.

Part of #4839

(cherry picked from commit 4527a4d)
kyukhin pushed a commit that referenced this issue Apr 15, 2020
@avtikhon @kyukhin
Add help instruction on 'product' option
957abcd 
Added instructions on 'product' option with examples.

Part of #4839

(cherry picked from commit cccc989)
kyukhin pushed a commit that referenced this issue Apr 15, 2020
@avtikhon @kyukhin
Added ability to remove packages from S3
930cce0 
Added ability to remove given in options package from S3. TO remove the
needed package need to set '-r=<package name with version>' option,
like:
  ./tools/update_repo.sh -o=<OS> -d=<DIST> -b=<S3 repo> \
    -r=tarantool-2.2.2.0
it will remove all found appropriate source and binaries packages from
the given S3 repository, also the meta files will be corrected there.

Close #4839

(cherry picked from commit d6c50af)
kyukhin pushed a commit that referenced this issue Apr 15, 2020
@avtikhon @kyukhin
Add metafiles cleanup routines at S3 pack script
b9ce0ba 
Added cleanup functionality for the meta files.
Script may have the following situations:

 - package files removed at S3, but it still registered:
   Script stores and registers the new packages at S3 and
   removes all the other registered blocks for the sames
   files in meta files.

 - package files already exists at S3 with the same hashes:
   Script passes it with warning message.

 - package files already exists at S3 with the old hashes:
   Script fails w/o force flag, otherwise it stores and
   registers the new packages at S3 and removes all the other
   registered blocks for the sames files in meta files.

Added '-s|skip_errors' option flag to skip errors on changed
packages to avoid of exits on script run.

Part of #4839

(cherry picked from commit ed49140)
kyukhin pushed a commit that referenced this issue Apr 15, 2020
@avtikhon @kyukhin
Enable script for saving packages in S3 for modules
983971e 
Found that modules may have only binaries packages w/o sources
packages. Script changed to be able to work with only binaries
either sources packages.

Part of #4839

(cherry picked from commit 4527a4d)
kyukhin pushed a commit that referenced this issue Apr 15, 2020
@avtikhon @kyukhin
Add help instruction on 'product' option
9ad45c4 
Added instructions on 'product' option with examples.

Part of #4839

(cherry picked from commit cccc989)
kyukhin pushed a commit that referenced this issue Apr 15, 2020
@avtikhon @kyukhin
Added ability to remove packages from S3
7a26feb 
Added ability to remove given in options package from S3. TO remove the
needed package need to set '-r=<package name with version>' option,
like:
  ./tools/update_repo.sh -o=<OS> -d=<DIST> -b=<S3 repo> \
    -r=tarantool-2.2.2.0
it will remove all found appropriate source and binaries packages from
the given S3 repository, also the meta files will be corrected there.

Close #4839

(cherry picked from commit d6c50af)
@avtikhon avtikhon moved this from ON REVIEW to DONE in Quality Assurance Apr 15, 2020
@avtikhon
Copy link
Contributor Author

Main patch set with 4 commits merged, but the last feature "checker for S3 packages" still on review at separate patch set at branch: avtikhon/gh-4839-s3-script-check

@avtikhon avtikhon reopened this Apr 15, 2020
@avtikhon avtikhon moved this from DONE to ON REVIEW in Quality Assurance Apr 15, 2020
avtikhon added a commit that referenced this issue Apr 15, 2020
@avtikhon
Add S3 package checker
a4fd2d1 
Created Dockerfile based script to check packages available at the S3
repository. Set this script to be run just after the package creation.

Part of #4839
avtikhon added a commit that referenced this issue Apr 23, 2020
@avtikhon
Add S3 package checker
99f1570 
Created Dockerfile based script to check packages available at the S3
repository. Set this script to be run just after the package creation.

Part of #4839
avtikhon added a commit that referenced this issue Apr 23, 2020
@avtikhon
Add S3 package checker
00981e7 
Created Dockerfile based script to check packages available at the S3
repository. Set this script to be run just after the package creation.

Part of #4839
@avtikhon avtikhon moved this from ON REVIEW to DOING in Quality Assurance Apr 23, 2020
avtikhon added a commit that referenced this issue Apr 23, 2020
@avtikhon
Add S3 package checker
094f225 
Created Dockerfile based script to check packages available at the S3
repository. Set this script to be run just after the package creation.

Part of #4839
@avtikhon avtikhon moved this from DOING to ON REVIEW in Quality Assurance Apr 23, 2020
@avtikhon avtikhon moved this from ON REVIEW to DOING in Quality Assurance Apr 23, 2020
avtikhon added a commit that referenced this issue Apr 28, 2020
@avtikhon
Add S3 package checker
afd2a15 
Created Dockerfile based script to check packages available at the S3
repository. Set this script to be run just after the package creation.

Part of #4839
avtikhon added a commit that referenced this issue May 6, 2020
@avtikhon
Enable script for saving RPM packages in S3 for modules
5fc7393 
Found that modules may have only sources packages w/o binaries
packages. Script updated to be able to work with only binaries
either sources RPM packages. The same fix was already done for
DEB packages at commit 4527a4d.

Part of #4839
@avtikhon avtikhon moved this from DOING to ON REVIEW in Quality Assurance May 6, 2020
avtikhon added a commit that referenced this issue May 6, 2020
@avtikhon
Enable script for saving RPM packages in S3 for modules
5d4407e 
Found that modules may have only sources packages w/o binaries
packages. Script updated to be able to work with only binaries
either sources RPM packages. The same fix was already done for
DEB packages at commit 4527a4d.

Close #4839
avtikhon added a commit that referenced this issue May 15, 2020
@avtikhon
Add S3 package checker
a839a80 
Created Dockerfile based script to check packages available at the S3
repository. Set this script to be run just after the package creation.

Part of #4839
kyukhin pushed a commit that referenced this issue May 20, 2020
@avtikhon @kyukhin
Enable script for saving RPM packages in S3 for modules
e8d7c52 
Found that modules may have only sources packages w/o binaries
packages. Script updated to be able to work with only binaries
either sources RPM packages. The same fix was already done for
DEB packages at commit 4527a4d.

Close #4839

(cherry picked from commit 64d1336)
kyukhin pushed a commit that referenced this issue May 20, 2020
@avtikhon @kyukhin
Enable script for saving RPM packages in S3 for modules
d3dbf35 
Found that modules may have only sources packages w/o binaries
packages. Script updated to be able to work with only binaries
either sources RPM packages. The same fix was already done for
DEB packages at commit 4527a4d.

Close #4839

(cherry picked from commit 64d1336)
kyukhin pushed a commit that referenced this issue May 20, 2020
@avtikhon @kyukhin
Enable script for saving RPM packages in S3 for modules
7db5cfc 
Found that modules may have only sources packages w/o binaries
packages. Script updated to be able to work with only binaries
either sources RPM packages. The same fix was already done for
DEB packages at commit 4527a4d.

Close #4839

(cherry picked from commit 64d1336)
@avtikhon avtikhon moved this from ON REVIEW to DONE in Quality Assurance May 20, 2020
@avtikhon avtikhon removed this from DONE in Quality Assurance May 26, 2020
@avtikhon avtikhon mentioned this issue Jun 25, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@avtikhon avtikhon

Labels
feature A new functionality qa Issues related to tests or testing subsystem
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@avtikhon