Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Finish creating project before downloading image. #410

Closed
xerrni opened this issue Mar 3, 2022 · 5 comments
Closed

Finish creating project before downloading image. #410

xerrni opened this issue Mar 3, 2022 · 5 comments
Labels
enhancement New feature or request
Milestone
v33.0.0

Comments

@xerrni
Copy link
Contributor

xerrni commented Mar 3, 2022 •
edited

Currently when project is created a POST request is sent.
Response for this request is sent after input is downloaded.

In my opinion it is wrong user experience as download can take long time (huge docker images). I propose that web container should return page with created project before image is downloaded.

Current behavior breaks some nginx configuration as timeout occurs in nginx before it receives response from web container and it returns 504 error code (gateway timeout) as response.

Furthermore when user hits refresh button request is resend and web re-downloads this input.
@pombredanne
Copy link
Member

@xerrni

In my opinion it is wrong user experience as download can take long time (huge docker images). I propose that web container should return page with created project before image is downloaded.

I completely agree. Large images (including huge VM or Docker images that can routinely be 30 to 50GB big) are problematic.
There are a few behaviours possible:

  1. download in the background after the project has been created
  2. kick in only when the pipeline is executed, download as a pre-pipeline process if there are downloadable URLs
  3. kick in only when the pipeline is executed, download as a first explicit step of a pipeline if there are downloadable URLs. A pipeline without a download step would not download.

Which ones would be the better way?

@cco3 what's your take there?

@cco3
Copy link

cco3 commented Mar 4, 2022 •
edited

I'm not sure I get the distinctions in the options, but returning immediately with a project created seems to make the most sense...the download could just be the first stage in the pipeline.

@aalexanderr
Copy link
Contributor

aalexanderr commented Mar 7, 2022 •
edited

I think 3rd option is best - fetching is already a pipe and it would make scaling amount of workers easier based on amount of runs to process

@tdruez tdruez added this to the v32.0.0 milestone Apr 4, 2022
@pombredanne pombredanne mentioned this issue Apr 15, 2022
Closed
@pombredanne
Copy link
Member

See also #372

@pombredanne pombredanne added the enhancement New feature or request label Apr 15, 2022
tdruez added a commit that referenced this issue May 13, 2022
@tdruez
Add a download inputs pre-step in the Pipeline class #410
dac45ec 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue May 16, 2022
@tdruez
Refactor the input_sources field into a new InputSource model #410
eaf8398 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue May 16, 2022
@tdruez
Refine the InputSource model implementation #410
9d39e7d 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue May 16, 2022
@tdruez
Display upload progress in browser tab #410
6fd1111 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue May 17, 2022
@tdruez
Adapt the project details view to new InputSource model #410
ed6ba7e 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue May 17, 2022
@tdruez
Consolidate the input sources values into a dict for usage in the view …
331cbc3 
…#410

Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue May 17, 2022
@tdruez
Update the RunStatusCommandMixin to the new InputSource system #410
7a3efcb 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue May 18, 2022
@tdruez
Update the can_add_input logic #410
e951eb4 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue May 23, 2022
@tdruez
Add modal confirmation for input deletion #410
08fcf11 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue May 23, 2022
@tdruez
Refine project inputs panel UI #410
4d62c2f 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue May 23, 2022
@tdruez
Add support for manually added inputs #410
4e6cce7 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Aug 9, 2022
@tdruez
Merge main branch and fix conflicts #410
8858c07 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Sep 12, 2022
@tdruez
Merge main and fix conflicts #410
315204c 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
@DennisClark DennisClark modified the milestones: v32.0.0, v33.0.0 Dec 8, 2022
@DennisClark DennisClark modified the milestones: v33.0.0, v33.1.0 Jan 12, 2023
tdruez added a commit that referenced this issue Jul 19, 2023
@tdruez
Display upload progress in browser tab #410
0136724 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jul 21, 2023
@tdruez
Update project inputs #813 (#814)
3ed1e9e 
* Add support for multiple from/to inputs in d2d pipeline #813

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Display upload progress in browser tab #410

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Add ability to delete project inputs #813

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Add ability to download project inputs #813

Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Nov 30, 2023
@tdruez
Migrate the Project.input_sources field to its own model #410
313874e 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Nov 30, 2023
@tdruez
Fix part of failing tests #410
3c0e4c4 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Dec 21, 2023
@tdruez
Merge main #410
a1b4dde 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 2, 2024
@tdruez
Add check on URL availability #410
e8862f2 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 2, 2024
@tdruez
Fix the model-related functions #410
884927b 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 2, 2024
@tdruez
Migrate the check_for_missing_inputs logic #410
f34de13 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 2, 2024
@tdruez
Fix the input related templates #410
8aa54df 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 2, 2024
@tdruez
Download missing inputs as a pipeline step #410
b65011b 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 2, 2024
@tdruez
Refine the reverse migration #410
f824f6a 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 2, 2024
@tdruez
Fix part of the failing tests #410
c5a7729 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 3, 2024
@tdruez
Fix part of the failing tests part2 #410
a6e28b0 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 3, 2024
@tdruez
Fix part of the failing tests part3 #410
ac71c1c 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 3, 2024
@tdruez
Fix part of the failing tests part4 #410
6cff2ae 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 3, 2024
@tdruez
Sort by filename for consistency across systems #410
8523409 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 4, 2024
@tdruez
Add changelog entry #410
6f320f0 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 4, 2024
@tdruez
Add unit test for the InputSource #410
b6e301c 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 4, 2024
@tdruez
Add unit test for the InputSource fetch method #410
cda5bb2 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 4, 2024
@tdruez
Add unit test for the download_missing_inputs method #410
d91d35f 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 4, 2024
@tdruez
Small change to the reverse migrations #410
2882edb 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 5, 2024
@tdruez
Add a tag field on the InputSource model #410
2bc9ff6 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 5, 2024
@tdruez
Add the tag field to the InputSourceSerializer #410
e7753bd 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 5, 2024
@tdruez
Fix failing CI tests #410
86e36a9 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 6, 2024
@tdruez
Update the input_sources system in the API #410
9115b45 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jan 8, 2024
@tdruez
Migrate the Project.input_sources field to its own model #410 (#1039)
d207673 
* Migrate the Project.input_sources field to its own model #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Fix part of failing tests #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Add check on URL availability #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Fix the model-related functions #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Migrate the check_for_missing_inputs logic #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Fix the input related templates #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Download missing inputs as a pipeline step #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Refine the reverse migration #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Fix part of the failing tests #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Fix part of the failing tests part2 #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Fix part of the failing tests part3 #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Fix part of the failing tests part4 #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Sort by filename for consistency across systems #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Add changelog entry #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Add unit test for the InputSource #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Add unit test for the InputSource fetch method #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Add unit test for the download_missing_inputs method #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Small change to the reverse migrations #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Add a tag field on the InputSource model #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Add the tag field to the InputSourceSerializer #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Fix failing CI tests #410

Signed-off-by: tdruez <tdruez@nexb.com>

* Update the input_sources system in the API #410

Signed-off-by: tdruez <tdruez@nexb.com>

---------

Signed-off-by: tdruez <tdruez@nexb.com>
@tdruez
Copy link
Member

tdruez commented Jan 8, 2024 •
edited

The download of the Project's inputs now takes place in the pipeline run.

@tdruez tdruez closed this as completed Jan 8, 2024
tdruez added a commit that referenced this issue Jan 9, 2024
@tdruez
Include input_sources in the API Project list #410
7806328 
Signed-off-by: tdruez <tdruez@nexb.com>
@tdruez tdruez modified the milestones: v33.1.0, v33.0.0 Jan 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
v33.0.0
Development

No branches or pull requests
6 participants
@cco3 @tdruez @pombredanne @DennisClark @aalexanderr @xerrni