Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Please provide tags #915

Closed
yann-morin-1998 opened this issue Mar 31, 2015 · 5 comments
Closed

Please provide tags #915

yann-morin-1998 opened this issue Mar 31, 2015 · 5 comments

Comments

@yann-morin-1998
Copy link
Contributor

Hello!

Currently, the branches are constantly rebased on top of the corresponding
stable branches. That's nice because we can all benefit from fixes from
upstream. :-)

However, this causes some annoyance to build systems, like Buildroot,
because the branches are not stable, and there is no way to have
reproducible builds over time.

The situation now is as follows:

  • either we use a branch name, and the build are non-reproducible
    because the branches change quite often, and two builds are not
    guaranteed to use the same sources;
  • or we use a sha1 to refer to a specific changeset, and we have
    reproducible builds, except the sha1 would disapear next time the
    branch is rebased.

This causes quite some pain to us. For example, the Buildroot project had
a demo booth [0] at the recently ended Embedded Linux Conference, where
we were demonstrating a Raspberry Pi 2 running a firmware built with Buildroot.
Part of the demo was to point to the recipe to reproduce it [1], so interested
people can try and see by themselves. At the time we did the demo, we were
using a specific changeset on the rpi-3.19.y branch, but that changeset is
now gone.

Another example is that when Buildroot tags a release, we point to a sha1
for the Rapsberry Pi Linux kernel to use. And that is only valid for a short
period of time, which is until the corresponding branch is rebased.

I guess other build systems (Yocto, OpenWRT...) do all have the same issue.

Would it be possible that you provide tags (and that they not be over-
written), so that it is possible to provide users with dependendable
references?

If I may, I'd suggest the following scheme:

  • for non-stable branches, after a rebase on an upstream version (say,
    v3.19.2), create a tag named v3.19.2-rpi-0. If there are more commits
    on that branch, you can create further tags, v3.19.2-rpi-1,
    v3.19.2-rpi-2 and so on. Then, on the next rebase (say, to v3.19.3),
    you can tag restarting the sequence to 0: v3.19.3-rpi-0, v3.19.3-rpi-1
    and so on...
  • for stable branches, after merging the upstream branch (say linux-3.18.y),
    create a similarly named sequence of tags.

The tags are guaranteed to be unique, because they contain both the upstream
version and a sequence number specific to RPi.

Of course, if you prefer another naming scheme or whatever, we would all be
glad to have tags we can depend on.

Thanks you! :-)

Regards,
Yann E. MORIN.

[0] https://plus.google.com/+ThomasPetazzoni/posts/iLcjbrbP1Gi
[1] http://elinux.org/Buildroot:TechShowcase2015_Demo

@popcornmix
Copy link
Collaborator

The stable branches (e.g. 3.18.y) don't get rebased, so should have a linear history (including the upstream merge comits). What's the benefit of tags here?

For non-stable branches I can see the use of tags. Not sure we need a tag every commit.
I think if a tag is created before every rebase, then you should be able to see any commit ever made from each of these tags. It will also prevent github from garbage collecting, so any git hash reachable from the tags (which is every commit) should persist permanently, and can be used as external references.

@yann-morin-1998
Copy link
Contributor Author

@popcornmix:

  • stable branches: indeed, they are not rebased. What I meant was that
    providing tags on stable branches would make them "homogeneous" with
    non-stable branches. One could use tags, whether from a stable or
    non-stable branch alike. Tagging the merge is probably the best and
    easiest solution.
  • non-stable branches: of course, I do not expect all commits to be
    tagged! That would be tedious and useless. Tagging before a rebase
    has the disadvantage that one can't use a stable reference untill
    there is a rebase [*]. Tagging right after the rebase is probably
    better IMHO. The best of both worlds would be to have a tag before
    and after! ;-) Of course, any tag will be greatly appreciated!

[*] Yes, sha1s topologically before the tag would still be reachable,
as you noticed. Still, if one could just be able to use tags, that
would be a "homogeneous" experience.

Thank you for the quick and positive response! :-)

Regards,
Yann E. MORIN.

@diederikdehaas
Copy link
Contributor

A tag/release would also help when ppl need to have the kernel headers to build their own modules.

So if a new version of (for example) raspberrypi-bootloader package gets created and a tag is made for that same source code, then ppl can check out the tag and would be assured to have the exact code that was used to build the kernel itself.

My 0.02

tpetazzoni pushed a commit to buildroot/buildroot that referenced this issue May 1, 2015
This changes the kernel used on the Raspberry Pi 2 from the 3.19 branch
back to the 3.18 branch. This provides a couple of advantages:

  1. mmc0 works again. Floris Bos found out that this was due to using
     the precompiled DTB files from rpi-firmware. These DTBs were built
     using 3.18.
  2. The rpi-3.18.y branch is not regularly rebased like rpi-3.19.y
     according to popcornmix. See
     raspberrypi/linux#915.

Signed-off-by: Frank Hunleth <fhunleth@troodon-software.com>
Reviewed-by: Floris Bos <bos@je-eigen-domein.nl>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
@diederikdehaas
Copy link
Contributor

As the releases now do get tagged, shouldn't this issue be closed?

@XECDesign
Copy link
Contributor

Yup, I suppose so.

popcornmix pushed a commit that referenced this issue Dec 24, 2018
…ase()"

This reverts commit 61c6de6.

The reverted commit added page reference counting to iomap page
structures that are used to track block size < page size state. This
was supposed to align the code with page migration page accounting
assumptions, but what it has done instead is break XFS filesystems.
Every fstests run I've done on sub-page block size XFS filesystems
has since picking up this commit 2 days ago has failed with bad page
state errors such as:

# ./run_check.sh "-m rmapbt=1,reflink=1 -i sparse=1 -b size=1k" "generic/038"
....
SECTION       -- xfs
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 test1 4.20.0-rc6-dgc+
MKFS_OPTIONS  -- -f -m rmapbt=1,reflink=1 -i sparse=1 -b size=1k /dev/sdc
MOUNT_OPTIONS -- /dev/sdc /mnt/scratch

generic/038 454s ...
 run fstests generic/038 at 2018-12-20 18:43:05
 XFS (sdc): Unmounting Filesystem
 XFS (sdc): Mounting V5 Filesystem
 XFS (sdc): Ending clean mount
 BUG: Bad page state in process kswapd0  pfn:3a7fa
 page:ffffea0000ccbeb0 count:0 mapcount:0 mapping:ffff88800d9b6360 index:0x1
 flags: 0xfffffc0000000()
 raw: 000fffffc0000000 dead000000000100 dead000000000200 ffff88800d9b6360
 raw: 0000000000000001 0000000000000000 00000000ffffffff
 page dumped because: non-NULL mapping
 CPU: 0 PID: 676 Comm: kswapd0 Not tainted 4.20.0-rc6-dgc+ #915
 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.1-1 04/01/2014
 Call Trace:
  dump_stack+0x67/0x90
  bad_page.cold.116+0x8a/0xbd
  free_pcppages_bulk+0x4bf/0x6a0
  free_unref_page_list+0x10f/0x1f0
  shrink_page_list+0x49d/0xf50
  shrink_inactive_list+0x19d/0x3b0
  shrink_node_memcg.constprop.77+0x398/0x690
  ? shrink_slab.constprop.81+0x278/0x3f0
  shrink_node+0x7a/0x2f0
  kswapd+0x34b/0x6d0
  ? node_reclaim+0x240/0x240
  kthread+0x11f/0x140
  ? __kthread_bind_mask+0x60/0x60
  ret_from_fork+0x24/0x30
 Disabling lock debugging due to kernel taint
....

The failures are from anyway that frees pages and empties the
per-cpu page magazines, so it's not a predictable failure or an easy
to debug failure.

generic/038 is a reliable reproducer of this problem - it has a 9 in
10 failure rate on one of my test machines. Failure on other
machines have been at random points in fstests runs but every run
has ended up tripping this problem. Hence generic/038 was used to
bisect the failure because it was the most reliable failure.

It is too close to the 4.20 release (not to mention holidays) to
try to diagnose, fix and test the underlying cause of the problem,
so reverting the commit is the only option we have right now. The
revert has been tested against a current tot 4.20-rc7+ kernel across
multiple machines running sub-page block size XFs filesystems and
none of the bad page state failures have been seen.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Cc: Piotr Jaroszynski <pjaroszynski@nvidia.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: William Kucharski <william.kucharski@oracle.com>
Cc: Darrick J. Wong <darrick.wong@oracle.com>
Cc: Brian Foster <bfoster@redhat.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
popcornmix pushed a commit that referenced this issue Jan 1, 2019
…ase()"

[ Upstream commit a837eca ]

This reverts commit 61c6de6.

The reverted commit added page reference counting to iomap page
structures that are used to track block size < page size state. This
was supposed to align the code with page migration page accounting
assumptions, but what it has done instead is break XFS filesystems.
Every fstests run I've done on sub-page block size XFS filesystems
has since picking up this commit 2 days ago has failed with bad page
state errors such as:

# ./run_check.sh "-m rmapbt=1,reflink=1 -i sparse=1 -b size=1k" "generic/038"
....
SECTION       -- xfs
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 test1 4.20.0-rc6-dgc+
MKFS_OPTIONS  -- -f -m rmapbt=1,reflink=1 -i sparse=1 -b size=1k /dev/sdc
MOUNT_OPTIONS -- /dev/sdc /mnt/scratch

generic/038 454s ...
 run fstests generic/038 at 2018-12-20 18:43:05
 XFS (sdc): Unmounting Filesystem
 XFS (sdc): Mounting V5 Filesystem
 XFS (sdc): Ending clean mount
 BUG: Bad page state in process kswapd0  pfn:3a7fa
 page:ffffea0000ccbeb0 count:0 mapcount:0 mapping:ffff88800d9b6360 index:0x1
 flags: 0xfffffc0000000()
 raw: 000fffffc0000000 dead000000000100 dead000000000200 ffff88800d9b6360
 raw: 0000000000000001 0000000000000000 00000000ffffffff
 page dumped because: non-NULL mapping
 CPU: 0 PID: 676 Comm: kswapd0 Not tainted 4.20.0-rc6-dgc+ #915
 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.1-1 04/01/2014
 Call Trace:
  dump_stack+0x67/0x90
  bad_page.cold.116+0x8a/0xbd
  free_pcppages_bulk+0x4bf/0x6a0
  free_unref_page_list+0x10f/0x1f0
  shrink_page_list+0x49d/0xf50
  shrink_inactive_list+0x19d/0x3b0
  shrink_node_memcg.constprop.77+0x398/0x690
  ? shrink_slab.constprop.81+0x278/0x3f0
  shrink_node+0x7a/0x2f0
  kswapd+0x34b/0x6d0
  ? node_reclaim+0x240/0x240
  kthread+0x11f/0x140
  ? __kthread_bind_mask+0x60/0x60
  ret_from_fork+0x24/0x30
 Disabling lock debugging due to kernel taint
....

The failures are from anyway that frees pages and empties the
per-cpu page magazines, so it's not a predictable failure or an easy
to debug failure.

generic/038 is a reliable reproducer of this problem - it has a 9 in
10 failure rate on one of my test machines. Failure on other
machines have been at random points in fstests runs but every run
has ended up tripping this problem. Hence generic/038 was used to
bisect the failure because it was the most reliable failure.

It is too close to the 4.20 release (not to mention holidays) to
try to diagnose, fix and test the underlying cause of the problem,
so reverting the commit is the only option we have right now. The
revert has been tested against a current tot 4.20-rc7+ kernel across
multiple machines running sub-page block size XFs filesystems and
none of the bad page state failures have been seen.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Cc: Piotr Jaroszynski <pjaroszynski@nvidia.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: William Kucharski <william.kucharski@oracle.com>
Cc: Darrick J. Wong <darrick.wong@oracle.com>
Cc: Brian Foster <bfoster@redhat.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
pfpacket pushed a commit to pfpacket/linux-rpi-rust that referenced this issue Apr 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants